IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v288y2021ics0306261921001586.html
   My bibliography  Save this article

Deep reinforcement learning framework for dynamic pricing demand response of regenerative electric heating

Author

Listed:
  • Zhong, Shengyuan
  • Wang, Xiaoyuan
  • Zhao, Jun
  • Li, Wenjia
  • Li, Hao
  • Wang, Yongzhen
  • Deng, Shuai
  • Zhu, Jiebei

Abstract

Applications of electric heating, which can improve carbon emission reduction and renewable energy utilization, have brought new challenges to the safe operation of energy systems around the world. Regenerative electric heating with load aggregators and demand response is an effective means to mitigate the wind curtailment and grid operational risks caused by electric heating. However, there is still a lack of models related to demand response, which results in participants not being able to obtain maximum benefits through dynamic subsidy prices. This study uses the Weber–Fechner law and a clustering algorithm to construct quantitative response characteristics models. The deep Q network was used to build a dynamic subsidy price generation framework for load aggregators. Through simulation analysis based on the evolutionary game model of a project in a rural area in Tianjin, China, the following conclusions were drawn: compared with the benchmark model, regenerative electric heating users can save up to 8.7% of costs, power grid companies can save 56.6% of their investment, and wind power plants can increase wind power consumption by 17.6%. The framework proposed in this study considers user behavior quantification of demand response participants and the differences among users. Therefore, the framework can provide a more reasonable, applicable, and intelligent system for regenerative electric heating.

Suggested Citation

  • Zhong, Shengyuan & Wang, Xiaoyuan & Zhao, Jun & Li, Wenjia & Li, Hao & Wang, Yongzhen & Deng, Shuai & Zhu, Jiebei, 2021. "Deep reinforcement learning framework for dynamic pricing demand response of regenerative electric heating," Applied Energy, Elsevier, vol. 288(C).
    • Handle: RePEc:eee:appene:v:288:y:2021:i:c:s0306261921001586
    DOI: 10.1016/j.apenergy.2021.116623
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261921001586
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2021.116623?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Fang, Kai & Tang, Yiqi & Zhang, Qifeng & Song, Junnian & Wen, Qi & Sun, Huaping & Ji, Chenyang & Xu, Anqi, 2019. "Will China peak its energy-related carbon emissions by 2030? Lessons from 30 Chinese provinces," Applied Energy, Elsevier, vol. 255(C).
    2. Perera, A.T.D. & Kamalaruban, Parameswaran, 2021. "Applications of reinforcement learning in energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 137(C).
    3. Ayón, X. & Gruber, J.K. & Hayes, B.P. & Usaola, J. & Prodanović, M., 2017. "An optimal day-ahead load scheduling approach based on the flexibility of aggregate demands," Applied Energy, Elsevier, vol. 198(C), pages 1-11.
    4. Vázquez-Canteli, José R. & Nagy, Zoltán, 2019. "Reinforcement learning for demand response: A review of algorithms and modeling techniques," Applied Energy, Elsevier, vol. 235(C), pages 1072-1089.
    5. Patteeuw, Dieter & Bruninx, Kenneth & Arteconi, Alessia & Delarue, Erik & D’haeseleer, William & Helsen, Lieve, 2015. "Integrated modeling of active demand response with electric heating systems coupled to thermal energy storage systems," Applied Energy, Elsevier, vol. 151(C), pages 306-319.
    6. Yang, Mao & Shi, Chaoyu & Liu, Huiyu, 2021. "Day-ahead wind power forecasting based on the clustering of equivalent power curves," Energy, Elsevier, vol. 218(C).
    7. Liu, Zuming & Zhao, Yingru & Wang, Xiaonan, 2020. "Long-term economic planning of combined cooling heating and power systems considering energy storage and demand response," Applied Energy, Elsevier, vol. 279(C).
    8. Dong, Changgui & Qi, Ye & Dong, Wenjuan & Lu, Xi & Liu, Tianle & Qian, Shuai, 2018. "Decomposing driving factors for wind curtailment under economic new normal in China," Applied Energy, Elsevier, vol. 217(C), pages 178-188.
    9. Wu, Jingda & He, Hongwen & Peng, Jiankun & Li, Yuecheng & Li, Zhanjiang, 2018. "Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus," Applied Energy, Elsevier, vol. 222(C), pages 799-811.
    10. Aunedi, Marko & Pantaleo, Antonio Marco & Kuriyan, Kamal & Strbac, Goran & Shah, Nilay, 2020. "Modelling of national and local interactions between heat and electricity networks in low-carbon energy systems," Applied Energy, Elsevier, vol. 276(C).
    11. Li, Xue & Li, Wenming & Zhang, Rufeng & Jiang, Tao & Chen, Houhe & Li, Guoqing, 2020. "Collaborative scheduling and flexibility assessment of integrated electricity and district heating systems utilizing thermal inertia of district heating network and aggregated buildings," Applied Energy, Elsevier, vol. 258(C).
    12. Zhou, Yue & Wang, Chengshan & Wu, Jianzhong & Wang, Jidong & Cheng, Meng & Li, Gen, 2017. "Optimal scheduling of aggregated thermostatically controlled loads with renewable generation in the intraday electricity market," Applied Energy, Elsevier, vol. 188(C), pages 456-465.
    13. Fang, Yujuan & Chen, Laijun & Mei, Shengwei & Wei, Wei & Huang, Shaowei & Liu, Feng, 2019. "Coal or electricity? An evolutionary game approach to investigate fuel choices of urban heat supply systems," Energy, Elsevier, vol. 181(C), pages 107-122.
    14. Eyre, Nick & Baruah, Pranab, 2015. "Uncertainties in future energy demand in UK residential heating," Energy Policy, Elsevier, vol. 87(C), pages 641-653.
    15. Kong, Xiangyu & Kong, Deqian & Yao, Jingtao & Bai, Linquan & Xiao, Jie, 2020. "Online pricing of demand response based on long short-term memory and reinforcement learning," Applied Energy, Elsevier, vol. 271(C).
    16. Romanchenko, Dmytro & Odenberger, Mikael & Göransson, Lisa & Johnsson, Filip, 2017. "Impact of electricity price fluctuations on the operation of district heating systems: A case study of district heating in Göteborg, Sweden," Applied Energy, Elsevier, vol. 204(C), pages 16-30.
    17. Wu, Xiaomin & Cao, Weihua & Wang, Dianhong & Ding, Min & Yu, Liangjun & Nakanishi, Yosuke, 2021. "Demand response model based on improved Pareto optimum considering seasonal electricity prices for Dongfushan Island," Renewable Energy, Elsevier, vol. 164(C), pages 926-936.
    18. Al-Wakeel, Ali & Wu, Jianzhong & Jenkins, Nick, 2017. "k-means based load estimation of domestic smart meter measurements," Applied Energy, Elsevier, vol. 194(C), pages 333-342.
    19. Dorotić, Hrvoje & Ban, Marko & Pukšec, Tomislav & Duić, Neven, 2020. "Impact of wind penetration in electricity markets on optimal power-to-heat capacities in a local district heating system," Renewable and Sustainable Energy Reviews, Elsevier, vol. 132(C).
    20. Tarroja, Brian & Chiang, Felicia & AghaKouchak, Amir & Samuelsen, Scott & Raghavan, Shuba V. & Wei, Max & Sun, Kaiyu & Hong, Tianzhen, 2018. "Translating climate change and heating system electrification impacts on building energy use to future greenhouse gas emissions and electric grid capacity requirements in California," Applied Energy, Elsevier, vol. 225(C), pages 522-534.
    21. Pan, Guangsheng & Gu, Wei & Wu, Zhi & Lu, Yuping & Lu, Shuai, 2019. "Optimal design and operation of multi-energy system with load aggregator considering nodal energy prices," Applied Energy, Elsevier, vol. 239(C), pages 280-295.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Shen, Rendong & Zhong, Shengyuan & Wen, Xin & An, Qingsong & Zheng, Ruifan & Li, Yang & Zhao, Jun, 2022. "Multi-agent deep reinforcement learning optimization framework for building energy system with renewable energy," Applied Energy, Elsevier, vol. 312(C).
    2. Sabarathinam Srinivasan & Suresh Kumarasamy & Zacharias E. Andreadakis & Pedro G. Lind, 2023. "Artificial Intelligence and Mathematical Models of Power Grids Driven by Renewable Energy Sources: A Survey," Energies, MDPI, vol. 16(14), pages 1-56, July.
    3. Yilin Liang & Yuping Hu & Dongjun Luo & Qi Zhu & Qingxuan Chen & Chunmei Wang, 2023. "Distributed Dynamic Pricing Strategy Based on Deep Reinforcement Learning Approach in a Presale Mechanism," Sustainability, MDPI, vol. 15(13), pages 1-20, July.
    4. Xu, Fangyuan & Zhu, Weidong & Wang, Yi Fei & Lai, Chun Sing & Yuan, Haoliang & Zhao, Yujia & Guo, Siming & Fu, Zhengxin, 2022. "A new deregulated demand response scheme for load over-shifting city in regulated power market," Applied Energy, Elsevier, vol. 311(C).
    5. Ma, Siyu & Liu, Hui & Wang, Ni & Huang, Lidong & Goh, Hui Hwang, 2023. "Incentive-based demand response under incomplete information based on the deep deterministic policy gradient," Applied Energy, Elsevier, vol. 351(C).
    6. Meng, Fanlin & Ma, Qian & Liu, Zixu & Zeng, Xiao-Jun, 2023. "Multiple dynamic pricing for demand response with adaptive clustering-based customer segmentation in smart grids," Applied Energy, Elsevier, vol. 333(C).
    7. Gao, Yuan & Matsunami, Yuki & Miyata, Shohei & Akashi, Yasunori, 2022. "Multi-agent reinforcement learning dealing with hybrid action spaces: A case study for off-grid oriented renewable building energy system," Applied Energy, Elsevier, vol. 326(C).
    8. Eduardo J. Salazar & Mauro Jurado & Mauricio E. Samper, 2023. "Reinforcement Learning-Based Pricing and Incentive Strategy for Demand Response in Smart Grids," Energies, MDPI, vol. 16(3), pages 1-33, February.
    9. Xie, Jiahan & Ajagekar, Akshay & You, Fengqi, 2023. "Multi-Agent attention-based deep reinforcement learning for demand response in grid-responsive buildings," Applied Energy, Elsevier, vol. 342(C).
    10. Mahdi Khodayar & Jacob Regan, 2023. "Deep Neural Networks in Power Systems: A Review," Energies, MDPI, vol. 16(12), pages 1-38, June.
    11. Guixing Yang & Haoran Liu & Weiqing Wang & Junru Chen & Shunbo Lei, 2023. "Distributed Optimal Coordination of a Virtual Power Plant with Residential Regenerative Electric Heating Systems," Energies, MDPI, vol. 16(11), pages 1-15, May.
    12. Halid Kaplan & Kambiz Tehrani & Mo Jamshidi, 2021. "A Fault Diagnosis Design Based on Deep Learning Approach for Electric Vehicle Applications," Energies, MDPI, vol. 14(20), pages 1-14, October.
    13. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Fan, Man & Wang, Jia & Kong, Xiangfei & Suo, Hanxiao & Zheng, Wandong & Li, Han, 2023. "Experimental evaluation of the cascaded energy storage radiator for constructing indoor thermal environment in winter," Applied Energy, Elsevier, vol. 332(C).
    2. Pied, Marie & Anjos, Miguel F. & Malhamé, Roland P., 2020. "A flexibility product for electric water heater aggregators on electricity markets," Applied Energy, Elsevier, vol. 280(C).
    3. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    4. Verástegui, Felipe & Lorca, Álvaro & Negrete-Pincetic, Matias & Olivares, Daniel, 2020. "Firewood heat electrification impacts in the Chilean power system," Energy Policy, Elsevier, vol. 144(C).
    5. Harrold, Daniel J.B. & Cao, Jun & Fan, Zhong, 2022. "Data-driven battery operation for energy arbitrage using rainbow deep reinforcement learning," Energy, Elsevier, vol. 238(PC).
    6. Zhu, Ziqing & Hu, Ze & Chan, Ka Wing & Bu, Siqi & Zhou, Bin & Xia, Shiwei, 2023. "Reinforcement learning in deregulated energy market: A comprehensive review," Applied Energy, Elsevier, vol. 329(C).
    7. Biemann, Marco & Scheller, Fabian & Liu, Xiufeng & Huang, Lizhen, 2021. "Experimental evaluation of model-free reinforcement learning algorithms for continuous HVAC control," Applied Energy, Elsevier, vol. 298(C).
    8. Xie, Jiahan & Ajagekar, Akshay & You, Fengqi, 2023. "Multi-Agent attention-based deep reinforcement learning for demand response in grid-responsive buildings," Applied Energy, Elsevier, vol. 342(C).
    9. Charitopoulos, V. & Fajardy, M. & Chyong, C. K. & Reiner, D., 2022. "The case of 100% electrification of domestic heat in Great Britain," Cambridge Working Papers in Economics 2210, Faculty of Economics, University of Cambridge.
    10. Eduardo J. Salazar & Mauro Jurado & Mauricio E. Samper, 2023. "Reinforcement Learning-Based Pricing and Incentive Strategy for Demand Response in Smart Grids," Energies, MDPI, vol. 16(3), pages 1-33, February.
    11. Ramya Kuppusamy & Srete Nikolovski & Yuvaraja Teekaraman, 2023. "Review of Machine Learning Techniques for Power Quality Performance Evaluation in Grid-Connected Systems," Sustainability, MDPI, vol. 15(20), pages 1-29, October.
    12. Carlos Cruz & Esther Palomar & Ignacio Bravo & Alfredo Gardel, 2020. "Cooperative Demand Response Framework for a Smart Community Targeting Renewables: Testbed Implementation and Performance Evaluation," Energies, MDPI, vol. 13(11), pages 1-20, June.
    13. Caputo, Cesare & Cardin, Michel-Alexandre & Ge, Pudong & Teng, Fei & Korre, Anna & Antonio del Rio Chanona, Ehecatl, 2023. "Design and planning of flexible mobile Micro-Grids using Deep Reinforcement Learning," Applied Energy, Elsevier, vol. 335(C).
    14. Eggimann, Sven & Hall, Jim W. & Eyre, Nick, 2019. "A high-resolution spatio-temporal energy demand simulation to explore the potential of heating demand side management with large-scale heat pump diffusion," Applied Energy, Elsevier, vol. 236(C), pages 997-1010.
    15. Sun, Fangyuan & Kong, Xiangyu & Wu, Jianzhong & Gao, Bixuan & Chen, Ke & Lu, Ning, 2022. "DSM pricing method based on A3C and LSTM under cloud-edge environment," Applied Energy, Elsevier, vol. 315(C).
    16. Mohammad Mahdi Forootan & Iman Larki & Rahim Zahedi & Abolfazl Ahmadi, 2022. "Machine Learning and Deep Learning in Energy Systems: A Review," Sustainability, MDPI, vol. 14(8), pages 1-49, April.
    17. Ahmed Abdelaziz & Vitor Santos & Miguel Sales Dias, 2021. "Machine Learning Techniques in the Energy Consumption of Buildings: A Systematic Literature Review Using Text Mining and Bibliometric Analysis," Energies, MDPI, vol. 14(22), pages 1-31, November.
    18. Lee, Zachary E. & Max Zhang, K., 2022. "Unintended consequences of smart thermostats in the transition to electrified heating," Applied Energy, Elsevier, vol. 322(C).
    19. Rocchetta, R. & Bellani, L. & Compare, M. & Zio, E. & Patelli, E., 2019. "A reinforcement learning framework for optimal operation and maintenance of power grids," Applied Energy, Elsevier, vol. 241(C), pages 291-301.
    20. Ahmad, Tanveer & Madonski, Rafal & Zhang, Dongdong & Huang, Chao & Mujeeb, Asad, 2022. "Data-driven probabilistic machine learning in sustainable smart energy/smart energy systems: Key developments, challenges, and future research opportunities in the context of smart grid paradigm," Renewable and Sustainable Energy Reviews, Elsevier, vol. 160(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:288:y:2021:i:c:s0306261921001586. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.