IDEAS home Printed from https://ideas.repec.org/a/gam/jsusta/v17y2025i2p407-d1562054.html
   My bibliography  Save this article

Deep Reinforcement Learning-Based Real-Time Energy Management for an Integrated Electric–Thermal Energy System

Author

Listed:
  • Qiang Shuai

    (College of Electrical Engineering, Sichuan University, Chengdu 610065, China)

  • Yue Yin

    (College of Electrical Engineering, Sichuan University, Chengdu 610065, China)

  • Shan Huang

    (College of Electrical Engineering, Sichuan University, Chengdu 610065, China)

  • Chao Chen

    (College of Electrical Engineering, Sichuan University, Chengdu 610065, China)

Abstract

Renewable energy plays a crucial role in achieving sustainable development and has the potential to meet humanity’s long-term energy requirements. Integrated electric–thermal energy systems are an important way to consume a high proportion of renewable energy. The intermittency and volatility of integrated electric–thermal energy systems make solving energy management optimization problems difficult. Thus, this paper proposes an energy management optimization method for an integrated electric–thermal energy system based on the improved proximal policy optimization algorithm, which effectively mitigates the problems of the traditional heuristic algorithms or mathematical planning methods with low accuracy and low solving efficiency. Meanwhile, the proposed algorithm enhances both the convergence speed and overall performance compared to the proximal policy optimization algorithm. This paper first establishes a mathematical model for the energy management of an integrated electric–thermal energy system. Then, the model is formulated as a Markov decision process, and a reward mechanism is designed to guide the agent to learn the uncertainty characteristics of renewable energy output and load consumption in the system through historical data. Finally, in the case study section, the proposed algorithm reduces the average running cost by 2.32% compared to the other algorithms discussed in this paper, thereby demonstrating its effectiveness and cost-efficiency.

Suggested Citation

  • Qiang Shuai & Yue Yin & Shan Huang & Chao Chen, 2025. "Deep Reinforcement Learning-Based Real-Time Energy Management for an Integrated Electric–Thermal Energy System," Sustainability, MDPI, vol. 17(2), pages 1-17, January.
    • Handle: RePEc:gam:jsusta:v:17:y:2025:i:2:p:407-:d:1562054
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2071-1050/17/2/407/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2071-1050/17/2/407/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Yang, Jun & Su, Changqi, 2021. "Robust optimization of microgrid based on renewable distributed power generation and load demand uncertainty," Energy, Elsevier, vol. 223(C).
    2. Wei, Hongqian & Zhang, Youtong & Wang, Yongzhen & Hua, Weiqi & Jing, Rui & Zhou, Yue, 2022. "Planning integrated energy systems coupling V2G as a flexible storage," Energy, Elsevier, vol. 239(PB).
    3. Hua, Haochen & Qin, Yuchao & Hao, Chuantong & Cao, Junwei, 2019. "Optimal energy management strategies for energy Internet via deep reinforcement learning approach," Applied Energy, Elsevier, vol. 239(C), pages 598-609.
    4. Jianhong Hao & Ting Huang & Qiuming Xu & Yi Sun, 2023. "Robust Optimal Scheduling of Microgrid with Electric Vehicles Based on Stackelberg Game," Sustainability, MDPI, vol. 15(24), pages 1-15, December.
    5. Chen, Maozhi & Lu, Hao & Chang, Xiqiang & Liao, Haiyan, 2023. "An optimization on an integrated energy system of combined heat and power, carbon capture system and power to gas by considering flexible load," Energy, Elsevier, vol. 273(C).
    6. Ding, Bing & Li, Zening & Li, Zhengmao & Xue, Yixun & Chang, Xinyue & Su, Jia & Jin, Xiaolong & Sun, Hongbin, 2024. "A CCP-based distributed cooperative operation strategy for multi-agent energy systems integrated with wind, solar, and buildings," Applied Energy, Elsevier, vol. 365(C).
    7. Dey, Bishwajit & Misra, Srikant & Garcia Marquez, Fausto Pedro, 2023. "Microgrid system energy management with demand response program for clean and economical operation," Applied Energy, Elsevier, vol. 334(C).
    8. Pinciroli, Luca & Baraldi, Piero & Compare, Michele & Zio, Enrico, 2023. "Optimal operation and maintenance of energy storage systems in grid-connected microgrids by deep reinforcement learning," Applied Energy, Elsevier, vol. 352(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zhou, Yanting & Ma, Zhongjing & Zhang, Jinhui & Zou, Suli, 2022. "Data-driven stochastic energy management of multi energy system using deep reinforcement learning," Energy, Elsevier, vol. 261(PA).
    2. Mohd Bilal & Pitshou N. Bokoro & Gulshan Sharma & Giovanni Pau, 2024. "A Cost-Effective Energy Management Approach for On-Grid Charging of Plug-in Electric Vehicles Integrated with Hybrid Renewable Energy Sources," Energies, MDPI, vol. 17(16), pages 1-35, August.
    3. Zhang, Chengquan & Kitamura, Hiroshi & Goto, Mika, 2024. "Feasibility of vehicle-to-grid (V2G) implementation in Japan: A regional analysis of the electricity supply and demand adjustment market," Energy, Elsevier, vol. 311(C).
    4. Østergaard, P.A. & Lund, H. & Thellufsen, J.Z. & Sorknæs, P. & Mathiesen, B.V., 2022. "Review and validation of EnergyPLAN," Renewable and Sustainable Energy Reviews, Elsevier, vol. 168(C).
    5. Sulman Shahzad & Muhammad Abbas Abbasi & Hassan Ali & Muhammad Iqbal & Rania Munir & Heybet Kilic, 2023. "Possibilities, Challenges, and Future Opportunities of Microgrids: A Review," Sustainability, MDPI, vol. 15(8), pages 1-28, April.
    6. Gao, Yang & Ai, Qian & He, Xing & Fan, Songli, 2023. "Coordination for regional integrated energy system through target cascade optimization," Energy, Elsevier, vol. 276(C).
    7. Alain Aoun & Mehdi Adda & Adrian Ilinca & Mazen Ghandour & Hussein Ibrahim, 2024. "Optimizing Virtual Power Plant Management: A Novel MILP Algorithm to Minimize Levelized Cost of Energy, Technical Losses, and Greenhouse Gas Emissions," Energies, MDPI, vol. 17(16), pages 1-23, August.
    8. Zhu, Jiaoyiling & Hu, Weihao & Xu, Xiao & Liu, Haoming & Pan, Li & Fan, Haoyang & Zhang, Zhenyuan & Chen, Zhe, 2022. "Optimal scheduling of a wind energy dominated distribution network via a deep reinforcement learning approach," Renewable Energy, Elsevier, vol. 201(P1), pages 792-801.
    9. Esraa M. Abd Elsadek & Hossam Kotb & Ayman Samy Abdel-Khalik & Yasser Aboelmagd & Aly. H. Abdelbaky Elbatran, 2024. "Experimental and Techno-Economic Analysis of Solar PV System for Sustainable Building and Greenhouse Gas Emission Mitigation in Harsh Climate: A Case Study of Aswan Educational Building," Sustainability, MDPI, vol. 16(13), pages 1-31, June.
    10. Zhang, Yijie & Ma, Tao & Elia Campana, Pietro & Yamaguchi, Yohei & Dai, Yanjun, 2020. "A techno-economic sizing method for grid-connected household photovoltaic battery systems," Applied Energy, Elsevier, vol. 269(C).
    11. Anjie Lu & Jianguo Zhou & Minglei Qin & Danchen Liu, 2024. "Considering Carbon–Hydrogen Coupled Integrated Energy Systems: A Pathway to Sustainable Energy Transition in China Under Uncertainty," Sustainability, MDPI, vol. 16(21), pages 1-32, October.
    12. Ahmad, Tanveer & Chen, Huanxin, 2019. "Deep learning for multi-scale smart energy forecasting," Energy, Elsevier, vol. 175(C), pages 98-112.
    13. Feng, Jie & Ran, Lun & Wang, Zhiyuan & Zhang, Mengling, 2024. "Optimal energy scheduling of virtual power plant integrating electric vehicles and energy storage systems under uncertainty," Energy, Elsevier, vol. 309(C).
    14. Zeyue Sun & Mohsen Eskandari & Chaoran Zheng & Ming Li, 2022. "Handling Computation Hardness and Time Complexity Issue of Battery Energy Storage Scheduling in Microgrids by Deep Reinforcement Learning," Energies, MDPI, vol. 16(1), pages 1-20, December.
    15. Kandasamy, Jeevitha & Ramachandran, Rajeswari & Veerasamy, Veerapandiyan & Irudayaraj, Andrew Xavier Raj, 2024. "Distributed leader-follower based adaptive consensus control for networked microgrids," Applied Energy, Elsevier, vol. 353(PA).
    16. Tithy Dev & Morteza Haghiri & Gabriela Sabau, 2024. "Impacts of Urbanization on Energy Consumption in the South Asian Association for Regional Cooperation Zone," Sustainability, MDPI, vol. 16(18), pages 1-15, September.
    17. Madani, Seyyedreza & Pineau, Pierre-Olivier, 2024. "Investment in vehicle-to-grid and distributed energy resources: Distributor versus prosumer perspectives and the impact of rate structures," Utilities Policy, Elsevier, vol. 88(C).
    18. Fathy, Ahmed, 2023. "Bald eagle search optimizer-based energy management strategy for microgrid with renewable sources and electric vehicles," Applied Energy, Elsevier, vol. 334(C).
    19. Wu, Yuankai & Tan, Huachun & Peng, Jiankun & Zhang, Hailong & He, Hongwen, 2019. "Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus," Applied Energy, Elsevier, vol. 247(C), pages 454-466.
    20. Yap, Kah Yung & Chin, Hon Huin & Klemeš, Jiří Jaromír, 2022. "Solar Energy-Powered Battery Electric Vehicle charging stations: Current development and future prospect review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 169(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jsusta:v:17:y:2025:i:2:p:407-:d:1562054. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.