IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v374y2024ics0306261924013333.html
   My bibliography  Save this article

Integrated energy hub dispatch with a multi-mode CAES–BESS hybrid system: An option-based hierarchical reinforcement learning approach

Author

Listed:
  • Cui, Feifei
  • An, Dou
  • Xi, Huan

Abstract

The high penetration of renewable energy sources (RES) in power generation has driven demand for advanced integrated energy management systems (IEMS). In this study, to address the challenges of insufficient adaptability to dynamic supply–demand, a multi-type energy IEMS combining compressed air energy storage (CAES) and a battery energy storage system (BESS) is proposed, which operates under a multi-mode energy storage (MES) mechanism with rapid response, long-term balance, and synergic adjustment modes. To address the complexity of sequential decisions, an option-critic based twin delayed deep deterministic policy gradient (OCTD3) algorithm is firstly proposed within the hierarchical reinforcement learning (HRL) framework, enhancing efficiency through encapsulation of subtasks within ”options”. Additionally, model precision is refined by fitting the electricity–gas–heat conversion dynamics of CAES under off-design conditions. Dispatch tasks are modeled as an option-based Semi-Markov Decision Process (SMDP) and optimized by the OCTD3 to improve the power fluctuations index (PFI), comprehensive costs index (CCI), and system response synergy index (SRSI). Comparative simulations reveal that the MES mechanism boosts SRSI by 91.8%, showcasing high adaptability to varied supply–demand scenarios. The OCTD3 algorithm develops five hybrid strategies for CAES–BESS across three modes, effectively cutting costs by reducing electricity purchases and fluctuations expenses, and lowering PFI by 42.2% through balancing peak–valley loads and swiftly responding to transient shifts.

Suggested Citation

  • Cui, Feifei & An, Dou & Xi, Huan, 2024. "Integrated energy hub dispatch with a multi-mode CAES–BESS hybrid system: An option-based hierarchical reinforcement learning approach," Applied Energy, Elsevier, vol. 374(C).
    • Handle: RePEc:eee:appene:v:374:y:2024:i:c:s0306261924013333
    DOI: 10.1016/j.apenergy.2024.123950
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261924013333
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2024.123950?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Dinh, Huy Truong & Lee, Kyu-haeng & Kim, Daehee, 2022. "Supervised-learning-based hour-ahead demand response for a behavior-based home energy management system approximating MILP optimization," Applied Energy, Elsevier, vol. 321(C).
    2. Rezaeimozafar, Mostafa & Duffy, Maeve & Monaghan, Rory F.D. & Barrett, Enda, 2024. "A hybrid heuristic-reinforcement learning-based real-time control model for residential behind-the-meter PV-battery systems," Applied Energy, Elsevier, vol. 355(C).
    3. Wang, Zixuan & Xiao, Fu & Ran, Yi & Li, Yanxue & Xu, Yang, 2024. "Scalable energy management approach of residential hybrid energy system using multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 367(C).
    4. Wang, Chen & He, Qijiao & Li, Zheng & Yu, Jie & Bello, Idris Temitope & Zheng, Keqing & Han, Minfang & Ni, Meng, 2024. "A novel in-tube reformer for solid oxide fuel cell for performance improvement and efficient thermal management: A numerical study based on artificial neural network and genetic algorithm," Applied Energy, Elsevier, vol. 357(C).
    5. Shang, Wen-Long & Ling, Yantao & Ochieng, Washington & Yang, Linchuan & Gao, Xing & Ren, Qingzhong & Chen, Yilin & Cao, Mengqiu, 2024. "Driving forces of CO2 emissions from the transport, storage and postal sectors: A pathway to achieving carbon neutrality," Applied Energy, Elsevier, vol. 365(C).
    6. Bullich-Massagué, Eduard & Cifuentes-García, Francisco-Javier & Glenny-Crende, Ignacio & Cheah-Mañé, Marc & Aragüés-Peñalba, Mònica & Díaz-González, Francisco & Gomis-Bellmunt, Oriol, 2020. "A review of energy storage technologies for large scale photovoltaic power plants," Applied Energy, Elsevier, vol. 274(C).
    7. Rahim, Sahar & Wang, Zhen & Ju, Ping, 2022. "Overview and applications of Robust optimization in the avant-garde energy grid infrastructure: A systematic review," Applied Energy, Elsevier, vol. 319(C).
    8. Chen, Yuanyi & Hu, Simon & Zheng, Yanchong & Xie, Shiwei & Yang, Qiang & Wang, Yubin & Hu, Qinru, 2024. "Coordinated optimization of logistics scheduling and electricity dispatch for electric logistics vehicles considering uncertain electricity prices and renewable generation," Applied Energy, Elsevier, vol. 364(C).
    9. Yi, Zonggen & Luo, Yusheng & Westover, Tyler & Katikaneni, Sravya & Ponkiya, Binaka & Sah, Suba & Mahmud, Sadab & Raker, David & Javaid, Ahmad & Heben, Michael J. & Khanna, Raghav, 2022. "Deep reinforcement learning based optimization for a tightly coupled nuclear renewable integrated energy system," Applied Energy, Elsevier, vol. 328(C).
    10. Wu, Danman & Bai, Jiayu & Wei, Wei & Chen, Laijun & Mei, Shengwei, 2021. "Optimal bidding and scheduling of AA-CAES based energy hub considering cascaded consumption of heat," Energy, Elsevier, vol. 233(C).
    11. Wang, Hanchen & Ye, Yiming & Zhang, Jiangfeng & Xu, Bin, 2023. "A comparative study of 13 deep reinforcement learning based energy management methods for a hybrid electric vehicle," Energy, Elsevier, vol. 266(C).
    12. Yin, Linfei & Li, Yu, 2022. "Hybrid multi-agent emotional deep Q network for generation control of multi-area integrated energy systems," Applied Energy, Elsevier, vol. 324(C).
    13. Giovanniello, Michael Anthony & Wu, Xiao-Yu, 2023. "Hybrid lithium-ion battery and hydrogen energy storage systems for a wind-supplied microgrid," Applied Energy, Elsevier, vol. 345(C).
    14. Qiu, Dawei & Wang, Yi & Zhang, Tingqi & Sun, Mingyang & Strbac, Goran, 2023. "Hierarchical multi-agent reinforcement learning for repair crews dispatch control towards multi-energy microgrid resilience," Applied Energy, Elsevier, vol. 336(C).
    15. Ma, Siyu & Liu, Hui & Wang, Ni & Huang, Lidong & Goh, Hui Hwang, 2023. "Incentive-based demand response under incomplete information based on the deep deterministic policy gradient," Applied Energy, Elsevier, vol. 351(C).
    16. Wang, Yi & Qiu, Dawei & Sun, Mingyang & Strbac, Goran & Gao, Zhiwei, 2023. "Secure energy management of multi-energy microgrid: A physical-informed safe reinforcement learning approach," Applied Energy, Elsevier, vol. 335(C).
    17. Ren, Kezheng & Liu, Jun & Wu, Zeyang & Liu, Xinglei & Nie, Yongxin & Xu, Haitao, 2024. "A data-driven DRL-based home energy management system optimization framework considering uncertain household parameters," Applied Energy, Elsevier, vol. 355(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Xu, Xuesong & Xu, Kai & Zeng, Ziyang & Tang, Jiale & He, Yuanxing & Shi, Guangze & Zhang, Tao, 2024. "Collaborative optimization of multi-energy multi-microgrid system: A hierarchical trust-region multi-agent reinforcement learning approach," Applied Energy, Elsevier, vol. 375(C).
    2. Li, Sichen & Hu, Weihao & Cao, Di & Chen, Zhe & Huang, Qi & Blaabjerg, Frede & Liao, Kaiji, 2023. "Physics-model-free heat-electricity energy management of multiple microgrids based on surrogate model-enabled multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 346(C).
    3. Dib, Ghady & Haberschill, Philippe & Rullière, Romuald & Revellin, Rémi, 2021. "Modelling small-scale trigenerative advanced adiabatic compressed air energy storage for building application," Energy, Elsevier, vol. 237(C).
    4. Angel Recalde & Ricardo Cajo & Washington Velasquez & Manuel S. Alvarez-Alvarado, 2024. "Machine Learning and Optimization in Energy Management Systems for Plug-In Hybrid Electric Vehicles: A Comprehensive Review," Energies, MDPI, vol. 17(13), pages 1-39, June.
    5. Jiali He & Xiangfei Liu & Xuetong Wang & Xueyang Li & Linger Yu & Beibei Niu, 2024. "Spatiotemporal Evolution of Territorial Spaces and Its Effect on Carbon Emissions in Qingdao City, China," Land, MDPI, vol. 13(10), pages 1-22, October.
    6. Liu, Mingzhe & Guo, Mingyue & Fu, Yangyang & O’Neill, Zheng & Gao, Yuan, 2024. "Expert-guided imitation learning for energy management: Evaluating GAIL’s performance in building control applications," Applied Energy, Elsevier, vol. 372(C).
    7. Zhao, Yincheng & Zhang, Guozhou & Hu, Weihao & Huang, Qi & Chen, Zhe & Blaabjerg, Frede, 2023. "Meta-learning based voltage control strategy for emergency faults of active distribution networks," Applied Energy, Elsevier, vol. 349(C).
    8. Keerthana Sivamayil & Elakkiya Rajasekar & Belqasem Aljafari & Srete Nikolovski & Subramaniyaswamy Vairavasundaram & Indragandhi Vairavasundaram, 2023. "A Systematic Study on Reinforcement Learning Based Applications," Energies, MDPI, vol. 16(3), pages 1-23, February.
    9. Ziółkowski, Paweł & Stasiak, Kamil & Amiri, Milad & Mikielewicz, Dariusz, 2023. "Negative carbon dioxide gas power plant integrated with gasification of sewage sludge," Energy, Elsevier, vol. 262(PB).
    10. Ziqi Liu & Tingting Su & Zhiying Quan & Quanli Wu & Yu Wang, 2023. "Review on the Optimal Configuration of Distributed Energy Storage," Energies, MDPI, vol. 16(14), pages 1-17, July.
    11. Wu, Wenjie & Hou, Hui & Zhu, Shaohua & Liu, Qin & Wei, Ruizeng & He, Huan & Wang, Lei & Luo, Yingting, 2024. "An intelligent power grid emergency allocation technology considering secondary disaster and public opinion under typhoon disaster," Applied Energy, Elsevier, vol. 353(PA).
    12. Pang, Xinfu & Wang, Yibao & Yu, Yang & Liu, Wei, 2024. "Optimal scheduling of a cogeneration system via Q-learning-based memetic algorithm considering demand-side response," Energy, Elsevier, vol. 300(C).
    13. Yang, Zhixue & Ren, Zhouyang & Li, Hui & Sun, Zhiyuan & Feng, Jianbing & Xia, Weiyi, 2024. "A multi-stage stochastic dispatching method for electricity‑hydrogen integrated energy systems driven by model and data," Applied Energy, Elsevier, vol. 371(C).
    14. Ren, Kezheng & Liu, Jun & Wu, Zeyang & Liu, Xinglei & Nie, Yongxin & Xu, Haitao, 2024. "A data-driven DRL-based home energy management system optimization framework considering uncertain household parameters," Applied Energy, Elsevier, vol. 355(C).
    15. Ji, Zhengsen & Li, Wanying & Niu, Dongxiao, 2024. "Optimal investment decision of agrivoltaic coupling energy storage project based on distributed linguistic trust and hybrid evaluation method," Applied Energy, Elsevier, vol. 353(PA).
    16. Shi, Dehua & Xu, Han & Wang, Shaohua & Hu, Jia & Chen, Long & Yin, Chunfang, 2024. "Deep reinforcement learning based adaptive energy management for plug-in hybrid electric vehicle with double deep Q-network," Energy, Elsevier, vol. 305(C).
    17. Kebede, Abraham Alem & Kalogiannis, Theodoros & Van Mierlo, Joeri & Berecibar, Maitane, 2022. "A comprehensive review of stationary energy storage devices for large scale renewable energy sources grid integration," Renewable and Sustainable Energy Reviews, Elsevier, vol. 159(C).
    18. Wu, Haochi & Qiu, Dawei & Zhang, Liyu & Sun, Mingyang, 2024. "Adaptive multi-agent reinforcement learning for flexible resource management in a virtual power plant with dynamic participating multi-energy buildings," Applied Energy, Elsevier, vol. 374(C).
    19. Chen, Qi & Kuang, Zhonghong & Liu, Xiaohua & Zhang, Tao, 2024. "Application-oriented assessment of grid-connected PV-battery system with deep reinforcement learning in buildings considering electricity price dynamics," Applied Energy, Elsevier, vol. 364(C).
    20. Masoumeh Sharifpour & Mohammad Taghi Ameli & Hossein Ameli & Goran Strbac, 2023. "A Resilience-Oriented Approach for Microgrid Energy Management with Hydrogen Integration during Extreme Events," Energies, MDPI, vol. 16(24), pages 1-18, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:374:y:2024:i:c:s0306261924013333. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.