IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v374y2024ics0306261924013333.html
   My bibliography  Save this article

Integrated energy hub dispatch with a multi-mode CAES–BESS hybrid system: An option-based hierarchical reinforcement learning approach

Author

Listed:
  • Cui, Feifei
  • An, Dou
  • Xi, Huan

Abstract

The high penetration of renewable energy sources (RES) in power generation has driven demand for advanced integrated energy management systems (IEMS). In this study, to address the challenges of insufficient adaptability to dynamic supply–demand, a multi-type energy IEMS combining compressed air energy storage (CAES) and a battery energy storage system (BESS) is proposed, which operates under a multi-mode energy storage (MES) mechanism with rapid response, long-term balance, and synergic adjustment modes. To address the complexity of sequential decisions, an option-critic based twin delayed deep deterministic policy gradient (OCTD3) algorithm is firstly proposed within the hierarchical reinforcement learning (HRL) framework, enhancing efficiency through encapsulation of subtasks within ”options”. Additionally, model precision is refined by fitting the electricity–gas–heat conversion dynamics of CAES under off-design conditions. Dispatch tasks are modeled as an option-based Semi-Markov Decision Process (SMDP) and optimized by the OCTD3 to improve the power fluctuations index (PFI), comprehensive costs index (CCI), and system response synergy index (SRSI). Comparative simulations reveal that the MES mechanism boosts SRSI by 91.8%, showcasing high adaptability to varied supply–demand scenarios. The OCTD3 algorithm develops five hybrid strategies for CAES–BESS across three modes, effectively cutting costs by reducing electricity purchases and fluctuations expenses, and lowering PFI by 42.2% through balancing peak–valley loads and swiftly responding to transient shifts.

Suggested Citation

  • Cui, Feifei & An, Dou & Xi, Huan, 2024. "Integrated energy hub dispatch with a multi-mode CAES–BESS hybrid system: An option-based hierarchical reinforcement learning approach," Applied Energy, Elsevier, vol. 374(C).
    • Handle: RePEc:eee:appene:v:374:y:2024:i:c:s0306261924013333
    DOI: 10.1016/j.apenergy.2024.123950
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261924013333
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2024.123950?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Rahim, Sahar & Wang, Zhen & Ju, Ping, 2022. "Overview and applications of Robust optimization in the avant-garde energy grid infrastructure: A systematic review," Applied Energy, Elsevier, vol. 319(C).
    2. Dinh, Huy Truong & Lee, Kyu-haeng & Kim, Daehee, 2022. "Supervised-learning-based hour-ahead demand response for a behavior-based home energy management system approximating MILP optimization," Applied Energy, Elsevier, vol. 321(C).
    3. Chen, Yuanyi & Hu, Simon & Zheng, Yanchong & Xie, Shiwei & Yang, Qiang & Wang, Yubin & Hu, Qinru, 2024. "Coordinated optimization of logistics scheduling and electricity dispatch for electric logistics vehicles considering uncertain electricity prices and renewable generation," Applied Energy, Elsevier, vol. 364(C).
    4. Yi, Zonggen & Luo, Yusheng & Westover, Tyler & Katikaneni, Sravya & Ponkiya, Binaka & Sah, Suba & Mahmud, Sadab & Raker, David & Javaid, Ahmad & Heben, Michael J. & Khanna, Raghav, 2022. "Deep reinforcement learning based optimization for a tightly coupled nuclear renewable integrated energy system," Applied Energy, Elsevier, vol. 328(C).
    5. Wu, Danman & Bai, Jiayu & Wei, Wei & Chen, Laijun & Mei, Shengwei, 2021. "Optimal bidding and scheduling of AA-CAES based energy hub considering cascaded consumption of heat," Energy, Elsevier, vol. 233(C).
    6. Wang, Hanchen & Ye, Yiming & Zhang, Jiangfeng & Xu, Bin, 2023. "A comparative study of 13 deep reinforcement learning based energy management methods for a hybrid electric vehicle," Energy, Elsevier, vol. 266(C).
    7. Rezaeimozafar, Mostafa & Duffy, Maeve & Monaghan, Rory F.D. & Barrett, Enda, 2024. "A hybrid heuristic-reinforcement learning-based real-time control model for residential behind-the-meter PV-battery systems," Applied Energy, Elsevier, vol. 355(C).
    8. Wang, Zixuan & Xiao, Fu & Ran, Yi & Li, Yanxue & Xu, Yang, 2024. "Scalable energy management approach of residential hybrid energy system using multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 367(C).
    9. Yin, Linfei & Li, Yu, 2022. "Hybrid multi-agent emotional deep Q network for generation control of multi-area integrated energy systems," Applied Energy, Elsevier, vol. 324(C).
    10. Wang, Chen & He, Qijiao & Li, Zheng & Yu, Jie & Bello, Idris Temitope & Zheng, Keqing & Han, Minfang & Ni, Meng, 2024. "A novel in-tube reformer for solid oxide fuel cell for performance improvement and efficient thermal management: A numerical study based on artificial neural network and genetic algorithm," Applied Energy, Elsevier, vol. 357(C).
    11. Giovanniello, Michael Anthony & Wu, Xiao-Yu, 2023. "Hybrid lithium-ion battery and hydrogen energy storage systems for a wind-supplied microgrid," Applied Energy, Elsevier, vol. 345(C).
    12. Qiu, Dawei & Wang, Yi & Zhang, Tingqi & Sun, Mingyang & Strbac, Goran, 2023. "Hierarchical multi-agent reinforcement learning for repair crews dispatch control towards multi-energy microgrid resilience," Applied Energy, Elsevier, vol. 336(C).
    13. Ma, Siyu & Liu, Hui & Wang, Ni & Huang, Lidong & Goh, Hui Hwang, 2023. "Incentive-based demand response under incomplete information based on the deep deterministic policy gradient," Applied Energy, Elsevier, vol. 351(C).
    14. Shang, Wen-Long & Ling, Yantao & Ochieng, Washington & Yang, Linchuan & Gao, Xing & Ren, Qingzhong & Chen, Yilin & Cao, Mengqiu, 2024. "Driving forces of CO2 emissions from the transport, storage and postal sectors: A pathway to achieving carbon neutrality," Applied Energy, Elsevier, vol. 365(C).
    15. Wang, Yi & Qiu, Dawei & Sun, Mingyang & Strbac, Goran & Gao, Zhiwei, 2023. "Secure energy management of multi-energy microgrid: A physical-informed safe reinforcement learning approach," Applied Energy, Elsevier, vol. 335(C).
    16. Bullich-Massagué, Eduard & Cifuentes-García, Francisco-Javier & Glenny-Crende, Ignacio & Cheah-Mañé, Marc & Aragüés-Peñalba, Mònica & Díaz-González, Francisco & Gomis-Bellmunt, Oriol, 2020. "A review of energy storage technologies for large scale photovoltaic power plants," Applied Energy, Elsevier, vol. 274(C).
    17. Ren, Kezheng & Liu, Jun & Wu, Zeyang & Liu, Xinglei & Nie, Yongxin & Xu, Haitao, 2024. "A data-driven DRL-based home energy management system optimization framework considering uncertain household parameters," Applied Energy, Elsevier, vol. 355(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Xu, Xuesong & Xu, Kai & Zeng, Ziyang & Tang, Jiale & He, Yuanxing & Shi, Guangze & Zhang, Tao, 2024. "Collaborative optimization of multi-energy multi-microgrid system: A hierarchical trust-region multi-agent reinforcement learning approach," Applied Energy, Elsevier, vol. 375(C).
    2. Li, Sichen & Hu, Weihao & Cao, Di & Chen, Zhe & Huang, Qi & Blaabjerg, Frede & Liao, Kaiji, 2023. "Physics-model-free heat-electricity energy management of multiple microgrids based on surrogate model-enabled multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 346(C).
    3. Nawaz Edoo & Robert T. F. Ah King, 2021. "Techno-Economic Analysis of Utility-Scale Solar Photovoltaic Plus Battery Power Plant," Energies, MDPI, vol. 14(23), pages 1-22, December.
    4. Qiu, Rui & Zhang, Haoran & Wang, Guotao & Liang, Yongtu & Yan, Jinyue, 2023. "Green hydrogen-based energy storage service via power-to-gas technologies integrated with multi-energy microgrid," Applied Energy, Elsevier, vol. 350(C).
    5. Dib, Ghady & Haberschill, Philippe & Rullière, Romuald & Revellin, Rémi, 2021. "Modelling small-scale trigenerative advanced adiabatic compressed air energy storage for building application," Energy, Elsevier, vol. 237(C).
    6. Angel Recalde & Ricardo Cajo & Washington Velasquez & Manuel S. Alvarez-Alvarado, 2024. "Machine Learning and Optimization in Energy Management Systems for Plug-In Hybrid Electric Vehicles: A Comprehensive Review," Energies, MDPI, vol. 17(13), pages 1-39, June.
    7. Omar A. Beg & Asad Ali Khan & Waqas Ur Rehman & Ali Hassan, 2023. "A Review of AI-Based Cyber-Attack Detection and Mitigation in Microgrids," Energies, MDPI, vol. 16(22), pages 1-23, November.
    8. Jiali He & Xiangfei Liu & Xuetong Wang & Xueyang Li & Linger Yu & Beibei Niu, 2024. "Spatiotemporal Evolution of Territorial Spaces and Its Effect on Carbon Emissions in Qingdao City, China," Land, MDPI, vol. 13(10), pages 1-22, October.
    9. Angel L. Cedeño & Reinier López Ahuar & José Rojas & Gonzalo Carvajal & César Silva & Juan C. Agüero, 2022. "Model Predictive Control for Photovoltaic Plants with Non-Ideal Energy Storage Using Mixed Integer Linear Programming," Energies, MDPI, vol. 15(17), pages 1-21, September.
    10. Shen, Boyang & Chen, Yu & Li, Chuanyue & Wang, Sheng & Chen, Xiaoyuan, 2021. "Superconducting fault current limiter (SFCL): Experiment and the simulation from finite-element method (FEM) to power/energy system software," Energy, Elsevier, vol. 234(C).
    11. Liu, Mingzhe & Guo, Mingyue & Fu, Yangyang & O’Neill, Zheng & Gao, Yuan, 2024. "Expert-guided imitation learning for energy management: Evaluating GAIL’s performance in building control applications," Applied Energy, Elsevier, vol. 372(C).
    12. Zhao, Yincheng & Zhang, Guozhou & Hu, Weihao & Huang, Qi & Chen, Zhe & Blaabjerg, Frede, 2023. "Meta-learning based voltage control strategy for emergency faults of active distribution networks," Applied Energy, Elsevier, vol. 349(C).
    13. Dudu Guo & Yinuo Su & Xiaojiang Zhang & Zhen Yang & Pengbin Duan, 2024. "Multi-Objective Optimization of Short-Inverted Transport Scheduling Strategy Based on Road–Railway Intermodal Transport," Sustainability, MDPI, vol. 16(15), pages 1-25, July.
    14. Zhu, Dafeng & Yang, Bo & Ma, Chengbin & Wang, Zhaojian & Zhu, Shanying & Ma, Kai & Guan, Xinping, 2022. "Stochastic gradient-based fast distributed multi-energy management for an industrial park with temporally-coupled constraints," Applied Energy, Elsevier, vol. 317(C).
    15. Pla, Benjamín & Bares, Pau & Aronis, André Nakaema & Anuratha, Sanjith, 2024. "Leveraging battery electric vehicle energy storage potential for home energy saving by model predictive control with backward induction," Applied Energy, Elsevier, vol. 372(C).
    16. Ting Zhang & Shuaishuai Cao & Lingying Pan & Chenyu Zhou, 2020. "A Policy Effect Analysis of China’s Energy Storage Development Based on a Multi-Agent Evolutionary Game Model," Energies, MDPI, vol. 13(23), pages 1-35, November.
    17. Keerthana Sivamayil & Elakkiya Rajasekar & Belqasem Aljafari & Srete Nikolovski & Subramaniyaswamy Vairavasundaram & Indragandhi Vairavasundaram, 2023. "A Systematic Study on Reinforcement Learning Based Applications," Energies, MDPI, vol. 16(3), pages 1-23, February.
    18. Ziółkowski, Paweł & Stasiak, Kamil & Amiri, Milad & Mikielewicz, Dariusz, 2023. "Negative carbon dioxide gas power plant integrated with gasification of sewage sludge," Energy, Elsevier, vol. 262(PB).
    19. Ziqi Liu & Tingting Su & Zhiying Quan & Quanli Wu & Yu Wang, 2023. "Review on the Optimal Configuration of Distributed Energy Storage," Energies, MDPI, vol. 16(14), pages 1-17, July.
    20. Tan, Yingqi & Xu, Jingyi & Ma, Junyi & Li, Zirui & Chen, Huiyan & Xi, Junqiang & Liu, Haiou, 2024. "A transferable perception-guided EMS for series hybrid electric unmanned tracked vehicles," Energy, Elsevier, vol. 306(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:374:y:2024:i:c:s0306261924013333. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.