IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v336y2023ics0306261923001903.html
   My bibliography  Save this article

Hierarchical multi-agent reinforcement learning for repair crews dispatch control towards multi-energy microgrid resilience

Author

Listed:
  • Qiu, Dawei
  • Wang, Yi
  • Zhang, Tingqi
  • Sun, Mingyang
  • Strbac, Goran

Abstract

Extreme events are greatly impacting the normal operations of microgrids, which can lead to severe outages and affect the continuous supply of energy to customers, incurring substantial restoration costs. Repair crews (RCs) are regarded as crucial resources to provide system resilience owing to their mobility and flexibility characteristics in handling both transportation and energy systems. Nevertheless, effectively coordinating the dispatch of RCs towards system resilience is a complex decision-making problem, especially in the context of a multi-energy microgrid (MEMG) with enormous dynamics and uncertainties. To this end, this paper formulates the dispatch problem of RCs in a coupled transportation and power-gas network as a decentralized partially observable Markov decision process (Dec-POMDP). To solve this Dec-POMDP, a hierarchical multi-agent reinforcement learning (MARL) algorithm is proposed by featuring a two-level framework, where the high-level action is used for switching decision-making between transportation and power-gas networks, and the lower-level action constructed via the multi-agent proximal policy optimization (MAPPO) algorithm is used to compute the routing and repairing decisions of RCs in the transportation and power-gas networks, respectively. The proposed algorithm also introduces an abstracted critic network by integrating the load restoration status, which captures the system dynamics and stabilizes the training performance with privacy protection. Extensive case studies are evaluated on a coupled 6-bus power and 6-bus gas network integrated with a 9-node 12-edge transportation network. The proposed algorithm outperforms the conventional MARL algorithms in terms of policy quality, learning stability, and computational performance. Furthermore, the dispatch strategies of RCs are analyzed and their corresponding benefits for load restoration are also evaluated. Finally, the scalability of the proposed method is also investigated for a larger 33-bus power and 15-bus gas network integrated with an 18-node 27-edge transportation network.

Suggested Citation

  • Qiu, Dawei & Wang, Yi & Zhang, Tingqi & Sun, Mingyang & Strbac, Goran, 2023. "Hierarchical multi-agent reinforcement learning for repair crews dispatch control towards multi-energy microgrid resilience," Applied Energy, Elsevier, vol. 336(C).
    • Handle: RePEc:eee:appene:v:336:y:2023:i:c:s0306261923001903
    DOI: 10.1016/j.apenergy.2023.120826
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261923001903
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2023.120826?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Zhou, Quan & Li, Yanfei & Zhao, Dezong & Li, Ji & Williams, Huw & Xu, Hongming & Yan, Fuwu, 2022. "Transferable representation modelling for real-time energy management of the plug-in hybrid vehicle based on k-fold fuzzy learning and Gaussian process regression," Applied Energy, Elsevier, vol. 305(C).
    2. Perera, A.T.D. & Kamalaruban, Parameswaran, 2021. "Applications of reinforcement learning in energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 137(C).
    3. Wang, Yi & Qiu, Dawei & Strbac, Goran, 2022. "Multi-agent deep reinforcement learning for resilience-driven routing and scheduling of mobile energy storage systems," Applied Energy, Elsevier, vol. 310(C).
    4. Sang, Maosheng & Ding, Yi & Bao, Minglei & Li, Siying & Ye, Chengjin & Fang, Youtong, 2021. "Resilience-based restoration strategy optimization for interdependent gas and power networks," Applied Energy, Elsevier, vol. 302(C).
    5. Karasu, Seçkin & Altan, Aytaç & Bekiros, Stelios & Ahmad, Wasim, 2020. "A new forecasting model with wrapper-based feature selection approach using multi-objective optimization technique for chaotic crude oil time series," Energy, Elsevier, vol. 212(C).
    6. Wu, Raphael & Sansavini, Giovanni, 2020. "Integrating reliability and resilience to support the transition from passive distribution grids to islanding microgrids," Applied Energy, Elsevier, vol. 272(C).
    7. Sayed, Ahmed R. & Wang, Cheng & Bi, Tianshu, 2019. "Resilient operational strategies for power systems considering the interactions with natural gas systems," Applied Energy, Elsevier, vol. 241(C), pages 548-566.
    8. Hussain, Akhtar & Bui, Van-Hai & Kim, Hak-Man, 2019. "Microgrids as a resilience resource and strategies used by microgrids for enhancing resilience," Applied Energy, Elsevier, vol. 240(C), pages 56-72.
    9. Karasu, Seçkin & Altan, Aytaç, 2022. "Crude oil time series prediction model based on LSTM network with chaotic Henry gas solubility optimization," Energy, Elsevier, vol. 242(C).
    10. Dehghani, Nariman L. & Jeddi, Ashkan B. & Shafieezadeh, Abdollah, 2021. "Intelligent hurricane resilience enhancement of power distribution systems via deep reinforcement learning," Applied Energy, Elsevier, vol. 285(C).
    11. Mishra, Sakshi & Anderson, Kate & Miller, Brian & Boyer, Kyle & Warren, Adam, 2020. "Microgrid resilience: A holistic approach for assessing threats, identifying vulnerabilities, and designing corresponding mitigation strategies," Applied Energy, Elsevier, vol. 264(C).
    12. Wu, Tao & Wang, Jianhui & Lu, Xiaonan & Du, Yuhua, 2022. "AC/DC hybrid distribution network reconfiguration with microgrid formation using multi-agent soft actor-critic," Applied Energy, Elsevier, vol. 307(C).
    13. Wang, Yi & Rousis, Anastasios Oulis & Strbac, Goran, 2020. "On microgrids and resilience: A comprehensive review on modeling and operational strategies," Renewable and Sustainable Energy Reviews, Elsevier, vol. 134(C).
    14. Bao, Minglei & Ding, Yi & Sang, Maosheng & Li, Daqing & Shao, Changzheng & Yan, Jinyue, 2020. "Modeling and evaluating nodal resilience of multi-energy systems under windstorms," Applied Energy, Elsevier, vol. 270(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Wu, Wenjie & Hou, Hui & Zhu, Shaohua & Liu, Qin & Wei, Ruizeng & He, Huan & Wang, Lei & Luo, Yingting, 2024. "An intelligent power grid emergency allocation technology considering secondary disaster and public opinion under typhoon disaster," Applied Energy, Elsevier, vol. 353(PA).
    2. Li, Sichen & Hu, Weihao & Cao, Di & Chen, Zhe & Huang, Qi & Blaabjerg, Frede & Liao, Kaiji, 2023. "Physics-model-free heat-electricity energy management of multiple microgrids based on surrogate model-enabled multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 346(C).
    3. Dong, Lei & Sun, Shiting & Zhang, Shiming & Zhang, Tao & Pu, Tianjiao, 2024. "Distributed restoration for integrated electricity-gas-heating energy systems with an iterative loop scheme," Energy, Elsevier, vol. 304(C).
    4. Cui, Feifei & An, Dou & Xi, Huan, 2024. "Integrated energy hub dispatch with a multi-mode CAES–BESS hybrid system: An option-based hierarchical reinforcement learning approach," Applied Energy, Elsevier, vol. 374(C).
    5. Wang, Jingyao & Li, Yao & Bian, Jiayu & Yu, Zhiyong & Zhang, Min & Wang, Cheng & Bi, Tianshu, 2023. "Multi-stage resilient operation strategy of urban electric–gas system against rainstorms," Applied Energy, Elsevier, vol. 348(C).
    6. Xu, Xuesong & Xu, Kai & Zeng, Ziyang & Tang, Jiale & He, Yuanxing & Shi, Guangze & Zhang, Tao, 2024. "Collaborative optimization of multi-energy multi-microgrid system: A hierarchical trust-region multi-agent reinforcement learning approach," Applied Energy, Elsevier, vol. 375(C).
    7. Yang, Sen & Zhang, Yi & Lu, Xinzheng & Guo, Wei & Miao, Huiquan, 2024. "Multi-agent deep reinforcement learning based decision support model for resilient community post-hazard recovery," Reliability Engineering and System Safety, Elsevier, vol. 242(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Liu, Hanchen & Wang, Chong & Ju, Ping & Li, Hongyu, 2022. "A sequentially preventive model enhancing power system resilience against extreme-weather-triggered failures," Renewable and Sustainable Energy Reviews, Elsevier, vol. 156(C).
    2. Shang, Ce & Lin, Teng & Li, Canbing & Wang, Keyou & Ai, Qian, 2021. "Joining resilience and reliability evaluation against both weather and ageing causes," Renewable and Sustainable Energy Reviews, Elsevier, vol. 152(C).
    3. Wang, Y. & Rousis, A. Oulis & Strbac, G., 2022. "Resilience-driven optimal sizing and pre-positioning of mobile energy storage systems in decentralized networked microgrids," Applied Energy, Elsevier, vol. 305(C).
    4. Wang, Yi & Qiu, Dawei & Strbac, Goran, 2022. "Multi-agent deep reinforcement learning for resilience-driven routing and scheduling of mobile energy storage systems," Applied Energy, Elsevier, vol. 310(C).
    5. Qiu, Dawei & Wang, Yi & Hua, Weiqi & Strbac, Goran, 2023. "Reinforcement learning for electric vehicle applications in power systems:A critical review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 173(C).
    6. Dui, Hongyan & Liu, Meng & Song, Jiaying & Wu, Shaomin, 2023. "Importance measure-based resilience management: Review, methodology and perspectives on maintenance," Reliability Engineering and System Safety, Elsevier, vol. 237(C).
    7. Gonzalez-Reina, Antonio Enrique & Garcia-Torres, Felix & Girona-Garcia, Victor & Sanchez-Sanchez-de-Puerta, Alvaro & Jimenez-Romero, F.J. & Jimenez-Hornero, Jorge E., 2024. "Cooperative model predictive control for avoiding critical instants of energy resilience in networked microgrids," Applied Energy, Elsevier, vol. 369(C).
    8. Sun, Qirun & Wu, Zhi & Ma, Zhoujun & Gu, Wei & Zhang, Xiao-Ping & Lu, Yuping & Liu, Pengxiang, 2022. "Resilience enhancement strategy for multi-energy systems considering multi-stage recovery process and multi-energy coordination," Energy, Elsevier, vol. 241(C).
    9. Xu, Jiuping & Tian, Yalou & Wang, Fengjuan & Yang, Guocan & Zhao, Chuandang, 2024. "Resilience-economy-environment equilibrium based configuration interaction approach towards distributed energy system in energy intensive industry parks," Renewable and Sustainable Energy Reviews, Elsevier, vol. 191(C).
    10. Qiu, Dawei & Wang, Yi & Sun, Mingyang & Strbac, Goran, 2022. "Multi-service provision for electric vehicles in power-transportation networks towards a low-carbon transition: A hierarchical and hybrid multi-agent reinforcement learning approach," Applied Energy, Elsevier, vol. 313(C).
    11. Wang, Chong & Ju, Ping & Wu, Feng & Pan, Xueping & Wang, Zhaoyu, 2022. "A systematic review on power system resilience from the perspective of generation, network, and load," Renewable and Sustainable Energy Reviews, Elsevier, vol. 167(C).
    12. Lei, Shunbo & Pozo, David & Wang, Ming-Hao & Li, Qifeng & Li, Yupeng & Peng, Chaoyi, 2022. "Power economic dispatch against extreme weather conditions: The price of resilience," Renewable and Sustainable Energy Reviews, Elsevier, vol. 157(C).
    13. Tobajas, Javier & Garcia-Torres, Felix & Roncero-Sánchez, Pedro & Vázquez, Javier & Bellatreche, Ladjel & Nieto, Emilio, 2022. "Resilience-oriented schedule of microgrids with hybrid energy storage system using model predictive control," Applied Energy, Elsevier, vol. 306(PB).
    14. Wang, Zekai & Ding, Tao & Jia, Wenhao & Huang, Can & Mu, Chenggang & Qu, Ming & Shahidehpour, Mohammad & Yang, Yongheng & Blaabjerg, Frede & Li, Li & Wang, Kang & Chi, Fangde, 2022. "Multi-stage stochastic programming for resilient integrated electricity and natural gas distribution systems against typhoon natural disaster attacks," Renewable and Sustainable Energy Reviews, Elsevier, vol. 159(C).
    15. Li, Xue & Du, Xiaoxue & Jiang, Tao & Zhang, Rufeng & Chen, Houhe, 2022. "Coordinating multi-energy to improve urban integrated energy system resilience against extreme weather events," Applied Energy, Elsevier, vol. 309(C).
    16. Wang, Zhaoqi & Zhang, Lu & Tang, Wei & Chen, Ying & Shen, Chen, 2022. "Equilibrium allocation strategy of multiple ESSs considering the economics and restoration capability in DNs," Applied Energy, Elsevier, vol. 306(PA).
    17. Zhang, Dongdong & Li, Chunjiao & Goh, Hui Hwang & Ahmad, Tanveer & Zhu, Hongyu & Liu, Hui & Wu, Thomas, 2022. "A comprehensive overview of modeling approaches and optimal control strategies for cyber-physical resilience in power systems," Renewable Energy, Elsevier, vol. 189(C), pages 1383-1406.
    18. Younes Zahraoui & Tarmo Korõtko & Argo Rosin & Saad Mekhilef & Mehdi Seyedmahmoudian & Alex Stojcevski & Ibrahim Alhamrouni, 2024. "AI Applications to Enhance Resilience in Power Systems and Microgrids—A Review," Sustainability, MDPI, vol. 16(12), pages 1-35, June.
    19. Antoine Boche & Clément Foucher & Luiz Fernando Lavado Villa, 2022. "Understanding Microgrid Sustainability: A Systemic and Comprehensive Review," Energies, MDPI, vol. 15(8), pages 1-29, April.
    20. Huang, Chunjun & Zong, Yi & You, Shi & Træholt, Chresten & Zheng, Yi & Wang, Jiawei & Zheng, Zixuan & Xiao, Xianyong, 2023. "Economic and resilient operation of hydrogen-based microgrids: An improved MPC-based optimal scheduling scheme considering security constraints of hydrogen facilities," Applied Energy, Elsevier, vol. 335(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:336:y:2023:i:c:s0306261923001903. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.