IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v16y2023i7p3248-d1116390.html
   My bibliography  Save this article

Multi-Microgrid Collaborative Optimization Scheduling Using an Improved Multi-Agent Soft Actor-Critic Algorithm

Author

Listed:
  • Jiankai Gao

    (School of Electrical Engineering, Northeast Electric Power University, Jilin 132012, China)

  • Yang Li

    (School of Electrical Engineering, Northeast Electric Power University, Jilin 132012, China)

  • Bin Wang

    (State Grid Jining Power Supply Company, Jining 272000, China)

  • Haibo Wu

    (School of Electrical Engineering, Northeast Electric Power University, Jilin 132012, China)

Abstract

The implementation of a multi-microgrid (MMG) system with multiple renewable energy sources enables the facilitation of electricity trading. To tackle the energy management problem of an MMG system, which consists of multiple renewable energy microgrids belonging to different operating entities, this paper proposes an MMG collaborative optimization scheduling model based on a multi-agent centralized training distributed execution framework. To enhance the generalization ability of dealing with various uncertainties, we also propose an improved multi-agent soft actor-critic (MASAC) algorithm, which facilitates energy transactions between multi-agents in MMG, and employs automated machine learning (AutoML) to optimize the MASAC hyperparameters to further improve the generalization of deep reinforcement learning (DRL). The test results demonstrate that the proposed method successfully achieves power complementarity between different entities and reduces the MMG system’s operating cost. Additionally, the proposal significantly outperforms other state-of-the-art reinforcement learning algorithms with better economy and higher calculation efficiency.

Suggested Citation

  • Jiankai Gao & Yang Li & Bin Wang & Haibo Wu, 2023. "Multi-Microgrid Collaborative Optimization Scheduling Using an Improved Multi-Agent Soft Actor-Critic Algorithm," Energies, MDPI, vol. 16(7), pages 1-21, April.
    • Handle: RePEc:gam:jeners:v:16:y:2023:i:7:p:3248-:d:1116390
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/16/7/3248/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1996-1073/16/7/3248/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Gao, Yuan & Matsunami, Yuki & Miyata, Shohei & Akashi, Yasunori, 2022. "Multi-agent reinforcement learning dealing with hybrid action spaces: A case study for off-grid oriented renewable building energy system," Applied Energy, Elsevier, vol. 326(C).
    2. Hakimi, Seyed Mehdi & Hasankhani, Arezoo & Shafie-khah, Miadreza & Catalão, João P.S., 2021. "Stochastic planning of a multi-microgrid considering integration of renewable energy resources and real-time electricity market," Applied Energy, Elsevier, vol. 298(C).
    3. Li, Yang & Han, Meng & Shahidehpour, Mohammad & Li, Jiazheng & Long, Chao, 2023. "Data-driven distributionally robust scheduling of community integrated energy systems with uncertain renewable generations considering integrated demand response," Applied Energy, Elsevier, vol. 335(C).
    4. Chen, Weidong & Wang, Junnan & Yu, Guanyi & Chen, Jiajia & Hu, Yumeng, 2022. "Research on day-ahead transactions between multi-microgrid based on cooperative game model," Applied Energy, Elsevier, vol. 316(C).
    5. Park, Keonwoo & Moon, Ilkyeong, 2022. "Multi-agent deep reinforcement learning approach for EV charging scheduling in a smart grid," Applied Energy, Elsevier, vol. 328(C).
    6. Soleimanzade, Mohammad Amin & Kumar, Amit & Sadrzadeh, Mohtada, 2022. "Novel data-driven energy management of a hybrid photovoltaic-reverse osmosis desalination system using deep reinforcement learning," Applied Energy, Elsevier, vol. 317(C).
    7. Li, Yang & Feng, Bo & Wang, Bin & Sun, Shuchao, 2022. "Joint planning of distributed generations and energy storage in active distribution networks: A Bi-Level programming approach," Energy, Elsevier, vol. 245(C).
    8. Qiu, Dawei & Wang, Yi & Sun, Mingyang & Strbac, Goran, 2022. "Multi-service provision for electric vehicles in power-transportation networks towards a low-carbon transition: A hierarchical and hybrid multi-agent reinforcement learning approach," Applied Energy, Elsevier, vol. 313(C).
    9. Li, Yang & Wang, Ruinong & Li, Yuanzheng & Zhang, Meng & Long, Chao, 2023. "Wind power forecasting considering data privacy protection: A federated deep reinforcement learning approach," Applied Energy, Elsevier, vol. 329(C).
    10. Zhang, Xizheng & Wang, Zeyu & Lu, Zhangyu, 2022. "Multi-objective load dispatch for microgrid with electric vehicles using modified gravitational search and particle swarm optimization algorithm," Applied Energy, Elsevier, vol. 306(PA).
    11. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    12. Xie, Peilin & Tan, Sen & Bazmohammadi, Najmeh & Guerrero, Josep. M. & Vasquez, Juan. C. & Alcala, Jose Matas & Carreño, Jorge El Mariachet, 2022. "A distributed real-time power management scheme for shipboard zonal multi-microgrid system," Applied Energy, Elsevier, vol. 317(C).
    13. Wang, Yong & Wu, Yuankai & Tang, Yingjuan & Li, Qin & He, Hongwen, 2023. "Cooperative energy management and eco-driving of plug-in hybrid electric vehicle via multi-agent reinforcement learning," Applied Energy, Elsevier, vol. 332(C).
    14. Parlikar, Anupam & Schott, Maximilian & Godse, Ketaki & Kucevic, Daniel & Jossen, Andreas & Hesse, Holger, 2023. "High-power electric vehicle charging: Low-carbon grid integration pathways with stationary lithium-ion battery systems and renewable generation," Applied Energy, Elsevier, vol. 333(C).
    15. Li, Yang & Bu, Fanjin & Li, Yuanzheng & Long, Chao, 2023. "Optimal scheduling of island integrated energy systems considering multi-uncertainties and hydrothermal simultaneous transmission: A deep reinforcement learning approach," Applied Energy, Elsevier, vol. 333(C).
    16. Nawaz, Arshad & Wu, Jing & Ye, Jun & Dong, Yidi & Long, Chengnian, 2023. "Distributed MPC-based energy scheduling for islanded multi-microgrid considering battery degradation and cyclic life deterioration," Applied Energy, Elsevier, vol. 329(C).
    17. Li, Yang & Wang, Bin & Yang, Zhen & Li, Jiazheng & Chen, Chen, 2022. "Hierarchical stochastic scheduling of multi-community integrated energy systems in uncertain environments via Stackelberg game," Applied Energy, Elsevier, vol. 308(C).
    18. Kim, H.J. & Kim, M.K., 2023. "A novel deep learning-based forecasting model optimized by heuristic algorithm for energy management of microgrid," Applied Energy, Elsevier, vol. 332(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Li, Xinyan & Wu, Nan, 2024. "A two-stage distributed robust optimal control strategy for energy collaboration in multi-regional integrated energy systems based on cooperative game," Energy, Elsevier, vol. 305(C).
    2. Li, Yang & Wang, Ruinong & Li, Yuanzheng & Zhang, Meng & Long, Chao, 2023. "Wind power forecasting considering data privacy protection: A federated deep reinforcement learning approach," Applied Energy, Elsevier, vol. 329(C).
    3. Li, Yang & Han, Meng & Shahidehpour, Mohammad & Li, Jiazheng & Long, Chao, 2023. "Data-driven distributionally robust scheduling of community integrated energy systems with uncertain renewable generations considering integrated demand response," Applied Energy, Elsevier, vol. 335(C).
    4. Li, Yang & Bu, Fanjin & Li, Yuanzheng & Long, Chao, 2023. "Optimal scheduling of island integrated energy systems considering multi-uncertainties and hydrothermal simultaneous transmission: A deep reinforcement learning approach," Applied Energy, Elsevier, vol. 333(C).
    5. Fan, Wei & Fan, Ying & Yao, Xing & Yi, Bowen & Jiang, Dalin & Wu, Lin, 2024. "Distributed transaction optimization model of multi-integrated energy systems based on nash negotiation," Renewable Energy, Elsevier, vol. 225(C).
    6. Han, Fengwu & Zeng, Jianfeng & Lin, Junjie & Zhao, Yunlong & Gao, Chong, 2023. "A stochastic hierarchical optimization and revenue allocation approach for multi-regional integrated energy systems based on cooperative games," Applied Energy, Elsevier, vol. 350(C).
    7. Jani, Ali & Jadid, Shahram, 2023. "Two-stage energy scheduling framework for multi-microgrid system in market environment," Applied Energy, Elsevier, vol. 336(C).
    8. Wang, Chunling & Liu, Chunming & Chen, Jian & Zhang, Gaoyuan, 2024. "Cooperative planning of renewable energy generation and multi-timescale flexible resources in active distribution networks," Applied Energy, Elsevier, vol. 356(C).
    9. Xu, Xun & Shao, Zhenguo & Chen, Feixiong & Cheng, Guoyang, 2024. "Multi-game optimization operation strategy for integrated energy system considering spatiotemporal correlation of renewable energy," Energy, Elsevier, vol. 303(C).
    10. Wei Wei & Li Ye & Yi Fang & Yingchun Wang & Xi Chen & Zhenhua Li, 2023. "Optimal Allocation of Energy Storage Capacity in Microgrids Considering the Uncertainty of Renewable Energy Generation," Sustainability, MDPI, vol. 15(12), pages 1-17, June.
    11. Shu, Lei & Mo, Yunjeong & Zhao, Dong, 2024. "Energy retrofits for smart and connected communities: Scopes and technologies," Renewable and Sustainable Energy Reviews, Elsevier, vol. 199(C).
    12. Zhang, Chao & Yin, Wanjun & Wen, Tao, 2024. "An advanced multi-objective collaborative scheduling strategy for large scale EV charging and discharging connected to the predictable wind power grid," Energy, Elsevier, vol. 287(C).
    13. Hongbin Sun & Hongyu Zou & Jianfeng Jia & Qiuzhen Shen & Zhenyu Duan & Xi Tang, 2024. "Master–Slave Game Optimization Scheduling of Multi-Microgrid Integrated Energy System Considering Comprehensive Demand Response and Wind and Storage Combination," Energies, MDPI, vol. 17(22), pages 1-29, November.
    14. Zhang, Bin & Hu, Weihao & Xu, Xiao & Li, Tao & Zhang, Zhenyuan & Chen, Zhe, 2022. "Physical-model-free intelligent energy management for a grid-connected hybrid wind-microturbine-PV-EV energy system via deep reinforcement learning approach," Renewable Energy, Elsevier, vol. 200(C), pages 433-448.
    15. Te Li & Mengze Zhang & Yan Zhou, 2024. "LTPNet Integration of Deep Learning and Environmental Decision Support Systems for Renewable Energy Demand Forecasting," Papers 2410.15286, arXiv.org.
    16. Pinthurat, Watcharakorn & Surinkaew, Tossaporn & Hredzak, Branislav, 2024. "An overview of reinforcement learning-based approaches for smart home energy management systems with energy storages," Renewable and Sustainable Energy Reviews, Elsevier, vol. 202(C).
    17. Wang, Zixuan & Xiao, Fu & Ran, Yi & Li, Yanxue & Xu, Yang, 2024. "Scalable energy management approach of residential hybrid energy system using multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 367(C).
    18. Zhou, Liwei & Preindl, Matthias, 2023. "Reconfigurable hybrid micro-grid with standardized power module for high performance energy conversion," Applied Energy, Elsevier, vol. 351(C).
    19. Zhao, Zhonghao & Lee, Carman K.M. & Yan, Xiaoyuan & Wang, Haonan, 2024. "Reinforcement learning for electric vehicle charging scheduling: A systematic review," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 190(C).
    20. Zhao, Zhonghao & Lee, Carman K.M. & Ren, Jingzheng, 2024. "A two-level charging scheduling method for public electric vehicle charging stations considering heterogeneous demand and nonlinear charging profile," Applied Energy, Elsevier, vol. 355(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:16:y:2023:i:7:p:3248-:d:1116390. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.