IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v17y2024i16p3990-d1454689.html
   My bibliography  Save this article

Multi-Layer Energy Management and Strategy Learning for Microgrids: A Proximal Policy Optimization Approach

Author

Listed:
  • Xiaohan Fang

    (Key Laboratory of Intelligent Computing and Signal Processing of Ministry of Education, Anhui University, Hefei 230601, China
    School of Electrical Engineering and Automation, Anhui University, Hefei 230601, China)

  • Peng Hong

    (School of Electrical Engineering and Automation, Anhui University, Hefei 230601, China)

  • Shuping He

    (Key Laboratory of Intelligent Computing and Signal Processing of Ministry of Education, Anhui University, Hefei 230601, China
    School of Electrical Engineering and Automation, Anhui University, Hefei 230601, China)

  • Yuhao Zhang

    (School of Electrical Engineering and Automation, Anhui University, Hefei 230601, China)

  • Di Tan

    (School of Electrical Engineering and Automation, Anhui University, Hefei 230601, China)

Abstract

An efficient energy management system (EMS) enhances microgrid performance in terms of stability, safety, and economy. Traditional centralized or decentralized energy management systems are unable to meet the increasing demands for autonomous decision-making, privacy protection, global optimization, and rapid collaboration simultaneously. This paper proposes a hierarchical multi-layer EMS for microgrid, comprising supply layer, demand layer, and neutral scheduling layer. Additionally, common mathematical optimization methods struggle with microgrid scheduling decision problem due to challenges in mechanism modeling, supply–demand uncertainty, and high real-time and autonomy requirements. Therefore, an improved proximal policy optimization (PPO) approach is proposed for the multi-layer EMS. Specifically, in the centrally managed supply layer, a centralized PPO algorithm is utilized to determine the optimal power generation strategy. In the decentralized demand layer, an auction market is established, and multi-agent proximal policy optimization (MAPPO) algorithm with an action-guidance-based mechanism is employed for each consumer, to implement individual auction strategy. The neutral scheduling layer interacts with other layers, manages information, and protects participant privacy. Numerical results validate the effectiveness of the proposed multi-layer EMS framework and the PPO-based optimization methods.

Suggested Citation

  • Xiaohan Fang & Peng Hong & Shuping He & Yuhao Zhang & Di Tan, 2024. "Multi-Layer Energy Management and Strategy Learning for Microgrids: A Proximal Policy Optimization Approach," Energies, MDPI, vol. 17(16), pages 1-22, August.
    • Handle: RePEc:gam:jeners:v:17:y:2024:i:16:p:3990-:d:1454689
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/17/16/3990/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1996-1073/17/16/3990/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Petrollese, Mario & Valverde, Luis & Cocco, Daniele & Cau, Giorgio & Guerra, José, 2016. "Real-time integration of optimal generation scheduling with MPC for the energy management of a renewable hydrogen-based microgrid," Applied Energy, Elsevier, vol. 166(C), pages 96-106.
    2. Zhu, Ziqing & Hu, Ze & Chan, Ka Wing & Bu, Siqi & Zhou, Bin & Xia, Shiwei, 2023. "Reinforcement learning in deregulated energy market: A comprehensive review," Applied Energy, Elsevier, vol. 329(C).
    3. Liu, Guodong & Jiang, Tao & Ollis, Thomas B. & Zhang, Xiaohu & Tomsovic, Kevin, 2019. "Distributed energy management for community microgrids considering network operational constraints and building thermal dynamics," Applied Energy, Elsevier, vol. 239(C), pages 83-95.
    4. Comodi, Gabriele & Giantomassi, Andrea & Severini, Marco & Squartini, Stefano & Ferracuti, Francesco & Fonti, Alessandro & Nardi Cesarini, Davide & Morodo, Matteo & Polonara, Fabio, 2015. "Multi-apartment residential microgrid with electrical and thermal storage devices: Experimental analysis and simulation of energy management strategies," Applied Energy, Elsevier, vol. 137(C), pages 854-866.
    5. Arroyo, Javier & Manna, Carlo & Spiessens, Fred & Helsen, Lieve, 2022. "Reinforced model predictive control (RL-MPC) for building energy management," Applied Energy, Elsevier, vol. 309(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. de la Hoz, Jordi & Martín, Helena & Alonso, Alex & Carolina Luna, Adriana & Matas, José & Vasquez, Juan C. & Guerrero, Josep M., 2019. "Regulatory-framework-embedded energy management system for microgrids: The case study of the Spanish self-consumption scheme," Applied Energy, Elsevier, vol. 251(C), pages 1-1.
    2. Álex Omar Topa Gavilema & José Domingo Álvarez & José Luis Torres Moreno & Manuel Pérez García, 2021. "Towards Optimal Management in Microgrids: An Overview," Energies, MDPI, vol. 14(16), pages 1-25, August.
    3. Fontenot, Hannah & Dong, Bing, 2019. "Modeling and control of building-integrated microgrids for optimal energy management – A review," Applied Energy, Elsevier, vol. 254(C).
    4. Zhang, Jingrui & Wu, Yihong & Guo, Yiran & Wang, Bo & Wang, Hengyue & Liu, Houde, 2016. "A hybrid harmony search algorithm with differential evolution for day-ahead scheduling problem of a microgrid with consideration of power flow constraints," Applied Energy, Elsevier, vol. 183(C), pages 791-804.
    5. Iolanda Saviuc & Herbert Peremans & Steven Van Passel & Kevin Milis, 2019. "Economic Performance of Using Batteries in European Residential Microgrids under the Net-Metering Scheme," Energies, MDPI, vol. 12(1), pages 1-28, January.
    6. Qiu, Rui & Zhang, Haoran & Wang, Guotao & Liang, Yongtu & Yan, Jinyue, 2023. "Green hydrogen-based energy storage service via power-to-gas technologies integrated with multi-energy microgrid," Applied Energy, Elsevier, vol. 350(C).
    7. Zhao, Jing & Yang, Zilan & Shi, Linyu & Liu, Dehan & Li, Haonan & Mi, Yumiao & Wang, Hongbin & Feng, Meili & Hutagaol, Timothy Joseph, 2024. "Photovoltaic capacity dynamic tracking model predictive control strategy of air-conditioning systems with consideration of flexible loads," Applied Energy, Elsevier, vol. 356(C).
    8. Osorio, J.D. & Rivera-Alvarez, A. & Swain, M. & Ordonez, J.C., 2015. "Exergy analysis of discharging multi-tank thermal energy storage systems with constant heat extraction," Applied Energy, Elsevier, vol. 154(C), pages 333-343.
    9. Wang, Yubin & Dong, Wei & Yang, Qiang, 2022. "Multi-stage optimal energy management of multi-energy microgrid in deregulated electricity markets," Applied Energy, Elsevier, vol. 310(C).
    10. Luca Brunelli & Emiliano Borri & Anna Laura Pisello & Andrea Nicolini & Carles Mateu & Luisa F. Cabeza, 2024. "Thermal Energy Storage in Energy Communities: A Perspective Overview through a Bibliometric Analysis," Sustainability, MDPI, vol. 16(14), pages 1-27, July.
    11. Abada, I. & Ehrenmann, A. & Lambin, X., 2017. "On the viability of energy communities," Cambridge Working Papers in Economics 1740, Faculty of Economics, University of Cambridge.
    12. Mikhail A. Bragin & Bing Yan & Akash Kumar & Nanpeng Yu & Peng Zhang, 2022. "Efficient Operations of Micro-Grids with Meshed Topology and Under Uncertainty through Exact Satisfaction of AC-PF, Droop Control and Tap-Changer Constraints," Energies, MDPI, vol. 15(10), pages 1-13, May.
    13. Zhao, Yincheng & Zhang, Guozhou & Hu, Weihao & Huang, Qi & Chen, Zhe & Blaabjerg, Frede, 2023. "Meta-learning based voltage control strategy for emergency faults of active distribution networks," Applied Energy, Elsevier, vol. 349(C).
    14. Bu, Yuntao & Yu, Hao & Ji, Haoran & Song, Guanyu & Xu, Jing & Li, Juan & Zhao, Jinli & Li, Peng, 2024. "Hybrid data-driven operation method for demand response of community integrated energy systems utilizing virtual and physical energy storage," Applied Energy, Elsevier, vol. 366(C).
    15. Miguel Carpintero-Rentería & David Santos-Martín & Josep M. Guerrero, 2019. "Microgrids Literature Review through a Layers Structure," Energies, MDPI, vol. 12(22), pages 1-22, November.
    16. Omaji Samuel & Nadeem Javaid & Mahmood Ashraf & Farruh Ishmanov & Muhammad Khalil Afzal & Zahoor Ali Khan, 2018. "Jaya based Optimization Method with High Dispatchable Distributed Generation for Residential Microgrid," Energies, MDPI, vol. 11(6), pages 1-29, June.
    17. Coelho, Vitor N. & Coelho, Igor M. & Coelho, Bruno N. & Cohen, Miri Weiss & Reis, Agnaldo J.R. & Silva, Sidelmo M. & Souza, Marcone J.F. & Fleming, Peter J. & Guimarães, Frederico G., 2016. "Multi-objective energy storage power dispatching using plug-in vehicles in a smart-microgrid," Renewable Energy, Elsevier, vol. 89(C), pages 730-742.
    18. Zhu, Kai & Li, Xueqiang & Campana, Pietro Elia & Li, Hailong & Yan, Jinyue, 2018. "Techno-economic feasibility of integrating energy storage systems in refrigerated warehouses," Applied Energy, Elsevier, vol. 216(C), pages 348-357.
    19. Dario Garozzo & Giuseppe Marco Tina, 2020. "Evaluation of the Effective Active Power Reserve for Fast Frequency Response of PV with BESS Inverters Considering Reactive Power Control," Energies, MDPI, vol. 13(13), pages 1-16, July.
    20. Guodong Liu & Maximiliano F. Ferrari & Thomas B. Ollis & Kevin Tomsovic, 2022. "An MILP-Based Distributed Energy Management for Coordination of Networked Microgrids," Energies, MDPI, vol. 15(19), pages 1-20, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:17:y:2024:i:16:p:3990-:d:1454689. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.