IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v13y2019i1p123-d301988.html
   My bibliography  Save this article

Multi-Agent Reinforcement Learning Approach for Residential Microgrid Energy Scheduling

Author

Listed:
  • Xiaohan Fang

    (College of Information Science and Engineering, Northeastern University, Shenyang 110819, China)

  • Jinkuan Wang

    (College of Information Science and Engineering, Northeastern University, Shenyang 110819, China)

  • Guanru Song

    (College of Information Science and Engineering, Northeastern University, Shenyang 110819, China)

  • Yinghua Han

    (School of Computer and Communication Engineering, Northeastern University at Qinhuangdao, Qinhuangdao 066004, China)

  • Qiang Zhao

    (School of Control Engineering, Northeastern University at Qinhuangdao, Qinhuangdao 066004, China)

  • Zhiao Cao

    (College of Information Science and Engineering, Northeastern University, Shenyang 110819, China)

Abstract

Residential microgrid is widely considered as a new paradigm of the home energy management system. The complexity of Microgrid Energy Scheduling (MES) is increasing with the integration of Electric Vehicles (EVs) and Renewable Generations (RGs). Moreover, it is challenging to determine optimal scheduling strategies to guarantee the efficiency of the microgrid market and to balance all market participants’ benefits. In this paper, a Multi-Agent Reinforcement Learning (MARL) approach for residential MES is proposed to promote the autonomy and fairness of microgrid market operation. First, a multi-agent based residential microgrid model including Vehicle-to-Grid (V2G) and RGs is constructed and an auction-based microgrid market is built. Then, distinguish from Single-Agent Reinforcement Learning (SARL), MARL can achieve distributed autonomous learning for each agent and realize the equilibrium of all agents’ benefits, therefore, we formulate an equilibrium-based MARL framework according to each participant’ market orientation. Finally, to guarantee the fairness and privacy of the MARL process, we proposed an improved optimal Equilibrium Selection-MARL (ES-MARL) algorithm based on two mechanisms, private negotiation and maximum average reward. Simulation results demonstrate the overall performance and efficiency of proposed MARL are superior to that of SARL. Besides, it is verified that the improved ES-MARL can get higher average profit to balance all agents.

Suggested Citation

  • Xiaohan Fang & Jinkuan Wang & Guanru Song & Yinghua Han & Qiang Zhao & Zhiao Cao, 2019. "Multi-Agent Reinforcement Learning Approach for Residential Microgrid Energy Scheduling," Energies, MDPI, vol. 13(1), pages 1-26, December.
    • Handle: RePEc:gam:jeners:v:13:y:2019:i:1:p:123-:d:301988
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/13/1/123/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1996-1073/13/1/123/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Kazmi, Hussain & Suykens, Johan & Balint, Attila & Driesen, Johan, 2019. "Multi-agent reinforcement learning for modeling and control of thermostatically controlled loads," Applied Energy, Elsevier, vol. 238(C), pages 1022-1035.
    2. Wang, Chengshan & Liu, Yixin & Li, Xialin & Guo, Li & Qiao, Lei & Lu, Hai, 2016. "Energy management system for stand-alone diesel-wind-biomass microgrid with energy storage system," Energy, Elsevier, vol. 97(C), pages 90-104.
    3. Xiong, Rui & Cao, Jiayi & Yu, Quanqing, 2018. "Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle," Applied Energy, Elsevier, vol. 211(C), pages 538-548.
    4. Thomas, Dimitrios & Deblecker, Olivier & Ioakimidis, Christos S., 2018. "Optimal operation of an energy management system for a grid-connected smart building considering photovoltaics’ uncertainty and stochastic electric vehicles’ driving schedule," Applied Energy, Elsevier, vol. 210(C), pages 1188-1206.
    5. Zhang, Xinan & Bao, Jie & Wang, Ruigang & Zheng, Chaoxu & Skyllas-Kazacos, Maria, 2017. "Dissipativity based distributed economic model predictive control for residential microgrids with renewable energy generation and battery energy storage," Renewable Energy, Elsevier, vol. 100(C), pages 18-34.
    6. Kofinas, P. & Dounis, A.I. & Vouros, G.A., 2018. "Fuzzy Q-Learning for multi-agent decentralized energy management in microgrids," Applied Energy, Elsevier, vol. 219(C), pages 53-67.
    7. Wang, Hewu & Zhang, Xiaobin & Ouyang, Minggao, 2015. "Energy consumption of electric vehicles based on real-world driving patterns: A case study of Beijing," Applied Energy, Elsevier, vol. 157(C), pages 710-719.
    8. Pascual, Julio & Barricarte, Javier & Sanchis, Pablo & Marroyo, Luis, 2015. "Energy management strategy for a renewable-based residential microgrid with generation and demand forecasting," Applied Energy, Elsevier, vol. 158(C), pages 12-25.
    9. Zhou, Guanghui & Ou, Xunmin & Zhang, Xiliang, 2013. "Development of electric vehicles use in China: A study from the perspective of life-cycle energy consumption and greenhouse gas emissions," Energy Policy, Elsevier, vol. 59(C), pages 875-884.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Dorokhova, Marina & Martinson, Yann & Ballif, Christophe & Wyrsch, Nicolas, 2021. "Deep reinforcement learning control of electric vehicle charging in the presence of photovoltaic generation," Applied Energy, Elsevier, vol. 301(C).
    2. Kapil Deshpande & Philipp Möhl & Alexander Hämmerle & Georg Weichhart & Helmut Zörrer & Andreas Pichler, 2022. "Energy Management Simulation with Multi-Agent Reinforcement Learning: An Approach to Achieve Reliability and Resilience," Energies, MDPI, vol. 15(19), pages 1-35, October.
    3. Eleonora Achiluzzi & Kirushaanth Kobikrishna & Abenayan Sivabalan & Carlos Sabillon & Bala Venkatesh, 2020. "Optimal Asset Planning for Prosumers Considering Energy Storage and Photovoltaic (PV) Units: A Stochastic Approach," Energies, MDPI, vol. 13(7), pages 1-20, April.
    4. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Perera, A.T.D. & Kamalaruban, Parameswaran, 2021. "Applications of reinforcement learning in energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 137(C).
    2. Fontenot, Hannah & Dong, Bing, 2019. "Modeling and control of building-integrated microgrids for optimal energy management – A review," Applied Energy, Elsevier, vol. 254(C).
    3. Nyong-Bassey, Bassey Etim & Giaouris, Damian & Patsios, Charalampos & Papadopoulou, Simira & Papadopoulos, Athanasios I. & Walker, Sara & Voutetakis, Spyros & Seferlis, Panos & Gadoue, Shady, 2020. "Reinforcement learning based adaptive power pinch analysis for energy management of stand-alone hybrid energy storage systems considering uncertainty," Energy, Elsevier, vol. 193(C).
    4. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    5. Pinto, Giuseppe & Kathirgamanathan, Anjukan & Mangina, Eleni & Finn, Donal P. & Capozzoli, Alfonso, 2022. "Enhancing energy management in grid-interactive buildings: A comparison among cooperative and coordinated architectures," Applied Energy, Elsevier, vol. 310(C).
    6. de la Hoz, Jordi & Martín, Helena & Alonso, Alex & Carolina Luna, Adriana & Matas, José & Vasquez, Juan C. & Guerrero, Josep M., 2019. "Regulatory-framework-embedded energy management system for microgrids: The case study of the Spanish self-consumption scheme," Applied Energy, Elsevier, vol. 251(C), pages 1-1.
    7. Xianchun Tan & Yuan Zeng & Baihe Gu & Yi Wang & Baoguang Xu, 2018. "Scenario Analysis of Urban Road Transportation Energy Demand and GHG Emissions in China—A Case Study for Chongqing," Sustainability, MDPI, vol. 10(6), pages 1-32, June.
    8. Wang, An & Tu, Ran & Gai, Yijun & Pereira, Lucas G. & Vaughan, J. & Posen, I. Daniel & Miller, Eric J. & Hatzopoulou, Marianne, 2020. "Capturing uncertainty in emission estimates related to vehicle electrification and implications for metropolitan greenhouse gas emission inventories," Applied Energy, Elsevier, vol. 265(C).
    9. Zia, Muhammad Fahad & Elbouchikhi, Elhoussin & Benbouzid, Mohamed, 2018. "Microgrids energy management systems: A critical review on methods, solutions, and prospects," Applied Energy, Elsevier, vol. 222(C), pages 1033-1055.
    10. Yi Kuang & Xiuli Wang & Hongyang Zhao & Yijun Huang & Xianlong Chen & Xifan Wang, 2020. "Agent-Based Energy Sharing Mechanism Using Deep Deterministic Policy Gradient Algorithm," Energies, MDPI, vol. 13(19), pages 1-20, September.
    11. Kong, Xiangyu & Kong, Deqian & Yao, Jingtao & Bai, Linquan & Xiao, Jie, 2020. "Online pricing of demand response based on long short-term memory and reinforcement learning," Applied Energy, Elsevier, vol. 271(C).
    12. Qiu, Dawei & Ye, Yujian & Papadaskalopoulos, Dimitrios & Strbac, Goran, 2021. "Scalable coordinated management of peer-to-peer energy trading: A multi-cluster deep reinforcement learning approach," Applied Energy, Elsevier, vol. 292(C).
    13. Vázquez-Canteli, José R. & Nagy, Zoltán, 2019. "Reinforcement learning for demand response: A review of algorithms and modeling techniques," Applied Energy, Elsevier, vol. 235(C), pages 1072-1089.
    14. Rubén López-Rodríguez & Adriana Aguilera-González & Ionel Vechiu & Seddik Bacha, 2021. "Day-Ahead MPC Energy Management System for an Island Wind/Storage Hybrid Power Plant," Energies, MDPI, vol. 14(4), pages 1-33, February.
    15. Ji, Zhenya & Huang, Xueliang, 2018. "Plug-in electric vehicle charging infrastructure deployment of China towards 2020: Policies, methodologies, and challenges," Renewable and Sustainable Energy Reviews, Elsevier, vol. 90(C), pages 710-727.
    16. Cagnano, A. & De Tuglie, E. & Mancarella, P., 2020. "Microgrids: Overview and guidelines for practical implementations and operation," Applied Energy, Elsevier, vol. 258(C).
    17. Àlex Alonso-Travesset & Helena Martín & Sergio Coronas & Jordi de la Hoz, 2022. "Optimization Models under Uncertainty in Distributed Generation Systems: A Review," Energies, MDPI, vol. 15(5), pages 1-40, March.
    18. Agüera-Pérez, Agustín & Palomares-Salas, José Carlos & González de la Rosa, Juan José & Florencias-Oliveros, Olivia, 2018. "Weather forecasts for microgrid energy management: Review, discussion and recommendations," Applied Energy, Elsevier, vol. 228(C), pages 265-278.
    19. Soodabeh Ghalambaz & Christopher Neil Hulme, 2022. "A Scientometric Analysis of Energy Management in the Past Five Years (2018–2022)," Sustainability, MDPI, vol. 14(18), pages 1-19, September.
    20. Plain, N. & Hingray, B. & Mathy, S., 2019. "Accounting for low solar resource days to size 100% solar microgrids power systems in Africa," Renewable Energy, Elsevier, vol. 131(C), pages 448-458.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:13:y:2019:i:1:p:123-:d:301988. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.