IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v368y2024ics0306261924008183.html
   My bibliography  Save this article

Energy management of a microgrid considering nonlinear losses in batteries through Deep Reinforcement Learning

Author

Listed:
  • Domínguez-Barbero, David
  • García-González, Javier
  • Sanz-Bobi, Miguel Á.
  • García-Cerrada, Aurelio

Abstract

The massive deployment of microgrids could play a significant role in achieving decarbonization of the electric sector amid the ongoing energy transition. The effective operation of these microgrids requires an Energy Management System (EMS), which establishes control set-points for all dispatchable components. EMSs can be formulated as classical optimization problems or as Partially-Observable Markov Decision Processes (POMDPs). Recently, Deep Reinforcement Learning (DRL) algorithms have been employed to solve the latter, gaining popularity in recent years. Since DRL methods promise to deal effectively with nonlinear dynamics, this paper examines the Twin-Delayed Deep Deterministic Policy Gradient (TD3) performance – a state-of-the-art method in DRL – for the EMS of a microgrid that includes nonlinear battery losses. Furthermore, the classical EMS-microgrid interaction is improved by refining the behavior of the underlying control system to obtain reliable results. The performance of this novel approach has been tested on two distinct microgrids – a residential one and a larger-scale grid – with a satisfactory outcome beyond reducing operational costs. Findings demonstrate the intrinsic potential of DRL-based algorithms for enhancing energy management and driving more efficient power systems.

Suggested Citation

  • Domínguez-Barbero, David & García-González, Javier & Sanz-Bobi, Miguel Á. & García-Cerrada, Aurelio, 2024. "Energy management of a microgrid considering nonlinear losses in batteries through Deep Reinforcement Learning," Applied Energy, Elsevier, vol. 368(C).
    • Handle: RePEc:eee:appene:v:368:y:2024:i:c:s0306261924008183
    DOI: 10.1016/j.apenergy.2024.123435
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261924008183
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2024.123435?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Amrollahi, Mohammad Hossein & Bathaee, Seyyed Mohammad Taghi, 2017. "Techno-economic optimization of hybrid photovoltaic/wind generation together with energy storage system in a stand-alone micro-grid subjected to demand response," Applied Energy, Elsevier, vol. 202(C), pages 66-77.
    2. David Domínguez-Barbero & Javier García-González & Miguel A. Sanz-Bobi & Eugenio F. Sánchez-Úbeda, 2020. "Optimising a Microgrid System by Deep Reinforcement Learning Techniques," Energies, MDPI, vol. 13(11), pages 1-18, June.
    3. Fotouhi, Abbas & Auger, Daniel J. & Propp, Karsten & Longo, Stefano & Wild, Mark, 2016. "A review on electric vehicle battery modelling: From Lithium-ion toward Lithium–Sulphur," Renewable and Sustainable Energy Reviews, Elsevier, vol. 56(C), pages 1008-1021.
    4. Sukumar, Shivashankar & Mokhlis, Hazlie & Mekhilef, Saad & Naidu, Kanendra & Karimi, Mazaher, 2017. "Mix-mode energy management strategy and battery sizing for economic operation of grid-tied microgrid," Energy, Elsevier, vol. 118(C), pages 1322-1333.
    5. Staffell, Iain & Pfenninger, Stefan, 2016. "Using bias-corrected reanalysis to simulate current and future wind power output," Energy, Elsevier, vol. 114(C), pages 1224-1239.
    6. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    7. Daniel Akinyele & Juri Belikov & Yoash Levron, 2018. "Challenges of Microgrids in Remote Communities: A STEEP Model Application," Energies, MDPI, vol. 11(2), pages 1-35, February.
    8. Zia, Muhammad Fahad & Elbouchikhi, Elhoussin & Benbouzid, Mohamed, 2018. "Microgrids energy management systems: A critical review on methods, solutions, and prospects," Applied Energy, Elsevier, vol. 222(C), pages 1033-1055.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. de la Hoz, Jordi & Martín, Helena & Alonso, Alex & Carolina Luna, Adriana & Matas, José & Vasquez, Juan C. & Guerrero, Josep M., 2019. "Regulatory-framework-embedded energy management system for microgrids: The case study of the Spanish self-consumption scheme," Applied Energy, Elsevier, vol. 251(C), pages 1-1.
    2. Ferahtia, Seydali & Houari, Azeddine & Cioara, Tudor & Bouznit, Mohammed & Rezk, Hegazy & Djerioui, Ali, 2024. "Recent advances on energy management and control of direct current microgrid for smart cities and industry: A Survey," Applied Energy, Elsevier, vol. 368(C).
    3. Solanke, Tirupati U. & Khatua, Pradeep K. & Ramachandaramurthy, Vigna K. & Yong, Jia Ying & Tan, Kang Miao, 2021. "Control and management of a multilevel electric vehicles infrastructure integrated with distributed resources: A comprehensive review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 144(C).
    4. Danny Espín-Sarzosa & Rodrigo Palma-Behnke & Oscar Núñez-Mata, 2020. "Energy Management Systems for Microgrids: Main Existing Trends in Centralized Control Architectures," Energies, MDPI, vol. 13(3), pages 1-32, January.
    5. Md Shafiullah & Akib Mostabe Refat & Md Ershadul Haque & Dewan Mabrur Hasan Chowdhury & Md Sanower Hossain & Abdullah G. Alharbi & Md Shafiul Alam & Amjad Ali & Shorab Hossain, 2022. "Review of Recent Developments in Microgrid Energy Management Strategies," Sustainability, MDPI, vol. 14(22), pages 1-30, November.
    6. Sun, Alexander Y., 2020. "Optimal carbon storage reservoir management through deep reinforcement learning," Applied Energy, Elsevier, vol. 278(C).
    7. Antoine Boche & Clément Foucher & Luiz Fernando Lavado Villa, 2022. "Understanding Microgrid Sustainability: A Systemic and Comprehensive Review," Energies, MDPI, vol. 15(8), pages 1-29, April.
    8. Wang, Yi & Qiu, Dawei & Sun, Mingyang & Strbac, Goran & Gao, Zhiwei, 2023. "Secure energy management of multi-energy microgrid: A physical-informed safe reinforcement learning approach," Applied Energy, Elsevier, vol. 335(C).
    9. Poolla, Chaitanya & Ishihara, Abraham K. & Milito, Rodolfo, 2019. "Designing near-optimal policies for energy management in a stochastic environment," Applied Energy, Elsevier, vol. 242(C), pages 1725-1737.
    10. Pinciroli, Luca & Baraldi, Piero & Compare, Michele & Zio, Enrico, 2023. "Optimal operation and maintenance of energy storage systems in grid-connected microgrids by deep reinforcement learning," Applied Energy, Elsevier, vol. 352(C).
    11. Huan Shen & Xingfa Shen & Yiming Chen, 2024. "Real-Time Microgrid Energy Scheduling Using Meta-Reinforcement Learning," Energies, MDPI, vol. 17(10), pages 1-15, May.
    12. Raya-Armenta, Jose Maurilio & Bazmohammadi, Najmeh & Avina-Cervantes, Juan Gabriel & Sáez, Doris & Vasquez, Juan C. & Guerrero, Josep M., 2021. "Energy management system optimization in islanded microgrids: An overview and future trends," Renewable and Sustainable Energy Reviews, Elsevier, vol. 149(C).
    13. Zia, Muhammad Fahad & Elbouchikhi, Elhoussin & Benbouzid, Mohamed, 2018. "Microgrids energy management systems: A critical review on methods, solutions, and prospects," Applied Energy, Elsevier, vol. 222(C), pages 1033-1055.
    14. Makbul A.M. Ramli & H.R.E.H. Bouchekara & Abdulsalam S. Alghamdi, 2019. "Efficient Energy Management in a Microgrid with Intermittent Renewable Energy and Storage Sources," Sustainability, MDPI, vol. 11(14), pages 1-28, July.
    15. Shanmugarajah Vinothine & Lidula N. Widanagama Arachchige & Athula D. Rajapakse & Roshani Kaluthanthrige, 2022. "Microgrid Energy Management and Methods for Managing Forecast Uncertainties," Energies, MDPI, vol. 15(22), pages 1-22, November.
    16. Silva, Jéssica Alice A. & López, Juan Camilo & Arias, Nataly Bañol & Rider, Marcos J. & da Silva, Luiz C.P., 2021. "An optimal stochastic energy management system for resilient microgrids," Applied Energy, Elsevier, vol. 300(C).
    17. Roslan, M.F. & Hannan, M.A. & Ker, Pin Jern & Uddin, M.N., 2019. "Microgrid control methods toward achieving sustainable energy management," Applied Energy, Elsevier, vol. 240(C), pages 583-607.
    18. Md Mainul Islam & Mahmood Nagrial & Jamal Rizk & Ali Hellany, 2021. "General Aspects, Islanding Detection, and Energy Management in Microgrids: A Review," Sustainability, MDPI, vol. 13(16), pages 1-45, August.
    19. Mao, Jiachen & Jafari, Mehdi & Botterud, Audun, 2022. "Planning low-carbon distributed power systems: Evaluating the role of energy storage," Energy, Elsevier, vol. 238(PA).
    20. Vinny Motjoadi & Pitshou N. Bokoro & Moses O. Onibonoje, 2020. "A Review of Microgrid-Based Approach to Rural Electrification in South Africa: Architecture and Policy Framework," Energies, MDPI, vol. 13(9), pages 1-22, May.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:368:y:2024:i:c:s0306261924008183. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.