IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v232y2021ics0360544221012834.html
   My bibliography  Save this article

Lifelong control of off-grid microgrid with model-based reinforcement learning

Author

Listed:
  • Totaro, Simone
  • Boukas, Ioannis
  • Jonsson, Anders
  • Cornélusse, Bertrand

Abstract

Off-grid microgrids are receiving a growing interest for rural electrification purposes in developing countries due to their ability to ensure affordable, sustainable and reliable energy services. Off-grid microgrids rely on renewable energy sources (RES) coupled with storage systems to supply the electrical consumption. The inherent uncertainty introduced by RES as well as the stochastic nature of the electrical demand in rural contexts pose significant challenges to the efficient control of off-grid microgrids throughout their entire life span. In this paper, we address the lifelong control problem of an isolated microgrid. We categorize the set of changes that may occur over its life span in progressive and abrupt changes. We propose a novel model-based reinforcement learning algorithm that is able to address both types of changes. In particular, the proposed algorithm demonstrates generalisation properties, transfer capabilities and better robustness in case of fast-changing system dynamics. The proposed algorithm is compared against a rule-based policy and a model predictive controller with look-ahead. The results show that the trained agent is able to outperform both benchmarks in the lifelong setting where the system dynamics are changing over time.

Suggested Citation

  • Totaro, Simone & Boukas, Ioannis & Jonsson, Anders & Cornélusse, Bertrand, 2021. "Lifelong control of off-grid microgrid with model-based reinforcement learning," Energy, Elsevier, vol. 232(C).
    • Handle: RePEc:eee:energy:v:232:y:2021:i:c:s0360544221012834
    DOI: 10.1016/j.energy.2021.121035
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0360544221012834
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.energy.2021.121035?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Zhang, Yan & Fu, Lijun & Zhu, Wanlu & Bao, Xianqiang & Liu, Cang, 2018. "Robust model predictive control for optimal energy management of island microgrids with uncertainties," Energy, Elsevier, vol. 164(C), pages 1229-1241.
    2. Parisio, Alessandra & Rikos, Evangelos & Tzamalis, George & Glielmo, Luigi, 2014. "Use of model predictive control for experimental microgrid optimization," Applied Energy, Elsevier, vol. 115(C), pages 37-46.
    3. Kofinas, P. & Dounis, A.I. & Vouros, G.A., 2018. "Fuzzy Q-Learning for multi-agent decentralized energy management in microgrids," Applied Energy, Elsevier, vol. 219(C), pages 53-67.
    4. Kuznetsova, Elizaveta & Li, Yan-Fu & Ruiz, Carlos & Zio, Enrico & Ault, Graham & Bell, Keith, 2013. "Reinforcement learning for microgrid energy management," Energy, Elsevier, vol. 59(C), pages 133-146.
    5. Sean J. Taylor & Benjamin Letham, 2018. "Forecasting at Scale," The American Statistician, Taylor & Francis Journals, vol. 72(1), pages 37-45, January.
    6. Balderrama, Sergio & Lombardi, Francesco & Riva, Fabio & Canedo, Walter & Colombo, Emanuela & Quoilin, Sylvain, 2019. "A two-stage linear programming optimization framework for isolated hybrid microgrids in a rural context: The case study of the “El Espino” community," Energy, Elsevier, vol. 188(C).
    7. Nyong-Bassey, Bassey Etim & Giaouris, Damian & Patsios, Charalampos & Papadopoulou, Simira & Papadopoulos, Athanasios I. & Walker, Sara & Voutetakis, Spyros & Seferlis, Panos & Gadoue, Shady, 2020. "Reinforcement learning based adaptive power pinch analysis for energy management of stand-alone hybrid energy storage systems considering uncertainty," Energy, Elsevier, vol. 193(C).
    8. Lombardi, Francesco & Balderrama, Sergio & Quoilin, Sylvain & Colombo, Emanuela, 2019. "Generating high-resolution multi-energy load profiles for remote areas with an open-source stochastic model," Energy, Elsevier, vol. 177(C), pages 433-444.
    9. Rocchetta, R. & Bellani, L. & Compare, M. & Zio, E. & Patelli, E., 2019. "A reinforcement learning framework for optimal operation and maintenance of power grids," Applied Energy, Elsevier, vol. 241(C), pages 291-301.
    10. Bruni, G. & Cordiner, S. & Mulone, V. & Sinisi, V. & Spagnolo, F., 2016. "Energy management in a domestic microgrid by means of model predictive controllers," Energy, Elsevier, vol. 108(C), pages 119-131.
    11. Arnab Nilim & Laurent El Ghaoui, 2005. "Robust Control of Markov Decision Processes with Uncertain Transition Matrices," Operations Research, INFORMS, vol. 53(5), pages 780-798, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Kang, Dongju & Kang, Doeun & Hwangbo, Sumin & Niaz, Haider & Lee, Won Bo & Liu, J. Jay & Na, Jonggeol, 2023. "Optimal planning of hybrid energy storage systems using curtailed renewable energy through deep reinforcement learning," Energy, Elsevier, vol. 284(C).
    2. Nakıp, Mert & Çopur, Onur & Biyik, Emrah & Güzeliş, Cüneyt, 2023. "Renewable energy management in smart home environment via forecast embedded scheduling based on Recurrent Trend Predictive Neural Network," Applied Energy, Elsevier, vol. 340(C).
    3. Li, Yanxue & Wang, Zixuan & Xu, Wenya & Gao, Weijun & Xu, Yang & Xiao, Fu, 2023. "Modeling and energy dynamic control for a ZEH via hybrid model-based deep reinforcement learning," Energy, Elsevier, vol. 277(C).
    4. Harrold, Daniel J.B. & Cao, Jun & Fan, Zhong, 2022. "Data-driven battery operation for energy arbitrage using rainbow deep reinforcement learning," Energy, Elsevier, vol. 238(PC).
    5. Tomin, Nikita & Shakirov, Vladislav & Kurbatsky, Victor & Muzychuk, Roman & Popova, Ekaterina & Sidorov, Denis & Kozlov, Alexandr & Yang, Dechang, 2022. "A multi-criteria approach to designing and managing a renewable energy community," Renewable Energy, Elsevier, vol. 199(C), pages 1153-1175.
    6. Zhou, Yanting & Ma, Zhongjing & Zhang, Jinhui & Zou, Suli, 2022. "Data-driven stochastic energy management of multi energy system using deep reinforcement learning," Energy, Elsevier, vol. 261(PA).
    7. Liu, Xiangfei & Ren, Mifeng & Yang, Zhile & Yan, Gaowei & Guo, Yuanjun & Cheng, Lan & Wu, Chengke, 2022. "A multi-step predictive deep reinforcement learning algorithm for HVAC control systems in smart buildings," Energy, Elsevier, vol. 259(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Nyong-Bassey, Bassey Etim & Giaouris, Damian & Patsios, Charalampos & Papadopoulou, Simira & Papadopoulos, Athanasios I. & Walker, Sara & Voutetakis, Spyros & Seferlis, Panos & Gadoue, Shady, 2020. "Reinforcement learning based adaptive power pinch analysis for energy management of stand-alone hybrid energy storage systems considering uncertainty," Energy, Elsevier, vol. 193(C).
    2. Zheng, Lingwei & Wu, Hao & Guo, Siqi & Sun, Xinyu, 2023. "Real-time dispatch of an integrated energy system based on multi-stage reinforcement learning with an improved action-choosing strategy," Energy, Elsevier, vol. 277(C).
    3. Harrold, Daniel J.B. & Cao, Jun & Fan, Zhong, 2022. "Data-driven battery operation for energy arbitrage using rainbow deep reinforcement learning," Energy, Elsevier, vol. 238(PC).
    4. Lilia Tightiz & Joon Yoo, 2022. "A Review on a Data-Driven Microgrid Management System Integrating an Active Distribution Network: Challenges, Issues, and New Trends," Energies, MDPI, vol. 15(22), pages 1-24, November.
    5. Yamashita, Daniela Yassuda & Vechiu, Ionel & Gaubert, Jean-Paul, 2020. "A review of hierarchical control for building microgrids," Renewable and Sustainable Energy Reviews, Elsevier, vol. 118(C).
    6. Perera, A.T.D. & Kamalaruban, Parameswaran, 2021. "Applications of reinforcement learning in energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 137(C).
    7. Fontenot, Hannah & Dong, Bing, 2019. "Modeling and control of building-integrated microgrids for optimal energy management – A review," Applied Energy, Elsevier, vol. 254(C).
    8. Liu, Yixin & Shi, Haoqi & Guo, Li & Xu, Tao & Zhao, Bo & Wang, Chengshan, 2022. "Towards long-period operational reliability of independent microgrid: A risk-aware energy scheduling and stochastic optimization method," Energy, Elsevier, vol. 254(PB).
    9. Juan Moreno-Castro & Victor Samuel Ocaña Guevara & Lesyani Teresa León Viltre & Yandi Gallego Landera & Oscar Cuaresma Zevallos & Miguel Aybar-Mejía, 2023. "Microgrid Management Strategies for Economic Dispatch of Electricity Using Model Predictive Control Techniques: A Review," Energies, MDPI, vol. 16(16), pages 1-24, August.
    10. Abdellatif Elmouatamid & Radouane Ouladsine & Mohamed Bakhouya & Najib El Kamoun & Mohammed Khaidar & Khalid Zine-Dine, 2020. "Review of Control and Energy Management Approaches in Micro-Grid Systems," Energies, MDPI, vol. 14(1), pages 1-30, December.
    11. Van-Hai Bui & Akhtar Hussain & Hak-Man Kim, 2019. "Q-Learning-Based Operation Strategy for Community Battery Energy Storage System (CBESS) in Microgrid System," Energies, MDPI, vol. 12(9), pages 1-17, May.
    12. Correa-Jullian, Camila & López Droguett, Enrique & Cardemil, José Miguel, 2020. "Operation scheduling in a solar thermal system: A reinforcement learning-based framework," Applied Energy, Elsevier, vol. 268(C).
    13. Zhang, Li & Gao, Yan & Zhu, Hongbo & Tao, Li, 2022. "Bi-level stochastic real-time pricing model in multi-energy generation system: A reinforcement learning approach," Energy, Elsevier, vol. 239(PA).
    14. Chen, Pengzhan & Liu, Mengchao & Chen, Chuanxi & Shang, Xin, 2019. "A battery management strategy in microgrid for personalized customer requirements," Energy, Elsevier, vol. 189(C).
    15. Sabarathinam Srinivasan & Suresh Kumarasamy & Zacharias E. Andreadakis & Pedro G. Lind, 2023. "Artificial Intelligence and Mathematical Models of Power Grids Driven by Renewable Energy Sources: A Survey," Energies, MDPI, vol. 16(14), pages 1-56, July.
    16. Harrold, Daniel J.B. & Cao, Jun & Fan, Zhong, 2022. "Renewable energy integration and microgrid energy trading using multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 318(C).
    17. Yi Kuang & Xiuli Wang & Hongyang Zhao & Yijun Huang & Xianlong Chen & Xifan Wang, 2020. "Agent-Based Energy Sharing Mechanism Using Deep Deterministic Policy Gradient Algorithm," Energies, MDPI, vol. 13(19), pages 1-20, September.
    18. Dimitrios Vamvakas & Panagiotis Michailidis & Christos Korkas & Elias Kosmatopoulos, 2023. "Review and Evaluation of Reinforcement Learning Frameworks on Smart Grid Applications," Energies, MDPI, vol. 16(14), pages 1-38, July.
    19. Tsianikas, Stamatis & Yousefi, Nooshin & Zhou, Jian & Rodgers, Mark D. & Coit, David, 2021. "A storage expansion planning framework using reinforcement learning and simulation-based optimization," Applied Energy, Elsevier, vol. 290(C).
    20. Danny Espín-Sarzosa & Rodrigo Palma-Behnke & Oscar Núñez-Mata, 2020. "Energy Management Systems for Microgrids: Main Existing Trends in Centralized Control Architectures," Energies, MDPI, vol. 13(3), pages 1-32, January.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:232:y:2021:i:c:s0360544221012834. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.