IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v346y2023ics0306261923006876.html
   My bibliography  Save this article

MERLIN: Multi-agent offline and transfer learning for occupant-centric operation of grid-interactive communities

Author

Listed:
  • Nweye, Kingsley
  • Sankaranarayanan, Siva
  • Nagy, Zoltan

Abstract

Building and power generation decarbonization present new challenges in electric grid reliability as a result of renewable energy source intermittency and increase in grid load caused by end-use electrification. To restore reliability, grid-interactive efficient buildings can provide grid flexibility services through demand response. Reinforcement learning is well-suited for energy management in grid-interactive efficient buildings as it is able to adapt to unique building characteristics compared to rule-based control and model predictive control. Yet, factors hindering the adoption of reinforcement learning in real-world applications include its sample inefficiency during training, control security and generalizability. Here we address these challenges by proposing the MERLIN framework for the training, evaluation, deployment and transfer of control policies for distributed energy resources in grid-interactive communities for different levels of data availability. We utilize a real-world community smart meter dataset to show that while independently trained battery control policies can learn unique occupant behavior and provide up to 60% performance improvement at the district level, transfer learning provides comparable building and district level performance while reducing training costs.

Suggested Citation

  • Nweye, Kingsley & Sankaranarayanan, Siva & Nagy, Zoltan, 2023. "MERLIN: Multi-agent offline and transfer learning for occupant-centric operation of grid-interactive communities," Applied Energy, Elsevier, vol. 346(C).
    • Handle: RePEc:eee:appene:v:346:y:2023:i:c:s0306261923006876
    DOI: 10.1016/j.apenergy.2023.121323
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261923006876
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2023.121323?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Burger, Scott & Chaves-Ávila, Jose Pablo & Batlle, Carlos & Pérez-Arriaga, Ignacio J., 2017. "A review of the value of aggregators in electricity systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 77(C), pages 395-405.
    2. Oriol Vinyals & Igor Babuschkin & Wojciech M. Czarnecki & Michaël Mathieu & Andrew Dudzik & Junyoung Chung & David H. Choi & Richard Powell & Timo Ewalds & Petko Georgiev & Junhyuk Oh & Dan Horgan & M, 2019. "Grandmaster level in StarCraft II using multi-agent reinforcement learning," Nature, Nature, vol. 575(7782), pages 350-354, November.
    3. Pinto, Giuseppe & Piscitelli, Marco Savino & Vázquez-Canteli, José Ramón & Nagy, Zoltán & Capozzoli, Alfonso, 2021. "Coordinated energy management for a cluster of buildings through deep reinforcement learning," Energy, Elsevier, vol. 229(C).
    4. Pinto, Giuseppe & Deltetto, Davide & Capozzoli, Alfonso, 2021. "Data-driven district energy management with surrogate models and deep reinforcement learning," Applied Energy, Elsevier, vol. 304(C).
    5. Arroyo, Javier & Manna, Carlo & Spiessens, Fred & Helsen, Lieve, 2022. "Reinforced model predictive control (RL-MPC) for building energy management," Applied Energy, Elsevier, vol. 309(C).
    6. Davide Deltetto & Davide Coraci & Giuseppe Pinto & Marco Savino Piscitelli & Alfonso Capozzoli, 2021. "Exploring the Potentialities of Deep Reinforcement Learning for Incentive-Based Demand Response in a Cluster of Small Commercial Buildings," Energies, MDPI, vol. 14(10), pages 1-25, May.
    7. Gautam Gowrisankaran & Stanley S. Reynolds & Mario Samano, 2016. "Intermittency and the Value of Renewable Energy," Journal of Political Economy, University of Chicago Press, vol. 124(4), pages 1187-1234.
    8. Leibowicz, Benjamin D. & Lanham, Christopher M. & Brozynski, Max T. & Vázquez-Canteli, José R. & Castejón, Nicolás Castillo & Nagy, Zoltan, 2018. "Optimal decarbonization pathways for urban residential building energy services," Applied Energy, Elsevier, vol. 230(C), pages 1311-1325.
    9. Peter R. Wurman & Samuel Barrett & Kenta Kawamoto & James MacGlashan & Kaushik Subramanian & Thomas J. Walsh & Roberto Capobianco & Alisa Devlic & Franziska Eckert & Florian Fuchs & Leilani Gilpin & P, 2022. "Outracing champion Gran Turismo drivers with deep reinforcement learning," Nature, Nature, vol. 602(7896), pages 223-228, February.
    10. Wang, Zhe & Hong, Tianzhen, 2020. "Reinforcement learning for building controls: The opportunities and challenges," Applied Energy, Elsevier, vol. 269(C).
    11. Houben, Nikolaus & Cosic, Armin & Stadler, Michael & Mansoor, Muhammad & Zellinger, Michael & Auer, Hans & Ajanovic, Amela & Haas, Reinhard, 2023. "Optimal dispatch of a multi-energy system microgrid under uncertainty: A renewable energy community in Austria," Applied Energy, Elsevier, vol. 337(C).
    12. Coraci, Davide & Brandi, Silvio & Hong, Tianzhen & Capozzoli, Alfonso, 2023. "Online transfer learning strategy for enhancing the scalability and deployment of deep reinforcement learning control in smart buildings," Applied Energy, Elsevier, vol. 333(C).
    13. Blum, David & Wang, Zhe & Weyandt, Chris & Kim, Donghun & Wetter, Michael & Hong, Tianzhen & Piette, Mary Ann, 2022. "Field demonstration and implementation analysis of model predictive control in an office HVAC system," Applied Energy, Elsevier, vol. 318(C).
    14. Deng, Zhipeng & Wang, Xuezheng & Dong, Bing, 2023. "Quantum computing for future real-time building HVAC controls," Applied Energy, Elsevier, vol. 334(C).
    15. Haider, Haider Tarish & See, Ong Hang & Elmenreich, Wilfried, 2016. "A review of residential demand response of smart grid," Renewable and Sustainable Energy Reviews, Elsevier, vol. 59(C), pages 166-178.
    16. Pinto, Giuseppe & Kathirgamanathan, Anjukan & Mangina, Eleni & Finn, Donal P. & Capozzoli, Alfonso, 2022. "Enhancing energy management in grid-interactive buildings: A comparison among cooperative and coordinated architectures," Applied Energy, Elsevier, vol. 310(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lefeng Cheng & Xin Wei & Manling Li & Can Tan & Meng Yin & Teng Shen & Tao Zou, 2024. "Integrating Evolutionary Game-Theoretical Methods and Deep Reinforcement Learning for Adaptive Strategy Optimization in User-Side Electricity Markets: A Comprehensive Review," Mathematics, MDPI, vol. 12(20), pages 1-56, October.
    2. Wu, Haochi & Qiu, Dawei & Zhang, Liyu & Sun, Mingyang, 2024. "Adaptive multi-agent reinforcement learning for flexible resource management in a virtual power plant with dynamic participating multi-energy buildings," Applied Energy, Elsevier, vol. 374(C).
    3. Xiao, Tianqi & You, Fengqi, 2024. "Physically consistent deep learning-based day-ahead energy dispatching and thermal comfort control for grid-interactive communities," Applied Energy, Elsevier, vol. 353(PB).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    2. Ayas Shaqour & Aya Hagishima, 2022. "Systematic Review on Deep Reinforcement Learning-Based Energy Management for Different Building Types," Energies, MDPI, vol. 15(22), pages 1-27, November.
    3. Liu, Mingzhe & Guo, Mingyue & Fu, Yangyang & O’Neill, Zheng & Gao, Yuan, 2024. "Expert-guided imitation learning for energy management: Evaluating GAIL’s performance in building control applications," Applied Energy, Elsevier, vol. 372(C).
    4. Pinto, Giuseppe & Deltetto, Davide & Capozzoli, Alfonso, 2021. "Data-driven district energy management with surrogate models and deep reinforcement learning," Applied Energy, Elsevier, vol. 304(C).
    5. Pinto, Giuseppe & Kathirgamanathan, Anjukan & Mangina, Eleni & Finn, Donal P. & Capozzoli, Alfonso, 2022. "Enhancing energy management in grid-interactive buildings: A comparison among cooperative and coordinated architectures," Applied Energy, Elsevier, vol. 310(C).
    6. Silvestri, Alberto & Coraci, Davide & Brandi, Silvio & Capozzoli, Alfonso & Borkowski, Esther & Köhler, Johannes & Wu, Duan & Zeilinger, Melanie N. & Schlueter, Arno, 2024. "Real building implementation of a deep reinforcement learning controller to enhance energy efficiency and indoor temperature control," Applied Energy, Elsevier, vol. 368(C).
    7. Wang, Xuezheng & Dong, Bing, 2024. "Long-term experimental evaluation and comparison of advanced controls for HVAC systems," Applied Energy, Elsevier, vol. 371(C).
    8. Dimitrios Vamvakas & Panagiotis Michailidis & Christos Korkas & Elias Kosmatopoulos, 2023. "Review and Evaluation of Reinforcement Learning Frameworks on Smart Grid Applications," Energies, MDPI, vol. 16(14), pages 1-38, July.
    9. Zhuang, Dian & Gan, Vincent J.L. & Duygu Tekler, Zeynep & Chong, Adrian & Tian, Shuai & Shi, Xing, 2023. "Data-driven predictive control for smart HVAC system in IoT-integrated buildings with time-series forecasting and reinforcement learning," Applied Energy, Elsevier, vol. 338(C).
    10. Gao, Yuan & Matsunami, Yuki & Miyata, Shohei & Akashi, Yasunori, 2022. "Multi-agent reinforcement learning dealing with hybrid action spaces: A case study for off-grid oriented renewable building energy system," Applied Energy, Elsevier, vol. 326(C).
    11. Coraci, Davide & Brandi, Silvio & Hong, Tianzhen & Capozzoli, Alfonso, 2023. "Online transfer learning strategy for enhancing the scalability and deployment of deep reinforcement learning control in smart buildings," Applied Energy, Elsevier, vol. 333(C).
    12. Yi, Zonggen & Luo, Yusheng & Westover, Tyler & Katikaneni, Sravya & Ponkiya, Binaka & Sah, Suba & Mahmud, Sadab & Raker, David & Javaid, Ahmad & Heben, Michael J. & Khanna, Raghav, 2022. "Deep reinforcement learning based optimization for a tightly coupled nuclear renewable integrated energy system," Applied Energy, Elsevier, vol. 328(C).
    13. Pinto, Giuseppe & Piscitelli, Marco Savino & Vázquez-Canteli, José Ramón & Nagy, Zoltán & Capozzoli, Alfonso, 2021. "Coordinated energy management for a cluster of buildings through deep reinforcement learning," Energy, Elsevier, vol. 229(C).
    14. Charalampos Rafail Lazaridis & Iakovos Michailidis & Georgios Karatzinis & Panagiotis Michailidis & Elias Kosmatopoulos, 2024. "Evaluating Reinforcement Learning Algorithms in Residential Energy Saving and Comfort Management," Energies, MDPI, vol. 17(3), pages 1-33, January.
    15. Zhou, Xinlei & Xue, Shan & Du, Han & Ma, Zhenjun, 2023. "Optimization of building demand flexibility using reinforcement learning and rule-based expert systems," Applied Energy, Elsevier, vol. 350(C).
    16. Homod, Raad Z. & Togun, Hussein & Kadhim Hussein, Ahmed & Noraldeen Al-Mousawi, Fadhel & Yaseen, Zaher Mundher & Al-Kouz, Wael & Abd, Haider J. & Alawi, Omer A. & Goodarzi, Marjan & Hussein, Omar A., 2022. "Dynamics analysis of a novel hybrid deep clustering for unsupervised learning by reinforcement of multi-agent to energy saving in intelligent buildings," Applied Energy, Elsevier, vol. 313(C).
    17. Davide Deltetto & Davide Coraci & Giuseppe Pinto & Marco Savino Piscitelli & Alfonso Capozzoli, 2021. "Exploring the Potentialities of Deep Reinforcement Learning for Incentive-Based Demand Response in a Cluster of Small Commercial Buildings," Energies, MDPI, vol. 14(10), pages 1-25, May.
    18. Ding, Yan & Zhang, Haozheng & Yang, Xiaochen & Tian, Zhe & Huang, Chen, 2024. "An adaptive switching control model for air conditioning systems based on information completeness," Applied Energy, Elsevier, vol. 375(C).
    19. Chen, Minghao & Sun, Yi & Xie, Zhiyuan & Lin, Nvgui & Wu, Peng, 2023. "An efficient and privacy-preserving algorithm for multiple energy hubs scheduling with federated and matching deep reinforcement learning," Energy, Elsevier, vol. 284(C).
    20. Hu, Zehuan & Gao, Yuan & Sun, Luning & Mae, Masayuki & Imaizumi, Taiji, 2024. "Improved robust model predictive control for residential building air conditioning and photovoltaic power generation with battery energy storage system under weather forecast uncertainty," Applied Energy, Elsevier, vol. 371(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:346:y:2023:i:c:s0306261923006876. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.