IDEAS home Printed from https://ideas.repec.org/a/eee/rensus/v202y2024ics1364032124003745.html
   My bibliography  Save this article

An overview of reinforcement learning-based approaches for smart home energy management systems with energy storages

Author

Listed:
  • Pinthurat, Watcharakorn
  • Surinkaew, Tossaporn
  • Hredzak, Branislav

Abstract

The paper’s state-of-the-art review focuses on an in-depth evaluation of smart home energy management systems which employ reinforcement learning-based methods to integrate energy storages. In order to optimize energy consumption and improve overall sustainability while maintaining technical and economic constraints, the paper first investigates the multi-faceted aspects of integrating energy storages into smart homes. Second, an overview of a smart home system and a theoretical background of reinforcement learning-based algorithms are given and discussed. Consequently, this study delves into the challenges and benefits of integrating energy storage, specifically looking at ways to lessen the impact of renewable sources’ intermittency, improve grid stability, and streamline efficient energy storage management. Thirdly, the paper highlights the beneficial features of smart home energy storage integration, including reduced costs, increased system resilience, and improved energy efficiency. Therefore, cutting-edge reinforcement learning-based methods utilized in smart home energy management systems that incorporate energy storage are thoroughly examined by evaluating their effectiveness and adaptability, taking into account both multi-agent and single-agent reinforcement learning-based methods. Finally, the study identifies potential research directions, including the development of hybrid reinforcement learning algorithms, integration of demand-side management strategies, and addressing privacy and security concerns in reinforcement learning-based smart home energy management systems. While some research has made use of single-agent reinforcement learning, smart home energy storage systems that use energy storages seldom use multi-agent reinforcement learning techniques. Researchers, practitioners, and policymakers will be able to use this work as a foundation to build smart, sustainable home energy systems.

Suggested Citation

  • Pinthurat, Watcharakorn & Surinkaew, Tossaporn & Hredzak, Branislav, 2024. "An overview of reinforcement learning-based approaches for smart home energy management systems with energy storages," Renewable and Sustainable Energy Reviews, Elsevier, vol. 202(C).
    • Handle: RePEc:eee:rensus:v:202:y:2024:i:c:s1364032124003745
    DOI: 10.1016/j.rser.2024.114648
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1364032124003745
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.rser.2024.114648?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Qiu, Dawei & Xue, Juxing & Zhang, Tingqi & Wang, Jianhong & Sun, Mingyang, 2023. "Federated reinforcement learning for smart building joint peer-to-peer energy and carbon allowance trading," Applied Energy, Elsevier, vol. 333(C).
    2. Dinh, Huy Truong & Lee, Kyu-haeng & Kim, Daehee, 2022. "Supervised-learning-based hour-ahead demand response for a behavior-based home energy management system approximating MILP optimization," Applied Energy, Elsevier, vol. 321(C).
    3. Zeng, Lanting & Qiu, Dawei & Sun, Mingyang, 2022. "Resilience enhancement of multi-agent reinforcement learning-based demand response against adversarial attacks," Applied Energy, Elsevier, vol. 324(C).
    4. Allegretti, G. & Montoya, M.A. & Bertussi, L.A.S. & Talamini, E., 2022. "When being renewable may not be enough: Typologies of trends in energy and carbon footprint towards sustainable development," Renewable and Sustainable Energy Reviews, Elsevier, vol. 168(C).
    5. Langer, Lissy & Volling, Thomas, 2022. "A reinforcement learning approach to home energy management for modulating heat pumps and photovoltaic systems," Applied Energy, Elsevier, vol. 327(C).
    6. Ronggang Zhang & Sathishkumar V E & R. Dinesh Jackson Samuel, 2020. "Fuzzy Efficient Energy Smart Home Management System for Renewable Energy Resources," Sustainability, MDPI, vol. 12(8), pages 1-14, April.
    7. Kim, Hakpyeong & Choi, Heeju & Kang, Hyuna & An, Jongbaek & Yeom, Seungkeun & Hong, Taehoon, 2021. "A systematic review of the smart energy conservation system: From smart homes to sustainable smart cities," Renewable and Sustainable Energy Reviews, Elsevier, vol. 140(C).
    8. Ahmadi, Mohammad Mahdi & Keyhani, Alireza & Rosen, Marc A. & Lam, Su Shiung & Pan, Junting & Tabatabaei, Meisam & Aghbashlo, Mortaza, 2022. "Towards sustainable net-zero districts using the extended exergy accounting concept," Renewable Energy, Elsevier, vol. 197(C), pages 747-764.
    9. Qiu, Dawei & Ye, Yujian & Papadaskalopoulos, Dimitrios & Strbac, Goran, 2021. "Scalable coordinated management of peer-to-peer energy trading: A multi-cluster deep reinforcement learning approach," Applied Energy, Elsevier, vol. 292(C).
    10. Izanloo, Milad & Noorollahi, Younes & Aslani, Alireza, 2021. "Future energy planning to maximize renewable energy share for the south Caspian Sea climate," Renewable Energy, Elsevier, vol. 175(C), pages 660-675.
    11. Isaías Gomes & Karol Bot & Maria Graça Ruano & António Ruano, 2022. "Recent Techniques Used in Home Energy Management Systems: A Review," Energies, MDPI, vol. 15(8), pages 1-41, April.
    12. Li, Yang & Wang, Ruinong & Li, Yuanzheng & Zhang, Meng & Long, Chao, 2023. "Wind power forecasting considering data privacy protection: A federated deep reinforcement learning approach," Applied Energy, Elsevier, vol. 329(C).
    13. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    14. Joel Alpízar-Castillo & Laura Ramirez-Elizondo & Pavol Bauer, 2022. "Assessing the Role of Energy Storage in Multiple Energy Carriers toward Providing Ancillary Services: A Review," Energies, MDPI, vol. 16(1), pages 1-31, December.
    15. Kamran Taghizad-Tavana & Mohsen Ghanbari-Ghalehjoughi & Nazila Razzaghi-Asl & Sayyad Nojavan & As’ad Alizadeh, 2022. "An Overview of the Architecture of Home Energy Management System as Microgrids, Automation Systems, Communication Protocols, Security, and Cyber Challenges," Sustainability, MDPI, vol. 14(23), pages 1-23, November.
    16. Rohit Trivedi & Sandipan Patra & Yousra Sidqi & Benjamin Bowler & Fiona Zimmermann & Geert Deconinck & Antonios Papaemmanouil & Shafi Khadem, 2022. "Community-Based Microgrids: Literature Review and Pathways to Decarbonise the Local Electricity Network," Energies, MDPI, vol. 15(3), pages 1-30, January.
    17. Aparna Kumari & Riya Kakkar & Rajesh Gupta & Smita Agrawal & Sudeep Tanwar & Fayez Alqahtani & Amr Tolba & Maria Simona Raboaca & Daniela Lucia Manea, 2023. "Blockchain-Driven Real-Time Incentive Approach for Energy Management System," Mathematics, MDPI, vol. 11(4), pages 1-17, February.
    18. Xie, Jiahan & Ajagekar, Akshay & You, Fengqi, 2023. "Multi-Agent attention-based deep reinforcement learning for demand response in grid-responsive buildings," Applied Energy, Elsevier, vol. 342(C).
    19. Lee, Sangyoon & Choi, Dae-Hyun, 2021. "Dynamic pricing and energy management for profit maximization in multiple smart electric vehicle charging stations: A privacy-preserving deep reinforcement learning approach," Applied Energy, Elsevier, vol. 304(C).
    20. Yang, Ting & Zhao, Liyuan & Li, Wei & Wu, Jianzhong & Zomaya, Albert Y., 2021. "Towards healthy and cost-effective indoor environment management in smart homes: A deep reinforcement learning approach," Applied Energy, Elsevier, vol. 300(C).
    21. Maksim Balashov & Anton Kiselev & Alena Kuryleva, 2021. "Reinforcement Learning Approach for Dynamic Pricing," Studies on Entrepreneurship, Structural Change and Industrial Dynamics, in: Tessaleno Devezas & João Leitão & Askar Sarygulov (ed.), The Economics of Digital Transformation, edition 1, pages 123-141, Springer.
    22. Shen, Rendong & Zhong, Shengyuan & Wen, Xin & An, Qingsong & Zheng, Ruifan & Li, Yang & Zhao, Jun, 2022. "Multi-agent deep reinforcement learning optimization framework for building energy system with renewable energy," Applied Energy, Elsevier, vol. 312(C).
    23. Amit Shewale & Anil Mokhade & Nitesh Funde & Neeraj Dhanraj Bokde, 2022. "A Survey of Efficient Demand-Side Management Techniques for the Residential Appliance Scheduling Problem in Smart Homes," Energies, MDPI, vol. 15(8), pages 1-34, April.
    24. Wenying Li & Ming Tang & Xinzhen Zhang & Danhui Gao & Jian Wang, 2021. "Operation of Distributed Battery Considering Demand Response Using Deep Reinforcement Learning in Grid Edge Control," Energies, MDPI, vol. 14(22), pages 1-18, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    2. Homod, Raad Z. & Togun, Hussein & Kadhim Hussein, Ahmed & Noraldeen Al-Mousawi, Fadhel & Yaseen, Zaher Mundher & Al-Kouz, Wael & Abd, Haider J. & Alawi, Omer A. & Goodarzi, Marjan & Hussein, Omar A., 2022. "Dynamics analysis of a novel hybrid deep clustering for unsupervised learning by reinforcement of multi-agent to energy saving in intelligent buildings," Applied Energy, Elsevier, vol. 313(C).
    3. Wenya Xu & Yanxue Li & Guanjie He & Yang Xu & Weijun Gao, 2023. "Performance Assessment and Comparative Analysis of Photovoltaic-Battery System Scheduling in an Existing Zero-Energy House Based on Reinforcement Learning Control," Energies, MDPI, vol. 16(13), pages 1-19, June.
    4. Wang, Zixuan & Xiao, Fu & Ran, Yi & Li, Yanxue & Xu, Yang, 2024. "Scalable energy management approach of residential hybrid energy system using multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 367(C).
    5. Qiu, Dawei & Wang, Yi & Hua, Weiqi & Strbac, Goran, 2023. "Reinforcement learning for electric vehicle applications in power systems:A critical review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 173(C).
    6. Chen, Bingyang & Zeng, Xingjie & Zhang, Weishan & Fan, Lulu & Cao, Shaohua & Zhou, Jiehan, 2023. "Knowledge sharing-based multi-block federated learning for few-shot oil layer identification," Energy, Elsevier, vol. 283(C).
    7. Harrold, Daniel J.B. & Cao, Jun & Fan, Zhong, 2022. "Renewable energy integration and microgrid energy trading using multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 318(C).
    8. Jiankai Gao & Yang Li & Bin Wang & Haibo Wu, 2023. "Multi-Microgrid Collaborative Optimization Scheduling Using an Improved Multi-Agent Soft Actor-Critic Algorithm," Energies, MDPI, vol. 16(7), pages 1-21, April.
    9. Samende, Cephas & Cao, Jun & Fan, Zhong, 2022. "Multi-agent deep deterministic policy gradient algorithm for peer-to-peer energy trading considering distribution network constraints," Applied Energy, Elsevier, vol. 317(C).
    10. Qiu, Dawei & Wang, Yi & Wang, Junkai & Jiang, Chuanwen & Strbac, Goran, 2023. "Personalized retail pricing design for smart metering consumers in electricity market," Applied Energy, Elsevier, vol. 348(C).
    11. Shama Naz Islam, 2024. "A Review of Peer-to-Peer Energy Trading Markets: Enabling Models and Technologies," Energies, MDPI, vol. 17(7), pages 1-18, April.
    12. Javier Parra-Domínguez & Esteban Sánchez & Ángel Ordóñez, 2023. "The Prosumer: A Systematic Review of the New Paradigm in Energy and Sustainable Development," Sustainability, MDPI, vol. 15(13), pages 1-44, July.
    13. Xie, Jiahan & Ajagekar, Akshay & You, Fengqi, 2023. "Multi-Agent attention-based deep reinforcement learning for demand response in grid-responsive buildings," Applied Energy, Elsevier, vol. 342(C).
    14. Ayas Shaqour & Aya Hagishima, 2022. "Systematic Review on Deep Reinforcement Learning-Based Energy Management for Different Building Types," Energies, MDPI, vol. 15(22), pages 1-27, November.
    15. Qiu, Dawei & Dong, Zihang & Zhang, Xi & Wang, Yi & Strbac, Goran, 2022. "Safe reinforcement learning for real-time automatic control in a smart energy-hub," Applied Energy, Elsevier, vol. 309(C).
    16. Gao, Yuan & Matsunami, Yuki & Miyata, Shohei & Akashi, Yasunori, 2022. "Multi-agent reinforcement learning dealing with hybrid action spaces: A case study for off-grid oriented renewable building energy system," Applied Energy, Elsevier, vol. 326(C).
    17. Qiu, Dawei & Wang, Yi & Sun, Mingyang & Strbac, Goran, 2022. "Multi-service provision for electric vehicles in power-transportation networks towards a low-carbon transition: A hierarchical and hybrid multi-agent reinforcement learning approach," Applied Energy, Elsevier, vol. 313(C).
    18. Xia, Qinqin & Wang, Yu & Zou, Yao & Yan, Ziming & Zhou, Niancheng & Chi, Yuan & Wang, Qianggang, 2024. "Regional-privacy-preserving operation of networked microgrids: Edge-cloud cooperative learning with differentiated policies," Applied Energy, Elsevier, vol. 370(C).
    19. Zhao, Zhonghao & Lee, Carman K.M. & Yan, Xiaoyuan & Wang, Haonan, 2024. "Reinforcement learning for electric vehicle charging scheduling: A systematic review," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 190(C).
    20. Zhou, Yanting & Ma, Zhongjing & Shi, Xingyu & Zou, Suli, 2024. "Multi-agent optimal scheduling for integrated energy system considering the global carbon emission constraint," Energy, Elsevier, vol. 288(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:rensus:v:202:y:2024:i:c:s1364032124003745. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/600126/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.