IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v327y2022ics0306261922012776.html
   My bibliography  Save this article

A reinforcement learning approach to home energy management for modulating heat pumps and photovoltaic systems

Author

Listed:
  • Langer, Lissy
  • Volling, Thomas

Abstract

Buildings are one of the main drivers of global energy consumption and CO2 emissions. Efficient energy management systems will have to integrate renewable energy sources with heating and/or cooling to mitigate climate change. In this study, we analyze the potential of deep reinforcement learning (DRL) to control a smart home with a modulating air-to-water heat pump, a photovoltaic system, a battery energy, and a thermal storage system for floor heating and hot water supply. We transform a mixed-integer linear program (MILP) into a DRL implementation. In our numerical analysis, we compare our results based on the deep deterministic policy gradient (DDPG) algorithm to the theoretical upper bound of the model predictive control (MPC) result under full information, as well as a practice-oriented rule-based benchmark. We show that our proposed DRL implementation outperforms the rule-based approach and achieves a self-sufficiency of 75% with only limited comfort violations. Analyzing different DRL formulations, we conclude that domain knowledge is key to formalizing an efficient decision problem with stable results. Our input data and models, developed using the Julia programming language, are available open source.

Suggested Citation

  • Langer, Lissy & Volling, Thomas, 2022. "A reinforcement learning approach to home energy management for modulating heat pumps and photovoltaic systems," Applied Energy, Elsevier, vol. 327(C).
    • Handle: RePEc:eee:appene:v:327:y:2022:i:c:s0306261922012776
    DOI: 10.1016/j.apenergy.2022.120020
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261922012776
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2022.120020?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Vázquez-Canteli, José R. & Nagy, Zoltán, 2019. "Reinforcement learning for demand response: A review of algorithms and modeling techniques," Applied Energy, Elsevier, vol. 235(C), pages 1072-1089.
    2. Langer, Lissy & Volling, Thomas, 2020. "An optimal home energy management system for modulating heat pumps and photovoltaic systems," Applied Energy, Elsevier, vol. 278(C).
    3. Pfenninger, Stefan & Staffell, Iain, 2016. "Long-term patterns of European PV output using 30 years of validated hourly reanalysis and satellite data," Energy, Elsevier, vol. 114(C), pages 1251-1265.
    4. Wang, Zhe & Hong, Tianzhen, 2020. "Reinforcement learning for building controls: The opportunities and challenges," Applied Energy, Elsevier, vol. 269(C).
    5. Fischer, David & Madani, Hatef, 2017. "On heat pumps in smart grids: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 70(C), pages 342-357.
    6. Lissy Langer, 2020. "An Optimal Peer-to-Peer Market Considering Modulating Heat Pumps and Photovoltaic Systems under the German Levy Regime," Energies, MDPI, vol. 13(20), pages 1-25, October.
    7. Dengiz, Thomas & Jochem, Patrick & Fichtner, Wolf, 2019. "Demand response with heuristic control strategies for modulating heat pumps," Applied Energy, Elsevier, vol. 238(C), pages 1346-1360.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Pinthurat, Watcharakorn & Surinkaew, Tossaporn & Hredzak, Branislav, 2024. "An overview of reinforcement learning-based approaches for smart home energy management systems with energy storages," Renewable and Sustainable Energy Reviews, Elsevier, vol. 202(C).
    2. Michael Bachseitz & Muhammad Sheryar & David Schmitt & Thorsten Summ & Christoph Trinkl & Wilfried Zörner, 2024. "PV-Optimized Heat Pump Control in Multi-Family Buildings Using a Reinforcement Learning Approach," Energies, MDPI, vol. 17(8), pages 1-16, April.
    3. Mohammed Qais & K. H. Loo & Hany M. Hasanien & Saad Alghuwainem, 2023. "Optimal Comfortable Load Schedule for Home Energy Management Including Photovoltaic and Battery Systems," Sustainability, MDPI, vol. 15(12), pages 1-15, June.
    4. Dominik Latoń & Jakub Grela & Andrzej Ożadowicz, 2024. "Applications of Deep Reinforcement Learning for Home Energy Management Systems: A Review," Energies, MDPI, vol. 17(24), pages 1-30, December.
    5. Schmitz, Simon & Brucke, Karoline & Kasturi, Pranay & Ansari, Esmail & Klement, Peter, 2024. "Forecast-based and data-driven reinforcement learning for residential heat pump operation," Applied Energy, Elsevier, vol. 371(C).
    6. Yin, Linfei & Xiong, Yi, 2024. "Fast-apply deep autoregressive recurrent proximal policy optimization for controlling hot water systems," Applied Energy, Elsevier, vol. 367(C).
    7. Zhang, Yiwen & Lin, Rui & Mei, Zhen & Lyu, Minghao & Jiang, Huaiguang & Xue, Ying & Zhang, Jun & Gao, David Wenzhong, 2024. "Interior-point policy optimization based multi-agent deep reinforcement learning method for secure home energy management under various uncertainties," Applied Energy, Elsevier, vol. 376(PA).
    8. Wenya Xu & Yanxue Li & Guanjie He & Yang Xu & Weijun Gao, 2023. "Performance Assessment and Comparative Analysis of Photovoltaic-Battery System Scheduling in an Existing Zero-Energy House Based on Reinforcement Learning Control," Energies, MDPI, vol. 16(13), pages 1-19, June.
    9. Heidari, Amirreza & Girardin, Luc & Dorsaz, Cédric & Maréchal, François, 2025. "A trustworthy reinforcement learning framework for autonomous control of a large-scale complex heating system: Simulation and field implementation," Applied Energy, Elsevier, vol. 378(PA).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Langer, Lissy & Volling, Thomas, 2020. "An optimal home energy management system for modulating heat pumps and photovoltaic systems," Applied Energy, Elsevier, vol. 278(C).
    2. Lissy Langer, 2020. "An Optimal Peer-to-Peer Market Considering Modulating Heat Pumps and Photovoltaic Systems under the German Levy Regime," Energies, MDPI, vol. 13(20), pages 1-25, October.
    3. Guo, Yurun & Wang, Shugang & Wang, Jihong & Zhang, Tengfei & Ma, Zhenjun & Jiang, Shuang, 2024. "Key district heating technologies for building energy flexibility: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 189(PB).
    4. Gokhale, Gargya & Claessens, Bert & Develder, Chris, 2022. "Physics informed neural networks for control oriented thermal modeling of buildings," Applied Energy, Elsevier, vol. 314(C).
    5. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    6. Pinto, Giuseppe & Piscitelli, Marco Savino & Vázquez-Canteli, José Ramón & Nagy, Zoltán & Capozzoli, Alfonso, 2021. "Coordinated energy management for a cluster of buildings through deep reinforcement learning," Energy, Elsevier, vol. 229(C).
    7. Pinto, Giuseppe & Deltetto, Davide & Capozzoli, Alfonso, 2021. "Data-driven district energy management with surrogate models and deep reinforcement learning," Applied Energy, Elsevier, vol. 304(C).
    8. Pinto, Giuseppe & Kathirgamanathan, Anjukan & Mangina, Eleni & Finn, Donal P. & Capozzoli, Alfonso, 2022. "Enhancing energy management in grid-interactive buildings: A comparison among cooperative and coordinated architectures," Applied Energy, Elsevier, vol. 310(C).
    9. Hindreen Rashid Abdulqadir & Adnan Mohsin Abdulazeez, 2021. "Reinforcement Learning and Modeling Techniques: A Review," International Journal of Science and Business, IJSAB International, vol. 5(3), pages 174-189.
    10. Touzani, Samir & Prakash, Anand Krishnan & Wang, Zhe & Agarwal, Shreya & Pritoni, Marco & Kiran, Mariam & Brown, Richard & Granderson, Jessica, 2021. "Controlling distributed energy resources via deep reinforcement learning for load flexibility and energy efficiency," Applied Energy, Elsevier, vol. 304(C).
    11. Guo, Fangzhou & Ham, Sang woo & Kim, Donghun & Moon, Hyeun Jun, 2025. "Deep reinforcement learning control for co-optimizing energy consumption, thermal comfort, and indoor air quality in an office building," Applied Energy, Elsevier, vol. 377(PA).
    12. Homod, Raad Z. & Togun, Hussein & Kadhim Hussein, Ahmed & Noraldeen Al-Mousawi, Fadhel & Yaseen, Zaher Mundher & Al-Kouz, Wael & Abd, Haider J. & Alawi, Omer A. & Goodarzi, Marjan & Hussein, Omar A., 2022. "Dynamics analysis of a novel hybrid deep clustering for unsupervised learning by reinforcement of multi-agent to energy saving in intelligent buildings," Applied Energy, Elsevier, vol. 313(C).
    13. Davide Deltetto & Davide Coraci & Giuseppe Pinto & Marco Savino Piscitelli & Alfonso Capozzoli, 2021. "Exploring the Potentialities of Deep Reinforcement Learning for Incentive-Based Demand Response in a Cluster of Small Commercial Buildings," Energies, MDPI, vol. 14(10), pages 1-25, May.
    14. Liu, Shuo & Liu, Xiaohua & Zhang, Tao & Wang, Chaoliang & Liu, Wei, 2024. "Joint optimization for temperature and humidity independent control system based on multi-agent reinforcement learning with cooperative mechanisms," Applied Energy, Elsevier, vol. 375(C).
    15. Michael Bachseitz & Muhammad Sheryar & David Schmitt & Thorsten Summ & Christoph Trinkl & Wilfried Zörner, 2024. "PV-Optimized Heat Pump Control in Multi-Family Buildings Using a Reinforcement Learning Approach," Energies, MDPI, vol. 17(8), pages 1-16, April.
    16. Lilia Tightiz & Joon Yoo, 2022. "A Review on a Data-Driven Microgrid Management System Integrating an Active Distribution Network: Challenges, Issues, and New Trends," Energies, MDPI, vol. 15(22), pages 1-24, November.
    17. Biemann, Marco & Scheller, Fabian & Liu, Xiufeng & Huang, Lizhen, 2021. "Experimental evaluation of model-free reinforcement learning algorithms for continuous HVAC control," Applied Energy, Elsevier, vol. 298(C).
    18. Xie, Jiahan & Ajagekar, Akshay & You, Fengqi, 2023. "Multi-Agent attention-based deep reinforcement learning for demand response in grid-responsive buildings," Applied Energy, Elsevier, vol. 342(C).
    19. Finck, Christian & Li, Rongling & Zeiler, Wim, 2019. "Economic model predictive control for demand flexibility of a residential building," Energy, Elsevier, vol. 176(C), pages 365-379.
    20. Han, Gwangwoo & Joo, Hong-Jin & Lim, Hee-Won & An, Young-Sub & Lee, Wang-Je & Lee, Kyoung-Ho, 2023. "Data-driven heat pump operation strategy using rainbow deep reinforcement learning for significant reduction of electricity cost," Energy, Elsevier, vol. 270(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:327:y:2022:i:c:s0306261922012776. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.