IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v327y2022ics0306261922012776.html
   My bibliography  Save this article

A reinforcement learning approach to home energy management for modulating heat pumps and photovoltaic systems

Author

Listed:
  • Langer, Lissy
  • Volling, Thomas

Abstract

Buildings are one of the main drivers of global energy consumption and CO2 emissions. Efficient energy management systems will have to integrate renewable energy sources with heating and/or cooling to mitigate climate change. In this study, we analyze the potential of deep reinforcement learning (DRL) to control a smart home with a modulating air-to-water heat pump, a photovoltaic system, a battery energy, and a thermal storage system for floor heating and hot water supply. We transform a mixed-integer linear program (MILP) into a DRL implementation. In our numerical analysis, we compare our results based on the deep deterministic policy gradient (DDPG) algorithm to the theoretical upper bound of the model predictive control (MPC) result under full information, as well as a practice-oriented rule-based benchmark. We show that our proposed DRL implementation outperforms the rule-based approach and achieves a self-sufficiency of 75% with only limited comfort violations. Analyzing different DRL formulations, we conclude that domain knowledge is key to formalizing an efficient decision problem with stable results. Our input data and models, developed using the Julia programming language, are available open source.

Suggested Citation

  • Langer, Lissy & Volling, Thomas, 2022. "A reinforcement learning approach to home energy management for modulating heat pumps and photovoltaic systems," Applied Energy, Elsevier, vol. 327(C).
    • Handle: RePEc:eee:appene:v:327:y:2022:i:c:s0306261922012776
    DOI: 10.1016/j.apenergy.2022.120020
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261922012776
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2022.120020?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Wang, Zhe & Hong, Tianzhen, 2020. "Reinforcement learning for building controls: The opportunities and challenges," Applied Energy, Elsevier, vol. 269(C).
    2. Dengiz, Thomas & Jochem, Patrick & Fichtner, Wolf, 2019. "Demand response with heuristic control strategies for modulating heat pumps," Applied Energy, Elsevier, vol. 238(C), pages 1346-1360.
    3. Lissy Langer, 2020. "An Optimal Peer-to-Peer Market Considering Modulating Heat Pumps and Photovoltaic Systems under the German Levy Regime," Energies, MDPI, vol. 13(20), pages 1-25, October.
    4. Langer, Lissy & Volling, Thomas, 2020. "An optimal home energy management system for modulating heat pumps and photovoltaic systems," Applied Energy, Elsevier, vol. 278(C).
    5. Fischer, David & Madani, Hatef, 2017. "On heat pumps in smart grids: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 70(C), pages 342-357.
    6. Vázquez-Canteli, José R. & Nagy, Zoltán, 2019. "Reinforcement learning for demand response: A review of algorithms and modeling techniques," Applied Energy, Elsevier, vol. 235(C), pages 1072-1089.
    7. Pfenninger, Stefan & Staffell, Iain, 2016. "Long-term patterns of European PV output using 30 years of validated hourly reanalysis and satellite data," Energy, Elsevier, vol. 114(C), pages 1251-1265.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Mohammed Qais & K. H. Loo & Hany M. Hasanien & Saad Alghuwainem, 2023. "Optimal Comfortable Load Schedule for Home Energy Management Including Photovoltaic and Battery Systems," Sustainability, MDPI, vol. 15(12), pages 1-15, June.
    2. Schmitz, Simon & Brucke, Karoline & Kasturi, Pranay & Ansari, Esmail & Klement, Peter, 2024. "Forecast-based and data-driven reinforcement learning for residential heat pump operation," Applied Energy, Elsevier, vol. 371(C).
    3. Yin, Linfei & Xiong, Yi, 2024. "Fast-apply deep autoregressive recurrent proximal policy optimization for controlling hot water systems," Applied Energy, Elsevier, vol. 367(C).
    4. Zhang, Yiwen & Lin, Rui & Mei, Zhen & Lyu, Minghao & Jiang, Huaiguang & Xue, Ying & Zhang, Jun & Gao, David Wenzhong, 2024. "Interior-point policy optimization based multi-agent deep reinforcement learning method for secure home energy management under various uncertainties," Applied Energy, Elsevier, vol. 376(PA).
    5. Pinthurat, Watcharakorn & Surinkaew, Tossaporn & Hredzak, Branislav, 2024. "An overview of reinforcement learning-based approaches for smart home energy management systems with energy storages," Renewable and Sustainable Energy Reviews, Elsevier, vol. 202(C).
    6. Michael Bachseitz & Muhammad Sheryar & David Schmitt & Thorsten Summ & Christoph Trinkl & Wilfried Zörner, 2024. "PV-Optimized Heat Pump Control in Multi-Family Buildings Using a Reinforcement Learning Approach," Energies, MDPI, vol. 17(8), pages 1-16, April.
    7. Dominik Latoń & Jakub Grela & Andrzej Ożadowicz, 2024. "Applications of Deep Reinforcement Learning for Home Energy Management Systems: A Review," Energies, MDPI, vol. 17(24), pages 1-30, December.
    8. Wenya Xu & Yanxue Li & Guanjie He & Yang Xu & Weijun Gao, 2023. "Performance Assessment and Comparative Analysis of Photovoltaic-Battery System Scheduling in an Existing Zero-Energy House Based on Reinforcement Learning Control," Energies, MDPI, vol. 16(13), pages 1-19, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Langer, Lissy & Volling, Thomas, 2020. "An optimal home energy management system for modulating heat pumps and photovoltaic systems," Applied Energy, Elsevier, vol. 278(C).
    2. Lissy Langer, 2020. "An Optimal Peer-to-Peer Market Considering Modulating Heat Pumps and Photovoltaic Systems under the German Levy Regime," Energies, MDPI, vol. 13(20), pages 1-25, October.
    3. Guo, Yurun & Wang, Shugang & Wang, Jihong & Zhang, Tengfei & Ma, Zhenjun & Jiang, Shuang, 2024. "Key district heating technologies for building energy flexibility: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 189(PB).
    4. Gokhale, Gargya & Claessens, Bert & Develder, Chris, 2022. "Physics informed neural networks for control oriented thermal modeling of buildings," Applied Energy, Elsevier, vol. 314(C).
    5. Pinto, Giuseppe & Deltetto, Davide & Capozzoli, Alfonso, 2021. "Data-driven district energy management with surrogate models and deep reinforcement learning," Applied Energy, Elsevier, vol. 304(C).
    6. Michael Bachseitz & Muhammad Sheryar & David Schmitt & Thorsten Summ & Christoph Trinkl & Wilfried Zörner, 2024. "PV-Optimized Heat Pump Control in Multi-Family Buildings Using a Reinforcement Learning Approach," Energies, MDPI, vol. 17(8), pages 1-16, April.
    7. Lilia Tightiz & Joon Yoo, 2022. "A Review on a Data-Driven Microgrid Management System Integrating an Active Distribution Network: Challenges, Issues, and New Trends," Energies, MDPI, vol. 15(22), pages 1-24, November.
    8. Biemann, Marco & Scheller, Fabian & Liu, Xiufeng & Huang, Lizhen, 2021. "Experimental evaluation of model-free reinforcement learning algorithms for continuous HVAC control," Applied Energy, Elsevier, vol. 298(C).
    9. Xie, Jiahan & Ajagekar, Akshay & You, Fengqi, 2023. "Multi-Agent attention-based deep reinforcement learning for demand response in grid-responsive buildings," Applied Energy, Elsevier, vol. 342(C).
    10. Finck, Christian & Li, Rongling & Zeiler, Wim, 2019. "Economic model predictive control for demand flexibility of a residential building," Energy, Elsevier, vol. 176(C), pages 365-379.
    11. Perera, A.T.D. & Kamalaruban, Parameswaran, 2021. "Applications of reinforcement learning in energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 137(C).
    12. Caputo, Cesare & Cardin, Michel-Alexandre & Ge, Pudong & Teng, Fei & Korre, Anna & Antonio del Rio Chanona, Ehecatl, 2023. "Design and planning of flexible mobile Micro-Grids using Deep Reinforcement Learning," Applied Energy, Elsevier, vol. 335(C).
    13. Meesenburg, Wiebke & Markussen, Wiebke Brix & Ommen, Torben & Elmegaard, Brian, 2020. "Optimizing control of two-stage ammonia heat pump for fast regulation of power uptake," Applied Energy, Elsevier, vol. 271(C).
    14. Svetozarevic, B. & Baumann, C. & Muntwiler, S. & Di Natale, L. & Zeilinger, M.N. & Heer, P., 2022. "Data-driven control of room temperature and bidirectional EV charging using deep reinforcement learning: Simulations and experiments," Applied Energy, Elsevier, vol. 307(C).
    15. Gao, Yuan & Matsunami, Yuki & Miyata, Shohei & Akashi, Yasunori, 2022. "Operational optimization for off-grid renewable building energy system using deep reinforcement learning," Applied Energy, Elsevier, vol. 325(C).
    16. Efkarpidis, Nikolaos A. & Vomva, Styliani A. & Christoforidis, Georgios C. & Papagiannis, Grigoris K., 2022. "Optimal day-to-day scheduling of multiple energy assets in residential buildings equipped with variable-speed heat pumps," Applied Energy, Elsevier, vol. 312(C).
    17. Sun, Fangyuan & Kong, Xiangyu & Wu, Jianzhong & Gao, Bixuan & Chen, Ke & Lu, Ning, 2022. "DSM pricing method based on A3C and LSTM under cloud-edge environment," Applied Energy, Elsevier, vol. 315(C).
    18. Maximilian Schulz & Thomas Kemmler & Julia Kumm & Kai Hufendiek & Bernd Thomas, 2020. "A More Realistic Heat Pump Control Approach by Application of an Integrated Two-Part Control," Energies, MDPI, vol. 13(11), pages 1-22, June.
    19. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    20. Pinto, Giuseppe & Piscitelli, Marco Savino & Vázquez-Canteli, José Ramón & Nagy, Zoltán & Capozzoli, Alfonso, 2021. "Coordinated energy management for a cluster of buildings through deep reinforcement learning," Energy, Elsevier, vol. 229(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:327:y:2022:i:c:s0306261922012776. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.