IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v144y2018icp159-168.html
   My bibliography  Save this article

Gigawatt-hour scale savings on a budget of zero: Deep reinforcement learning based optimal control of hot water systems

Author

Listed:
  • Kazmi, Hussain
  • Mehmood, Fahad
  • Lodeweyckx, Stefan
  • Driesen, Johan

Abstract

Energy consumption for hot water production is a major draw in high efficiency buildings. Optimizing this has typically been approached from a thermodynamics perspective, decoupled from occupant influence. Furthermore, optimization usually presupposes existence of a detailed dynamics model for the hot water system. These assumptions lead to suboptimal energy efficiency in the real world. In this paper, we present a novel reinforcement learning based methodology which optimizes hot water production. The proposed methodology is completely generalizable, and does not require an offline step or human domain knowledge to build a model for the hot water vessel or the heating element. Occupant preferences too are learnt on the fly. The proposed system is applied to a set of 32 houses in the Netherlands where it reduces energy consumption for hot water production by roughly 20% with no loss of occupant comfort. Extrapolating, this translates to absolute savings of roughly 200 kWh for a single household on an annual basis. This performance can be replicated to any domestic hot water system and optimization objective, given that the fairly minimal requirements on sensor data are met. With millions of hot water systems operational worldwide, the proposed framework has the potential to reduce energy consumption in existing and new systems on a multi Gigawatt-hour scale in the years to come.

Suggested Citation

  • Kazmi, Hussain & Mehmood, Fahad & Lodeweyckx, Stefan & Driesen, Johan, 2018. "Gigawatt-hour scale savings on a budget of zero: Deep reinforcement learning based optimal control of hot water systems," Energy, Elsevier, vol. 144(C), pages 159-168.
    • Handle: RePEc:eee:energy:v:144:y:2018:i:c:p:159-168
    DOI: 10.1016/j.energy.2017.12.019
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0360544217320388
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.energy.2017.12.019?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Kazmi, H. & D’Oca, S. & Delmastro, C. & Lodeweyckx, S. & Corgnati, S.P., 2016. "Generalizable occupant-driven optimization model for domestic hot water production in NZEB," Applied Energy, Elsevier, vol. 175(C), pages 1-15.
    2. Frederik Ruelens & Sandro Iacovella & Bert J. Claessens & Ronnie Belmans, 2015. "Learning Agent for a Heat-Pump Thermostat with a Set-Back Strategy Using Model-Free Reinforcement Learning," Energies, MDPI, vol. 8(8), pages 1-19, August.
    3. E. L. Lawler & D. E. Wood, 1966. "Branch-and-Bound Methods: A Survey," Operations Research, INFORMS, vol. 14(4), pages 699-719, August.
    4. Chua, K.J. & Chou, S.K. & Yang, W.M., 2010. "Advances in heat pump systems: A review," Applied Energy, Elsevier, vol. 87(12), pages 3611-3624, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ashitosh Rajesh Varne & Simon Blouin & Baxter Lorenzo McIntosh Williams & David Denkenberger, 2024. "The Impact of Abrupt Sunlight Reduction Scenarios on Renewable Energy Production," Energies, MDPI, vol. 17(20), pages 1-16, October.
    2. Baxter Williams & Daniel Bishop & Patricio Gallardo & J. Geoffrey Chase, 2023. "Demand Side Management in Industrial, Commercial, and Residential Sectors: A Review of Constraints and Considerations," Energies, MDPI, vol. 16(13), pages 1-28, July.
    3. Daeil Lee & Seoryong Koo & Inseok Jang & Jonghyun Kim, 2022. "Comparison of Deep Reinforcement Learning and PID Controllers for Automatic Cold Shutdown Operation," Energies, MDPI, vol. 15(8), pages 1-25, April.
    4. Heidari, Amirreza & Maréchal, François & Khovalyg, Dolaana, 2022. "Reinforcement Learning for proactive operation of residential energy systems by learning stochastic occupant behavior and fluctuating solar energy: Balancing comfort, hygiene and energy use," Applied Energy, Elsevier, vol. 318(C).
    5. Shen, Rendong & Zhong, Shengyuan & Wen, Xin & An, Qingsong & Zheng, Ruifan & Li, Yang & Zhao, Jun, 2022. "Multi-agent deep reinforcement learning optimization framework for building energy system with renewable energy," Applied Energy, Elsevier, vol. 312(C).
    6. Vázquez-Canteli, José R. & Nagy, Zoltán, 2019. "Reinforcement learning for demand response: A review of algorithms and modeling techniques," Applied Energy, Elsevier, vol. 235(C), pages 1072-1089.
    7. Wang, Zhe & Hong, Tianzhen, 2020. "Reinforcement learning for building controls: The opportunities and challenges," Applied Energy, Elsevier, vol. 269(C).
    8. Yin, Linfei & Xiong, Yi, 2024. "Fast-apply deep autoregressive recurrent proximal policy optimization for controlling hot water systems," Applied Energy, Elsevier, vol. 367(C).
    9. Yildiz, Baran & Bilbao, Jose I. & Roberts, Mike & Heslop, Simon & Dore, Jonathon & Bruce, Anna & MacGill, Iain & Egan, Renate J. & Sproul, Alistair B., 2021. "Analysis of electricity consumption and thermal storage of domestic electric water heating systems to utilize excess PV generation," Energy, Elsevier, vol. 235(C).
    10. Ceusters, Glenn & Rodríguez, Román Cantú & García, Alberte Bouso & Franke, Rüdiger & Deconinck, Geert & Helsen, Lieve & Nowé, Ann & Messagie, Maarten & Camargo, Luis Ramirez, 2021. "Model-predictive control and reinforcement learning in multi-energy system case studies," Applied Energy, Elsevier, vol. 303(C).
    11. Pinto, Giuseppe & Piscitelli, Marco Savino & Vázquez-Canteli, José Ramón & Nagy, Zoltán & Capozzoli, Alfonso, 2021. "Coordinated energy management for a cluster of buildings through deep reinforcement learning," Energy, Elsevier, vol. 229(C).
    12. Heidari, Amirreza & Maréchal, François & Khovalyg, Dolaana, 2022. "An occupant-centric control framework for balancing comfort, energy use and hygiene in hot water systems: A model-free reinforcement learning approach," Applied Energy, Elsevier, vol. 312(C).
    13. Kazmi, Hussain & Mehmood, Fahad & Tao, Zhenmin & Riaz, Zainab & Driesen, Johan, 2019. "Electricity load-shedding in Pakistan: Unintended consequences, opportunities and policy recommendations," Energy Policy, Elsevier, vol. 128(C), pages 411-417.
    14. Touzani, Samir & Prakash, Anand Krishnan & Wang, Zhe & Agarwal, Shreya & Pritoni, Marco & Kiran, Mariam & Brown, Richard & Granderson, Jessica, 2021. "Controlling distributed energy resources via deep reinforcement learning for load flexibility and energy efficiency," Applied Energy, Elsevier, vol. 304(C).
    15. Yin, Linfei & Gao, Qi & Zhao, Lulin & Wang, Tao, 2020. "Expandable deep learning for real-time economic generation dispatch and control of three-state energies based future smart grids," Energy, Elsevier, vol. 191(C).
    16. Seppo Sierla & Heikki Ihasalo & Valeriy Vyatkin, 2022. "A Review of Reinforcement Learning Applications to Control of Heating, Ventilation and Air Conditioning Systems," Energies, MDPI, vol. 15(10), pages 1-25, May.
    17. de Gracia, Alvaro & Tarragona, Joan & Crespo, Alicia & Fernández, Cèsar, 2020. "Smart control of dynamic phase change material wall system," Applied Energy, Elsevier, vol. 279(C).
    18. Lei, Yue & Zhan, Sicheng & Ono, Eikichi & Peng, Yuzhen & Zhang, Zhiang & Hasama, Takamasa & Chong, Adrian, 2022. "A practical deep reinforcement learning framework for multivariate occupant-centric control in buildings," Applied Energy, Elsevier, vol. 324(C).
    19. Perera, A.T.D. & Kamalaruban, Parameswaran, 2021. "Applications of reinforcement learning in energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 137(C).
    20. Zahra Fallahi & Gregor P. Henze, 2019. "Interactive Buildings: A Review," Sustainability, MDPI, vol. 11(14), pages 1-26, July.
    21. Haji Hosseinloo, Ashkan & Ryzhov, Alexander & Bischi, Aldo & Ouerdane, Henni & Turitsyn, Konstantin & Dahleh, Munther A., 2020. "Data-driven control of micro-climate in buildings: An event-triggered reinforcement learning approach," Applied Energy, Elsevier, vol. 277(C).
    22. Correa-Jullian, Camila & López Droguett, Enrique & Cardemil, José Miguel, 2020. "Operation scheduling in a solar thermal system: A reinforcement learning-based framework," Applied Energy, Elsevier, vol. 268(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wang, Zhe & Hong, Tianzhen, 2020. "Reinforcement learning for building controls: The opportunities and challenges," Applied Energy, Elsevier, vol. 269(C).
    2. Vázquez-Canteli, José R. & Nagy, Zoltán, 2019. "Reinforcement learning for demand response: A review of algorithms and modeling techniques," Applied Energy, Elsevier, vol. 235(C), pages 1072-1089.
    3. Perera, A.T.D. & Kamalaruban, Parameswaran, 2021. "Applications of reinforcement learning in energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 137(C).
    4. Sun, Fangtian & Fu, Lin & Sun, Jian & Zhang, Shigang, 2014. "A new waste heat district heating system with combined heat and power (CHP) based on ejector heat exchangers and absorption heat pumps," Energy, Elsevier, vol. 69(C), pages 516-524.
    5. Fredrik Skaug Fadnes & Reyhaneh Banihabib & Mohsen Assadi, 2023. "Using Artificial Neural Networks to Gather Intelligence on a Fully Operational Heat Pump System in an Existing Building Cluster," Energies, MDPI, vol. 16(9), pages 1-33, May.
    6. Coşar Gözükırmızı & Metin Demiralp, 2019. "Solving ODEs by Obtaining Purely Second Degree Multinomials via Branch and Bound with Admissible Heuristic," Mathematics, MDPI, vol. 7(4), pages 1-23, April.
    7. Blarke, Morten B., 2012. "Towards an intermittency-friendly energy system: Comparing electric boilers and heat pumps in distributed cogeneration," Applied Energy, Elsevier, vol. 91(1), pages 349-365.
    8. Jie, Ji & Jingyong, Cai & Wenzhu, Huang & Yan, Feng, 2015. "Experimental study on the performance of solar-assisted multi-functional heat pump based on enthalpy difference lab with solar simulator," Renewable Energy, Elsevier, vol. 75(C), pages 381-388.
    9. Yassine Chemingui & Adel Gastli & Omar Ellabban, 2020. "Reinforcement Learning-Based School Energy Management System," Energies, MDPI, vol. 13(23), pages 1-21, December.
    10. Agnieszka Malec & Tomasz Cholewa & Alicja Siuta-Olcha, 2021. "Influence of Cold Water Inlets and Obstacles on the Energy Efficiency of the Hot Water Production Process in a Hot Water Storage Tank," Energies, MDPI, vol. 14(20), pages 1-26, October.
    11. Kazmi, Hussain & Suykens, Johan & Balint, Attila & Driesen, Johan, 2019. "Multi-agent reinforcement learning for modeling and control of thermostatically controlled loads," Applied Energy, Elsevier, vol. 238(C), pages 1022-1035.
    12. Kezong Tang & Xiong-Fei Wei & Yuan-Hao Jiang & Zi-Wei Chen & Lihua Yang, 2023. "An Adaptive Ant Colony Optimization for Solving Large-Scale Traveling Salesman Problem," Mathematics, MDPI, vol. 11(21), pages 1-26, October.
    13. Waheed, M.A. & Oni, A.O. & Adejuyigbe, S.B. & Adewumi, B.A. & Fadare, D.A., 2014. "Performance enhancement of vapor recompression heat pump," Applied Energy, Elsevier, vol. 114(C), pages 69-79.
    14. Amine Lamine & Mahdi Khemakhem & Brahim Hnich & Habib Chabchoub, 2016. "Solving constrained optimization problems by solution-based decomposition search," Journal of Combinatorial Optimization, Springer, vol. 32(3), pages 672-695, October.
    15. Roberto Bruno & Francesco Nicoletti & Giorgio Cuconati & Stefania Perrella & Daniela Cirone, 2020. "Performance Indexes of an Air-Water Heat Pump Versus the Capacity Ratio: Analysis by Means of Experimental Data," Energies, MDPI, vol. 13(13), pages 1-19, July.
    16. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    17. Fouladvand, Javanshir & Aranguren Rojas, Maria & Hoppe, Thomas & Ghorbani, Amineh, 2022. "Simulating thermal energy community formation: Institutional enablers outplaying technological choice," Applied Energy, Elsevier, vol. 306(PA).
    18. Sun, Fangtian & Fu, Lin & Sun, Jian & Zhang, Shigang, 2014. "A new ejector heat exchanger based on an ejector heat pump and a water-to-water heat exchanger," Applied Energy, Elsevier, vol. 121(C), pages 245-251.
    19. Nguyen, Hiep V. & Law, Ying Lam E. & Alavy, Masih & Walsh, Philip R. & Leong, Wey H. & Dworkin, Seth B., 2014. "An analysis of the factors affecting hybrid ground-source heat pump installation potential in North America," Applied Energy, Elsevier, vol. 125(C), pages 28-38.
    20. Weiqiang Pan & Zhilong Shan & Ting Chen & Fangjiong Chen & Jing Feng, 2016. "Optimal pilot design for OFDM systems with non-contiguous subcarriers based on semi-definite programming," Telecommunication Systems: Modelling, Analysis, Design and Management, Springer, vol. 63(2), pages 297-305, October.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:144:y:2018:i:c:p:159-168. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.