IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v144y2018icp159-168.html
   My bibliography  Save this article

Gigawatt-hour scale savings on a budget of zero: Deep reinforcement learning based optimal control of hot water systems

Author

Listed:
  • Kazmi, Hussain
  • Mehmood, Fahad
  • Lodeweyckx, Stefan
  • Driesen, Johan

Abstract

Energy consumption for hot water production is a major draw in high efficiency buildings. Optimizing this has typically been approached from a thermodynamics perspective, decoupled from occupant influence. Furthermore, optimization usually presupposes existence of a detailed dynamics model for the hot water system. These assumptions lead to suboptimal energy efficiency in the real world. In this paper, we present a novel reinforcement learning based methodology which optimizes hot water production. The proposed methodology is completely generalizable, and does not require an offline step or human domain knowledge to build a model for the hot water vessel or the heating element. Occupant preferences too are learnt on the fly. The proposed system is applied to a set of 32 houses in the Netherlands where it reduces energy consumption for hot water production by roughly 20% with no loss of occupant comfort. Extrapolating, this translates to absolute savings of roughly 200 kWh for a single household on an annual basis. This performance can be replicated to any domestic hot water system and optimization objective, given that the fairly minimal requirements on sensor data are met. With millions of hot water systems operational worldwide, the proposed framework has the potential to reduce energy consumption in existing and new systems on a multi Gigawatt-hour scale in the years to come.

Suggested Citation

  • Kazmi, Hussain & Mehmood, Fahad & Lodeweyckx, Stefan & Driesen, Johan, 2018. "Gigawatt-hour scale savings on a budget of zero: Deep reinforcement learning based optimal control of hot water systems," Energy, Elsevier, vol. 144(C), pages 159-168.
    • Handle: RePEc:eee:energy:v:144:y:2018:i:c:p:159-168
    DOI: 10.1016/j.energy.2017.12.019
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0360544217320388
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.energy.2017.12.019?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Kazmi, H. & D’Oca, S. & Delmastro, C. & Lodeweyckx, S. & Corgnati, S.P., 2016. "Generalizable occupant-driven optimization model for domestic hot water production in NZEB," Applied Energy, Elsevier, vol. 175(C), pages 1-15.
    2. Frederik Ruelens & Sandro Iacovella & Bert J. Claessens & Ronnie Belmans, 2015. "Learning Agent for a Heat-Pump Thermostat with a Set-Back Strategy Using Model-Free Reinforcement Learning," Energies, MDPI, vol. 8(8), pages 1-19, August.
    3. E. L. Lawler & D. E. Wood, 1966. "Branch-and-Bound Methods: A Survey," Operations Research, INFORMS, vol. 14(4), pages 699-719, August.
    4. Chua, K.J. & Chou, S.K. & Yang, W.M., 2010. "Advances in heat pump systems: A review," Applied Energy, Elsevier, vol. 87(12), pages 3611-3624, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Baxter Williams & Daniel Bishop & Patricio Gallardo & J. Geoffrey Chase, 2023. "Demand Side Management in Industrial, Commercial, and Residential Sectors: A Review of Constraints and Considerations," Energies, MDPI, vol. 16(13), pages 1-28, July.
    2. Daeil Lee & Seoryong Koo & Inseok Jang & Jonghyun Kim, 2022. "Comparison of Deep Reinforcement Learning and PID Controllers for Automatic Cold Shutdown Operation," Energies, MDPI, vol. 15(8), pages 1-25, April.
    3. Heidari, Amirreza & Maréchal, François & Khovalyg, Dolaana, 2022. "Reinforcement Learning for proactive operation of residential energy systems by learning stochastic occupant behavior and fluctuating solar energy: Balancing comfort, hygiene and energy use," Applied Energy, Elsevier, vol. 318(C).
    4. Shen, Rendong & Zhong, Shengyuan & Wen, Xin & An, Qingsong & Zheng, Ruifan & Li, Yang & Zhao, Jun, 2022. "Multi-agent deep reinforcement learning optimization framework for building energy system with renewable energy," Applied Energy, Elsevier, vol. 312(C).
    5. Vázquez-Canteli, José R. & Nagy, Zoltán, 2019. "Reinforcement learning for demand response: A review of algorithms and modeling techniques," Applied Energy, Elsevier, vol. 235(C), pages 1072-1089.
    6. Wang, Zhe & Hong, Tianzhen, 2020. "Reinforcement learning for building controls: The opportunities and challenges," Applied Energy, Elsevier, vol. 269(C).
    7. Yildiz, Baran & Bilbao, Jose I. & Roberts, Mike & Heslop, Simon & Dore, Jonathon & Bruce, Anna & MacGill, Iain & Egan, Renate J. & Sproul, Alistair B., 2021. "Analysis of electricity consumption and thermal storage of domestic electric water heating systems to utilize excess PV generation," Energy, Elsevier, vol. 235(C).
    8. Ceusters, Glenn & Rodríguez, Román Cantú & García, Alberte Bouso & Franke, Rüdiger & Deconinck, Geert & Helsen, Lieve & Nowé, Ann & Messagie, Maarten & Camargo, Luis Ramirez, 2021. "Model-predictive control and reinforcement learning in multi-energy system case studies," Applied Energy, Elsevier, vol. 303(C).
    9. Pinto, Giuseppe & Piscitelli, Marco Savino & Vázquez-Canteli, José Ramón & Nagy, Zoltán & Capozzoli, Alfonso, 2021. "Coordinated energy management for a cluster of buildings through deep reinforcement learning," Energy, Elsevier, vol. 229(C).
    10. Heidari, Amirreza & Maréchal, François & Khovalyg, Dolaana, 2022. "An occupant-centric control framework for balancing comfort, energy use and hygiene in hot water systems: A model-free reinforcement learning approach," Applied Energy, Elsevier, vol. 312(C).
    11. Kazmi, Hussain & Mehmood, Fahad & Tao, Zhenmin & Riaz, Zainab & Driesen, Johan, 2019. "Electricity load-shedding in Pakistan: Unintended consequences, opportunities and policy recommendations," Energy Policy, Elsevier, vol. 128(C), pages 411-417.
    12. Touzani, Samir & Prakash, Anand Krishnan & Wang, Zhe & Agarwal, Shreya & Pritoni, Marco & Kiran, Mariam & Brown, Richard & Granderson, Jessica, 2021. "Controlling distributed energy resources via deep reinforcement learning for load flexibility and energy efficiency," Applied Energy, Elsevier, vol. 304(C).
    13. Yin, Linfei & Gao, Qi & Zhao, Lulin & Wang, Tao, 2020. "Expandable deep learning for real-time economic generation dispatch and control of three-state energies based future smart grids," Energy, Elsevier, vol. 191(C).
    14. Seppo Sierla & Heikki Ihasalo & Valeriy Vyatkin, 2022. "A Review of Reinforcement Learning Applications to Control of Heating, Ventilation and Air Conditioning Systems," Energies, MDPI, vol. 15(10), pages 1-25, May.
    15. de Gracia, Alvaro & Tarragona, Joan & Crespo, Alicia & Fernández, Cèsar, 2020. "Smart control of dynamic phase change material wall system," Applied Energy, Elsevier, vol. 279(C).
    16. Lei, Yue & Zhan, Sicheng & Ono, Eikichi & Peng, Yuzhen & Zhang, Zhiang & Hasama, Takamasa & Chong, Adrian, 2022. "A practical deep reinforcement learning framework for multivariate occupant-centric control in buildings," Applied Energy, Elsevier, vol. 324(C).
    17. Perera, A.T.D. & Kamalaruban, Parameswaran, 2021. "Applications of reinforcement learning in energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 137(C).
    18. Zahra Fallahi & Gregor P. Henze, 2019. "Interactive Buildings: A Review," Sustainability, MDPI, vol. 11(14), pages 1-26, July.
    19. Haji Hosseinloo, Ashkan & Ryzhov, Alexander & Bischi, Aldo & Ouerdane, Henni & Turitsyn, Konstantin & Dahleh, Munther A., 2020. "Data-driven control of micro-climate in buildings: An event-triggered reinforcement learning approach," Applied Energy, Elsevier, vol. 277(C).
    20. Correa-Jullian, Camila & López Droguett, Enrique & Cardemil, José Miguel, 2020. "Operation scheduling in a solar thermal system: A reinforcement learning-based framework," Applied Energy, Elsevier, vol. 268(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wang, Zhe & Hong, Tianzhen, 2020. "Reinforcement learning for building controls: The opportunities and challenges," Applied Energy, Elsevier, vol. 269(C).
    2. Perera, A.T.D. & Kamalaruban, Parameswaran, 2021. "Applications of reinforcement learning in energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 137(C).
    3. Vázquez-Canteli, José R. & Nagy, Zoltán, 2019. "Reinforcement learning for demand response: A review of algorithms and modeling techniques," Applied Energy, Elsevier, vol. 235(C), pages 1072-1089.
    4. Sun, Fangtian & Fu, Lin & Sun, Jian & Zhang, Shigang, 2014. "A new waste heat district heating system with combined heat and power (CHP) based on ejector heat exchangers and absorption heat pumps," Energy, Elsevier, vol. 69(C), pages 516-524.
    5. Fredrik Skaug Fadnes & Reyhaneh Banihabib & Mohsen Assadi, 2023. "Using Artificial Neural Networks to Gather Intelligence on a Fully Operational Heat Pump System in an Existing Building Cluster," Energies, MDPI, vol. 16(9), pages 1-33, May.
    6. Jie, Ji & Jingyong, Cai & Wenzhu, Huang & Yan, Feng, 2015. "Experimental study on the performance of solar-assisted multi-functional heat pump based on enthalpy difference lab with solar simulator," Renewable Energy, Elsevier, vol. 75(C), pages 381-388.
    7. Kezong Tang & Xiong-Fei Wei & Yuan-Hao Jiang & Zi-Wei Chen & Lihua Yang, 2023. "An Adaptive Ant Colony Optimization for Solving Large-Scale Traveling Salesman Problem," Mathematics, MDPI, vol. 11(21), pages 1-26, October.
    8. Sun, Fangtian & Fu, Lin & Sun, Jian & Zhang, Shigang, 2014. "A new ejector heat exchanger based on an ejector heat pump and a water-to-water heat exchanger," Applied Energy, Elsevier, vol. 121(C), pages 245-251.
    9. Nguyen, Hiep V. & Law, Ying Lam E. & Alavy, Masih & Walsh, Philip R. & Leong, Wey H. & Dworkin, Seth B., 2014. "An analysis of the factors affecting hybrid ground-source heat pump installation potential in North America," Applied Energy, Elsevier, vol. 125(C), pages 28-38.
    10. Weiqiang Pan & Zhilong Shan & Ting Chen & Fangjiong Chen & Jing Feng, 2016. "Optimal pilot design for OFDM systems with non-contiguous subcarriers based on semi-definite programming," Telecommunication Systems: Modelling, Analysis, Design and Management, Springer, vol. 63(2), pages 297-305, October.
    11. Kayaci, Nurullah, 2020. "Energy and exergy analysis and thermo-economic optimization of the ground source heat pump integrated with radiant wall panel and fan-coil unit with floor heating or radiator," Renewable Energy, Elsevier, vol. 160(C), pages 333-349.
    12. Giovanni Murano & Francesca Caffari & Nicolandrea Calabrese, 2024. "Energy Potential of Existing Reversible Air-to-Air Heat Pumps for Residential Heating," Sustainability, MDPI, vol. 16(14), pages 1-23, July.
    13. Fox, B. L. & Lenstra, J. K. & Rinnooy Kan, A. H. G. & Schrage, L. E., 1977. "Branching From The Largest Upper Bound: Folklore And Facts," Econometric Institute Archives 272158, Erasmus University Rotterdam.
    14. Aste, Niccolò & Adhikari, R.S. & Manfren, Massimiliano, 2013. "Cost optimal analysis of heat pump technology adoption in residential reference buildings," Renewable Energy, Elsevier, vol. 60(C), pages 615-624.
    15. Treichel, Calene & Cruickshank, Cynthia A., 2021. "Energy analysis of heat pump water heaters coupled with air-based solar thermal collectors in Canada and the United States," Energy, Elsevier, vol. 221(C).
    16. Balghouthi, M. & Chahbani, M.H. & Guizani, A., 2012. "Investigation of a solar cooling installation in Tunisia," Applied Energy, Elsevier, vol. 98(C), pages 138-148.
    17. Mohamed, Elamin & Riffat, Saffa & Omer, Siddig & Zeinelabdein, Rami, 2019. "A comprehensive investigation of using mutual air and water heating in multi-functional DX-SAMHP for moderate cold climate," Renewable Energy, Elsevier, vol. 130(C), pages 582-600.
    18. Li, Sihui & Gong, Guangcai & Peng, Jinqing, 2019. "Dynamic coupling method between air-source heat pumps and buildings in China’s hot-summer/cold-winter zone," Applied Energy, Elsevier, vol. 254(C).
    19. Antonio Paone & Jean-Philippe Bacher, 2018. "The Impact of Building Occupant Behavior on Energy Efficiency and Methods to Influence It: A Review of the State of the Art," Energies, MDPI, vol. 11(4), pages 1-19, April.
    20. Li, Shuangjun & Deng, Shuai & Zhao, Li & Zhao, Ruikai & Yuan, Xiangzhou, 2021. "Thermodynamic carbon pump 2.0: Elucidating energy efficiency through the thermodynamic cycle," Energy, Elsevier, vol. 215(PB).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:144:y:2018:i:c:p:159-168. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.