IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v318y2022ics0306261922005712.html
   My bibliography  Save this article

Reinforcement Learning for proactive operation of residential energy systems by learning stochastic occupant behavior and fluctuating solar energy: Balancing comfort, hygiene and energy use

Author

Listed:
  • Heidari, Amirreza
  • Maréchal, François
  • Khovalyg, Dolaana

Abstract

When it comes to residential buildings, there are several stochastic parameters, such as renewable energy production, outdoor air conditions, and occupants’ behavior, that are hard to model and predict accurately, with some being unique in each specific building. This increases the complexity of developing a generalizable optimal control method that can be transferred to different buildings. Rather than hard-programming human knowledge into the controller (in terms of rules or models), a learning ability can be provided to the controller such that over the time it can learn by itself how to maintain an optimal operation in each specific building. This research proposes a model-free control framework based on Reinforcement Learning that takes into account the stochastic hot water use behavior of occupants, solar power generation, and weather conditions, and learns how to make a balance between the energy use, occupant comfort and water hygiene in a solar-assisted space heating and hot water production system. A stochastic-based offline training procedure is proposed to give a prior experience to the agent in a safe simulation environment, and further ensure occupants comfort and health when the algorithm starts online learning on the real house. To make a realistic assessment without interrupting the occupants, weather conditions and hot water use behavior are experimentally monitored in three case studies in different regions of Switzerland, and the collected data are used in simulations to evaluate the proposed control framework against two rule-based methods. Results indicate that the proposed framework could achieve an energy saving from 7% to 60%, mainly by adapting to solar power generation, without violating comfort or compromising the health of occupants.

Suggested Citation

  • Heidari, Amirreza & Maréchal, François & Khovalyg, Dolaana, 2022. "Reinforcement Learning for proactive operation of residential energy systems by learning stochastic occupant behavior and fluctuating solar energy: Balancing comfort, hygiene and energy use," Applied Energy, Elsevier, vol. 318(C).
  • Handle: RePEc:eee:appene:v:318:y:2022:i:c:s0306261922005712
    DOI: 10.1016/j.apenergy.2022.119206
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261922005712
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.apenergy.2022.119206?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Kondziella, Hendrik & Bruckner, Thomas, 2016. "Flexibility requirements of renewable energy based electricity systems – a review of research results and methodologies," Renewable and Sustainable Energy Reviews, Elsevier, vol. 53(C), pages 10-22.
    2. Smarra, Francesco & Jain, Achin & de Rubeis, Tullio & Ambrosini, Dario & D’Innocenzo, Alessandro & Mangharam, Rahul, 2018. "Data-driven model predictive control using random forests for building energy optimization and climate control," Applied Energy, Elsevier, vol. 226(C), pages 1252-1272.
    3. Ormandy, David & Ezratty, Véronique, 2012. "Health and thermal comfort: From WHO guidance to housing strategies," Energy Policy, Elsevier, vol. 49(C), pages 116-121.
    4. Haji Hosseinloo, Ashkan & Ryzhov, Alexander & Bischi, Aldo & Ouerdane, Henni & Turitsyn, Konstantin & Dahleh, Munther A., 2020. "Data-driven control of micro-climate in buildings: An event-triggered reinforcement learning approach," Applied Energy, Elsevier, vol. 277(C).
    5. Kazmi, Hussain & Mehmood, Fahad & Lodeweyckx, Stefan & Driesen, Johan, 2018. "Gigawatt-hour scale savings on a budget of zero: Deep reinforcement learning based optimal control of hot water systems," Energy, Elsevier, vol. 144(C), pages 159-168.
    6. Fiorentini, Massimo & Wall, Josh & Ma, Zhenjun & Braslavsky, Julio H. & Cooper, Paul, 2017. "Hybrid model predictive control of a residential HVAC system with on-site thermal energy generation and storage," Applied Energy, Elsevier, vol. 187(C), pages 465-479.
    7. Yue, Ting & Long, Ruyin & Chen, Hong, 2013. "Factors influencing energy-saving behavior of urban households in Jiangsu Province," Energy Policy, Elsevier, vol. 62(C), pages 665-675.
    8. Heidari, Amirreza & Maréchal, François & Khovalyg, Dolaana, 2022. "An occupant-centric control framework for balancing comfort, energy use and hygiene in hot water systems: A model-free reinforcement learning approach," Applied Energy, Elsevier, vol. 312(C).
    9. Schreiber, Thomas & Netsch, Christoph & Eschweiler, Sören & Wang, Tianyuan & Storek, Thomas & Baranski, Marc & Müller, Dirk, 2021. "Application of data-driven methods for energy system modelling demonstrated on an adaptive cooling supply system," Energy, Elsevier, vol. 230(C).
    10. Vanhoudt, D. & Geysen, D. & Claessens, B. & Leemans, F. & Jespers, L. & Van Bael, J., 2014. "An actively controlled residential heat pump: Potential on peak shaving and maximization of self-consumption of renewable energy," Renewable Energy, Elsevier, vol. 63(C), pages 531-543.
    11. Correa-Jullian, Camila & López Droguett, Enrique & Cardemil, José Miguel, 2020. "Operation scheduling in a solar thermal system: A reinforcement learning-based framework," Applied Energy, Elsevier, vol. 268(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yan, Biao & Yang, Wansheng & He, Fuquan & Zeng, Wenhao, 2023. "Occupant behavior impact in buildings and the artificial intelligence-based techniques and data-driven approach solutions," Renewable and Sustainable Energy Reviews, Elsevier, vol. 184(C).
    2. Sabarathinam Srinivasan & Suresh Kumarasamy & Zacharias E. Andreadakis & Pedro G. Lind, 2023. "Artificial Intelligence and Mathematical Models of Power Grids Driven by Renewable Energy Sources: A Survey," Energies, MDPI, vol. 16(14), pages 1-56, July.
    3. Hidayatus Sibyan & Jozef Svajlenka & Hermawan Hermawan & Nasyiin Faqih & Annisa Nabila Arrizqi, 2022. "Thermal Comfort Prediction Accuracy with Machine Learning between Regression Analysis and Naïve Bayes Classifier," Sustainability, MDPI, vol. 14(23), pages 1-18, November.
    4. Chen, Minghao & Xie, Zhiyuan & Sun, Yi & Zheng, Shunlin, 2023. "The predictive management in campus heating system based on deep reinforcement learning and probabilistic heat demands forecasting," Applied Energy, Elsevier, vol. 350(C).
    5. Chen, Minghao & Sun, Yi & Xie, Zhiyuan & Lin, Nvgui & Wu, Peng, 2023. "An efficient and privacy-preserving algorithm for multiple energy hubs scheduling with federated and matching deep reinforcement learning," Energy, Elsevier, vol. 284(C).
    6. Ayas Shaqour & Aya Hagishima, 2022. "Systematic Review on Deep Reinforcement Learning-Based Energy Management for Different Building Types," Energies, MDPI, vol. 15(22), pages 1-27, November.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    2. Heidari, Amirreza & Maréchal, François & Khovalyg, Dolaana, 2022. "An occupant-centric control framework for balancing comfort, energy use and hygiene in hot water systems: A model-free reinforcement learning approach," Applied Energy, Elsevier, vol. 312(C).
    3. Finck, Christian & Li, Rongling & Kramer, Rick & Zeiler, Wim, 2018. "Quantifying demand flexibility of power-to-heat and thermal energy storage in the control of building heating systems," Applied Energy, Elsevier, vol. 209(C), pages 409-425.
    4. Samy Faddel & Guanyu Tian & Qun Zhou, 2021. "Decentralized Management of Commercial HVAC Systems," Energies, MDPI, vol. 14(11), pages 1-18, May.
    5. Frank Florez & Pedro Fernández de Córdoba & José Luis Higón & Gerard Olivar & John Taborda, 2019. "Modeling, Simulation, and Temperature Control of a Thermal Zone with Sliding Modes Strategy," Mathematics, MDPI, vol. 7(6), pages 1-13, June.
    6. Finck, Christian & Li, Rongling & Zeiler, Wim, 2019. "Economic model predictive control for demand flexibility of a residential building," Energy, Elsevier, vol. 176(C), pages 365-379.
    7. Yin, Linfei & Xiong, Yi, 2024. "Fast-apply deep autoregressive recurrent proximal policy optimization for controlling hot water systems," Applied Energy, Elsevier, vol. 367(C).
    8. de Gracia, Alvaro & Tarragona, Joan & Crespo, Alicia & Fernández, Cèsar, 2020. "Smart control of dynamic phase change material wall system," Applied Energy, Elsevier, vol. 279(C).
    9. Haji Hosseinloo, Ashkan & Ryzhov, Alexander & Bischi, Aldo & Ouerdane, Henni & Turitsyn, Konstantin & Dahleh, Munther A., 2020. "Data-driven control of micro-climate in buildings: An event-triggered reinforcement learning approach," Applied Energy, Elsevier, vol. 277(C).
    10. Seppo Sierla & Heikki Ihasalo & Valeriy Vyatkin, 2022. "A Review of Reinforcement Learning Applications to Control of Heating, Ventilation and Air Conditioning Systems," Energies, MDPI, vol. 15(10), pages 1-25, May.
    11. Guo, Yurun & Wang, Shugang & Wang, Jihong & Zhang, Tengfei & Ma, Zhenjun & Jiang, Shuang, 2024. "Key district heating technologies for building energy flexibility: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 189(PB).
    12. Daniela D’Alessandro & Andrea Rebecchi & Letizia Appolloni & Andrea Brambilla & Silvio Brusaferro & Maddalena Buffoli & Maurizio Carta & Alessandra Casuccio & Liliana Coppola & Maria Vittoria Corazza , 2023. "Re-Thinking the Environment, Cities, and Living Spaces for Public Health Purposes, According with the COVID-19 Lesson: The LVII Erice Charter," Land, MDPI, vol. 12(10), pages 1-17, September.
    13. Le Thi Dieu Hien & Khuu Ngoc Huyen & Thi Hong Loc Hoang, 2023. "Factors Affecting Energy-Saving Intentions among Youth in Vietnam," International Journal of Energy Economics and Policy, Econjournals, vol. 13(6), pages 603-609, November.
    14. Jenkins, J.D. & Zhou, Z. & Ponciroli, R. & Vilim, R.B. & Ganda, F. & de Sisternes, F. & Botterud, A., 2018. "The benefits of nuclear flexibility in power system operations with renewable energy," Applied Energy, Elsevier, vol. 222(C), pages 872-884.
    15. Yang, Shiyu & Wan, Man Pun & Ng, Bing Feng & Dubey, Swapnil & Henze, Gregor P. & Chen, Wanyu & Baskaran, Krishnamoorthy, 2020. "Experimental study of model predictive control for an air-conditioning system with dedicated outdoor air system," Applied Energy, Elsevier, vol. 257(C).
    16. Reza Fachrizal & Joakim Munkhammar, 2020. "Improved Photovoltaic Self-Consumption in Residential Buildings with Distributed and Centralized Smart Charging of Electric Vehicles," Energies, MDPI, vol. 13(5), pages 1-19, March.
    17. Burlinson, Andrew & Giulietti, Monica & Law, Cherry & Liu, Hui-Hsuan, 2021. "Fuel poverty and financial distress," Energy Economics, Elsevier, vol. 102(C).
    18. Gao, Datong & Zhao, Bin & Kwan, Trevor Hocksun & Hao, Yong & Pei, Gang, 2022. "The spatial and temporal mismatch phenomenon in solar space heating applications: status and solutions," Applied Energy, Elsevier, vol. 321(C).
    19. Lork, Clement & Li, Wen-Tai & Qin, Yan & Zhou, Yuren & Yuen, Chau & Tushar, Wayes & Saha, Tapan K., 2020. "An uncertainty-aware deep reinforcement learning framework for residential air conditioning energy management," Applied Energy, Elsevier, vol. 276(C).
    20. Zhao, Jing & Yang, Zilan & Shi, Linyu & Liu, Dehan & Li, Haonan & Mi, Yumiao & Wang, Hongbin & Feng, Meili & Hutagaol, Timothy Joseph, 2024. "Photovoltaic capacity dynamic tracking model predictive control strategy of air-conditioning systems with consideration of flexible loads," Applied Energy, Elsevier, vol. 356(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:318:y:2022:i:c:s0306261922005712. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.