IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v300y2021ics0306261921007431.html
   My bibliography  Save this article

Towards healthy and cost-effective indoor environment management in smart homes: A deep reinforcement learning approach

Author

Listed:
  • Yang, Ting
  • Zhao, Liyuan
  • Li, Wei
  • Wu, Jianzhong
  • Zomaya, Albert Y.

Abstract

Indoor environmental quality is an important issue since people spend most of their time indoors. This paper aims to develop an autonomous indoor environment management approach to ensure a healthy indoor environment with minimized energy cost via the optimal control of ventilation system and heating/cooling system in smart homes. Due to the uncertainties of weather conditions, electricity price and home occupancy, as well as the complex interaction between indoor air quality and indoor thermal environment, it is challenging to develop an efficient control strategy. To address this challenge, the indoor environment management problem is formulated as a Markov decision process, and then a deep reinforcement learning control strategy, which combines double deep Q network with prioritized experience replay mechanism, is proposed to solve the Markov decision process. The proposed approach can make adaptive control decisions based on the current observations without requiring any forecast information of system uncertainties. Control performance under different scenarios show the proposed approach has good adaptability to the variation of weather conditions, electricity prices, home occupancy patterns and indoor temperature requirements. Moreover, the proposed approach is compared with a double deep Q network-based approach and a model predictive control-based approach. Comparison results show that the proposed approach reduces the average daily energy cost by 3.51% and 8.56% in winter scenarios and 4.05%, 7.88% in summer scenarios while achieving the smallest indoor air quality and temperature violations.

Suggested Citation

  • Yang, Ting & Zhao, Liyuan & Li, Wei & Wu, Jianzhong & Zomaya, Albert Y., 2021. "Towards healthy and cost-effective indoor environment management in smart homes: A deep reinforcement learning approach," Applied Energy, Elsevier, vol. 300(C).
    • Handle: RePEc:eee:appene:v:300:y:2021:i:c:s0306261921007431
    DOI: 10.1016/j.apenergy.2021.117335
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261921007431
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2021.117335?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Maharjan, Pukar & Salauddin, Md & Cho, Hyunok & Park, Jae Yeong, 2018. "An indoor power line based magnetic field energy harvester for self-powered wireless sensors in smart home applications," Applied Energy, Elsevier, vol. 232(C), pages 398-408.
    2. Hua, Haochen & Qin, Yuchao & Hao, Chuantong & Cao, Junwei, 2019. "Optimal energy management strategies for energy Internet via deep reinforcement learning approach," Applied Energy, Elsevier, vol. 239(C), pages 598-609.
    3. Jin, Xiaolong & Mu, Yunfei & Jia, Hongjie & Wu, Jianzhong & Jiang, Tao & Yu, Xiaodan, 2017. "Dynamic economic dispatch of a hybrid energy microgrid considering building based virtual energy storage system," Applied Energy, Elsevier, vol. 194(C), pages 386-398.
    4. Chaudhuri, Tanaya & Soh, Yeng Chai & Li, Hua & Xie, Lihua, 2019. "A feedforward neural network based indoor-climate control framework for thermal comfort and energy saving in buildings," Applied Energy, Elsevier, vol. 248(C), pages 44-53.
    5. Gianluca Serale & Massimo Fiorentini & Alfonso Capozzoli & Daniele Bernardini & Alberto Bemporad, 2018. "Model Predictive Control (MPC) for Enhancing Building and HVAC System Energy Efficiency: Problem Formulation, Applications and Opportunities," Energies, MDPI, vol. 11(3), pages 1-35, March.
    6. Hong, Taehoon & Kim, Jimin & Lee, Myeonghwi, 2018. "Integrated task performance score for the building occupants based on the CO2 concentration and indoor climate factors changes," Applied Energy, Elsevier, vol. 228(C), pages 1707-1713.
    7. Mei, Jun & Xia, Xiaohua & Song, Mengjie, 2018. "An autonomous hierarchical control for improving indoor comfort and energy efficiency of a direct expansion air conditioning system," Applied Energy, Elsevier, vol. 221(C), pages 450-463.
    8. Haji Hosseinloo, Ashkan & Ryzhov, Alexander & Bischi, Aldo & Ouerdane, Henni & Turitsyn, Konstantin & Dahleh, Munther A., 2020. "Data-driven control of micro-climate in buildings: An event-triggered reinforcement learning approach," Applied Energy, Elsevier, vol. 277(C).
    9. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    10. Afroz, Zakia & Urmee, Tania & Shafiullah, G.M. & Higgins, Gary, 2018. "Real-time prediction model for indoor temperature in a commercial building," Applied Energy, Elsevier, vol. 231(C), pages 29-53.
    11. Madurai Elavarasan, Rajvikram & Shafiullah, GM & Raju, Kannadasan & Mudgal, Vijay & Arif, M.T. & Jamal, Taskin & Subramanian, Senthilkumar & Sriraja Balaguru, V.S. & Reddy, K.S. & Subramaniam, Umashan, 2020. "COVID-19: Impact analysis and recommendations for power sector operation," Applied Energy, Elsevier, vol. 279(C).
    12. Li, Wenzhuo & Wang, Shengwei, 2020. "A multi-agent based distributed approach for optimal control of multi-zone ventilation systems considering indoor air quality and energy use," Applied Energy, Elsevier, vol. 275(C).
    13. Kong, Xiangyu & Kong, Deqian & Yao, Jingtao & Bai, Linquan & Xiao, Jie, 2020. "Online pricing of demand response based on long short-term memory and reinforcement learning," Applied Energy, Elsevier, vol. 271(C).
    14. Chatterjee, Arnab & Zhang, Lijun & Xia, Xiaohua, 2015. "Optimization of mine ventilation fan speeds according to ventilation on demand and time of use tariff," Applied Energy, Elsevier, vol. 146(C), pages 65-73.
    15. Dounis, A.I. & Caraiscos, C., 2009. "Advanced control systems engineering for energy and comfort management in a building environment--A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 13(6-7), pages 1246-1261, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Omar al-Ani & Sanjoy Das & Hongyu Wu, 2023. "Imitation Learning with Deep Attentive Tabular Neural Networks for Environmental Prediction and Control in Smart Home," Energies, MDPI, vol. 16(13), pages 1-19, June.
    2. Shen, Rendong & Zhong, Shengyuan & Wen, Xin & An, Qingsong & Zheng, Ruifan & Li, Yang & Zhao, Jun, 2022. "Multi-agent deep reinforcement learning optimization framework for building energy system with renewable energy," Applied Energy, Elsevier, vol. 312(C).
    3. Kirill Y. Kulakov & Svetlana S. Uvarova & Alexandr K. Orlov & Vadim S. Kankhva & Anna A. Sudakova, 2024. "Determining the Level of and Potential for the Development of Tourism Clusters, Taking into Account Infrastructure and Urban Planning Factors," Sustainability, MDPI, vol. 16(19), pages 1-20, October.
    4. Cui, Can & Xue, Jing, 2024. "Energy and comfort aware operation of multi-zone HVAC system through preference-inspired deep reinforcement learning," Energy, Elsevier, vol. 292(C).
    5. Di Natale, L. & Svetozarevic, B. & Heer, P. & Jones, C.N., 2023. "Towards scalable physically consistent neural networks: An application to data-driven multi-zone thermal building models," Applied Energy, Elsevier, vol. 340(C).
    6. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    7. Ayas Shaqour & Aya Hagishima, 2022. "Systematic Review on Deep Reinforcement Learning-Based Energy Management for Different Building Types," Energies, MDPI, vol. 15(22), pages 1-27, November.
    8. Liu, Yuntao & Cui, Can, 2024. "A bi-level real-time optimal control strategy for thermal coupled multi-zone dedicated outside air system-assisted HVAC systems," Energy, Elsevier, vol. 306(C).
    9. Víctor Sanz i López & Ramon Costa-Castelló & Carles Batlle, 2022. "Literature Review of Energy Management in Combined Heat and Power Systems Based on High-Temperature Proton Exchange Membrane Fuel Cells for Residential Comfort Applications," Energies, MDPI, vol. 15(17), pages 1-22, September.
    10. Fang, Xi & Gong, Guangcai & Li, Guannan & Chun, Liang & Peng, Pei & Li, Wenqiang & Shi, Xing, 2023. "Cross temporal-spatial transferability investigation of deep reinforcement learning control strategy in the building HVAC system level," Energy, Elsevier, vol. 263(PB).
    11. Dalia Mohammed Talat Ebrahim Ali & Violeta Motuzienė & Rasa Džiugaitė-Tumėnienė, 2024. "AI-Driven Innovations in Building Energy Management Systems: A Review of Potential Applications and Energy Savings," Energies, MDPI, vol. 17(17), pages 1-35, August.
    12. Seppo Sierla & Heikki Ihasalo & Valeriy Vyatkin, 2022. "A Review of Reinforcement Learning Applications to Control of Heating, Ventilation and Air Conditioning Systems," Energies, MDPI, vol. 15(10), pages 1-25, May.
    13. Pinthurat, Watcharakorn & Surinkaew, Tossaporn & Hredzak, Branislav, 2024. "An overview of reinforcement learning-based approaches for smart home energy management systems with energy storages," Renewable and Sustainable Energy Reviews, Elsevier, vol. 202(C).
    14. Zhou, Yanting & Ma, Zhongjing & Zhang, Jinhui & Zou, Suli, 2022. "Data-driven stochastic energy management of multi energy system using deep reinforcement learning," Energy, Elsevier, vol. 261(PA).
    15. Wang, Zixuan & Xiao, Fu & Ran, Yi & Li, Yanxue & Xu, Yang, 2024. "Scalable energy management approach of residential hybrid energy system using multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 367(C).
    16. Zhao, Liyuan & Yang, Ting & Li, Wei & Zomaya, Albert Y., 2022. "Deep reinforcement learning-based joint load scheduling for household multi-energy system," Applied Energy, Elsevier, vol. 324(C).
    17. Homod, Raad Z. & Togun, Hussein & Kadhim Hussein, Ahmed & Noraldeen Al-Mousawi, Fadhel & Yaseen, Zaher Mundher & Al-Kouz, Wael & Abd, Haider J. & Alawi, Omer A. & Goodarzi, Marjan & Hussein, Omar A., 2022. "Dynamics analysis of a novel hybrid deep clustering for unsupervised learning by reinforcement of multi-agent to energy saving in intelligent buildings," Applied Energy, Elsevier, vol. 313(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Cui, Can & Xue, Jing, 2024. "Energy and comfort aware operation of multi-zone HVAC system through preference-inspired deep reinforcement learning," Energy, Elsevier, vol. 292(C).
    2. Halhoul Merabet, Ghezlane & Essaaidi, Mohamed & Ben Haddou, Mohamed & Qolomany, Basheer & Qadir, Junaid & Anan, Muhammad & Al-Fuqaha, Ala & Abid, Mohamed Riduan & Benhaddou, Driss, 2021. "Intelligent building control systems for thermal comfort and energy-efficiency: A systematic review of artificial intelligence-assisted techniques," Renewable and Sustainable Energy Reviews, Elsevier, vol. 144(C).
    3. Liu, Yuntao & Cui, Can, 2024. "A bi-level real-time optimal control strategy for thermal coupled multi-zone dedicated outside air system-assisted HVAC systems," Energy, Elsevier, vol. 306(C).
    4. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    5. Panagiotis Michailidis & Iakovos Michailidis & Dimitrios Vamvakas & Elias Kosmatopoulos, 2023. "Model-Free HVAC Control in Buildings: A Review," Energies, MDPI, vol. 16(20), pages 1-45, October.
    6. Sun, Hongchang & Niu, Yanlei & Li, Chengdong & Zhou, Changgeng & Zhai, Wenwen & Chen, Zhe & Wu, Hao & Niu, Lanqiang, 2022. "Energy consumption optimization of building air conditioning system via combining the parallel temporal convolutional neural network and adaptive opposition-learning chimp algorithm," Energy, Elsevier, vol. 259(C).
    7. Wang, Ran & Lu, Shilei & Feng, Wei, 2020. "A novel improved model for building energy consumption prediction based on model integration," Applied Energy, Elsevier, vol. 262(C).
    8. Zhu, Ziqing & Hu, Ze & Chan, Ka Wing & Bu, Siqi & Zhou, Bin & Xia, Shiwei, 2023. "Reinforcement learning in deregulated energy market: A comprehensive review," Applied Energy, Elsevier, vol. 329(C).
    9. Pinto, Giuseppe & Deltetto, Davide & Capozzoli, Alfonso, 2021. "Data-driven district energy management with surrogate models and deep reinforcement learning," Applied Energy, Elsevier, vol. 304(C).
    10. Qi, Chunyang & Zhu, Yiwen & Song, Chuanxue & Yan, Guangfu & Xiao, Feng & Da wang, & Zhang, Xu & Cao, Jingwei & Song, Shixin, 2022. "Hierarchical reinforcement learning based energy management strategy for hybrid electric vehicle," Energy, Elsevier, vol. 238(PA).
    11. Zhao, Liyuan & Yang, Ting & Li, Wei & Zomaya, Albert Y., 2022. "Deep reinforcement learning-based joint load scheduling for household multi-energy system," Applied Energy, Elsevier, vol. 324(C).
    12. Ma, Siyu & Liu, Hui & Wang, Ni & Huang, Lidong & Goh, Hui Hwang, 2023. "Incentive-based demand response under incomplete information based on the deep deterministic policy gradient," Applied Energy, Elsevier, vol. 351(C).
    13. Clara Ceccolini & Roozbeh Sangi, 2022. "Benchmarking Approaches for Assessing the Performance of Building Control Strategies: A Review," Energies, MDPI, vol. 15(4), pages 1-30, February.
    14. Qiu, Dawei & Ye, Yujian & Papadaskalopoulos, Dimitrios & Strbac, Goran, 2021. "Scalable coordinated management of peer-to-peer energy trading: A multi-cluster deep reinforcement learning approach," Applied Energy, Elsevier, vol. 292(C).
    15. Raman, Naren Srivaths & Devaprasad, Karthikeya & Chen, Bo & Ingley, Herbert A. & Barooah, Prabir, 2020. "Model predictive control for energy-efficient HVAC operation with humidity and latent heat considerations," Applied Energy, Elsevier, vol. 279(C).
    16. Jiaxi Luo, 2022. "A Bibliometric Review on Artificial Intelligence for Smart Buildings," Sustainability, MDPI, vol. 14(16), pages 1-22, August.
    17. Xie, Jiahan & Ajagekar, Akshay & You, Fengqi, 2023. "Multi-Agent attention-based deep reinforcement learning for demand response in grid-responsive buildings," Applied Energy, Elsevier, vol. 342(C).
    18. Du, Yan & Zandi, Helia & Kotevska, Olivera & Kurte, Kuldeep & Munk, Jeffery & Amasyali, Kadir & Mckee, Evan & Li, Fangxing, 2021. "Intelligent multi-zone residential HVAC control strategy based on deep reinforcement learning," Applied Energy, Elsevier, vol. 281(C).
    19. Abhinandana Boodi & Karim Beddiar & Yassine Amirat & Mohamed Benbouzid, 2022. "Building Thermal-Network Models: A Comparative Analysis, Recommendations, and Perspectives," Energies, MDPI, vol. 15(4), pages 1-27, February.
    20. Perera, A.T.D. & Kamalaruban, Parameswaran, 2021. "Applications of reinforcement learning in energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 137(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:300:y:2021:i:c:s0306261921007431. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.