IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v259y2022ics0360544222017601.html
   My bibliography  Save this article

A multi-step predictive deep reinforcement learning algorithm for HVAC control systems in smart buildings

Author

Listed:
  • Liu, Xiangfei
  • Ren, Mifeng
  • Yang, Zhile
  • Yan, Gaowei
  • Guo, Yuanjun
  • Cheng, Lan
  • Wu, Chengke

Abstract

The development of the building energy management systems (BEMS) enable users to intelligently control Heating, Ventilation, Air-conditioning and Cooling (HVAC) systems based on digital information. In order to reduce the power consumption cost of the HVAC system while ensuring user satisfaction, a novel HVAC control system for building system based on a multi-step predictive deep reinforcement learning (MSP-DRL) algorithm is proposed in this paper. In the proposed method, the outdoor ambient temperature is predicted first by a featured deep learning method named GC-LSTM, where the Long Short-term Memory (LSTM) is enhanced by the generalized correntropy (GC) loss function to deal with the non-Gaussian characteristics of the collected outdoor temperature. In addition, the proposed temperature prediction model is combined with a reinforcement learning algorithm named Deep Deterministic Policy Gradient (DDPG) aiming to flexibly adjust the output power of the HVAC system under the dynamic changing of electricity prices. Finally, comprehensive simulation based on real world data is delivered. Numerical results show that the GC-LSTM algorithm is more accurate than other counterparts prediction algorithms, and the proposed HVAC control system based on the multi-step prediction deep reinforcement learning algorithm is effective and could save over 12% cost compared to other approaches, where the user comfort is maintained simultaneously.

Suggested Citation

  • Liu, Xiangfei & Ren, Mifeng & Yang, Zhile & Yan, Gaowei & Guo, Yuanjun & Cheng, Lan & Wu, Chengke, 2022. "A multi-step predictive deep reinforcement learning algorithm for HVAC control systems in smart buildings," Energy, Elsevier, vol. 259(C).
    • Handle: RePEc:eee:energy:v:259:y:2022:i:c:s0360544222017601
    DOI: 10.1016/j.energy.2022.124857
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0360544222017601
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.energy.2022.124857?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Turhan, Cihan & Simani, Silvio & Gokcen Akkurt, Gulden, 2021. "Development of a personalized thermal comfort driven controller for HVAC systems," Energy, Elsevier, vol. 237(C).
    2. Lu, Qing & Lü, Shuaikang & Leng, Yajun & Zhang, Zhixin, 2020. "Optimal household energy management based on smart residential energy hub considering uncertain behaviors," Energy, Elsevier, vol. 195(C).
    3. Wang, Huilong & Wang, Shengwei, 2021. "A hierarchical optimal control strategy for continuous demand response of building HVAC systems to provide frequency regulation service to smart power grids," Energy, Elsevier, vol. 230(C).
    4. Zhou, Guangyou & Zhu, Zhiwei & Luo, Sumei, 2022. "Location optimization of electric vehicle charging stations: Based on cost model and genetic algorithm," Energy, Elsevier, vol. 247(C).
    5. Yuan, Zhi & Wang, Weiqing & Wang, Haiyun & Mizzi, Scott, 2020. "Combination of cuckoo search and wavelet neural network for midterm building energy forecast," Energy, Elsevier, vol. 202(C).
    6. Yin, Linfei & Xie, Jiaxing, 2022. "Multi-feature-scale fusion temporal convolution networks for metal temperature forecasting of ultra-supercritical coal-fired power plant reheater tubes," Energy, Elsevier, vol. 238(PA).
    7. Chen, Hao & Wang, Yu & Zuo, Mingsheng & Zhang, Chao & Jia, Ninghong & Liu, Xiliang & Yang, Shenglai, 2022. "A new prediction model of CO2 diffusion coefficient in crude oil under reservoir conditions based on BP neural network," Energy, Elsevier, vol. 239(PC).
    8. Mohammed M. Olama & Teja Kuruganti & James Nutaro & Jin Dong, 2018. "Coordination and Control of Building HVAC Systems to Provide Frequency Regulation to the Electric Grid," Energies, MDPI, vol. 11(7), pages 1-15, July.
    9. Jing Zhao & Yaoqi Duan & Xiaojuan Liu, 2018. "Uncertainty Analysis of Weather Forecast Data for Cooling Load Forecasting Based on the Monte Carlo Method," Energies, MDPI, vol. 11(7), pages 1-18, July.
    10. Yuan, Chaoqing & Liu, Sifeng & Fang, Zhigeng, 2016. "Comparison of China's primary energy consumption forecasting by using ARIMA (the autoregressive integrated moving average) model and GM(1,1) model," Energy, Elsevier, vol. 100(C), pages 384-390.
    11. Baniassadi, Amir & Heusinger, Jannik & Gonzalez, Pablo Izaga & Weber, Stephan & Samuelson, Holly W., 2022. "Co-benefits of energy efficiency in residential buildings," Energy, Elsevier, vol. 238(PB).
    12. Totaro, Simone & Boukas, Ioannis & Jonsson, Anders & Cornélusse, Bertrand, 2021. "Lifelong control of off-grid microgrid with model-based reinforcement learning," Energy, Elsevier, vol. 232(C).
    13. Zhang, Meng & Guo, Huan & Sun, Ming & Liu, Sifeng & Forrest, Jeffrey, 2022. "A novel flexible grey multivariable model and its application in forecasting energy consumption in China," Energy, Elsevier, vol. 239(PE).
    14. Xiong, Guojiang & Shuai, Maohang & Hu, Xiao, 2022. "Combined heat and power economic emission dispatch using improved bare-bone multi-objective particle swarm optimization," Energy, Elsevier, vol. 244(PB).
    15. Saha, Moumita & Santara, Anirban & Mitra, Pabitra & Chakraborty, Arun & Nanjundiah, Ravi S., 2021. "Prediction of the Indian summer monsoon using a stacked autoencoder and ensemble regression model," International Journal of Forecasting, Elsevier, vol. 37(1), pages 58-71.
    16. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    17. Tashtoush, Bourhan & Molhim, M. & Al-Rousan, M., 2005. "Dynamic model of an HVAC system for control analysis," Energy, Elsevier, vol. 30(10), pages 1729-1745.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Zhuang, Dian & Gan, Vincent J.L. & Duygu Tekler, Zeynep & Chong, Adrian & Tian, Shuai & Shi, Xing, 2023. "Data-driven predictive control for smart HVAC system in IoT-integrated buildings with time-series forecasting and reinforcement learning," Applied Energy, Elsevier, vol. 338(C).
    2. Zhang, Lidong & Li, Jiao & Xu, Xiandong & Liu, Fengrui & Guo, Yuanjun & Yang, Zhile & Hu, Tianyu, 2023. "High spatial granularity residential heating load forecast based on Dendrite net model," Energy, Elsevier, vol. 269(C).
    3. Cui, Can & Xue, Jing, 2024. "Energy and comfort aware operation of multi-zone HVAC system through preference-inspired deep reinforcement learning," Energy, Elsevier, vol. 292(C).
    4. Park, Jong-Whi & Ju, Young-Min & Kim, You-Gwon & Kim, Hak-Sung, 2023. "50% reduction in energy consumption in an actual cold storage facility using a deep reinforcement learning-based control algorithm," Applied Energy, Elsevier, vol. 352(C).
    5. Chen, Xiaodong & Ge, Xinxin & Sun, Rongfu & Wang, Fei & Mi, Zengqiang, 2024. "A SVM based demand response capacity prediction model considering internal factors under composite program," Energy, Elsevier, vol. 300(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sun, Hongchang & Niu, Yanlei & Li, Chengdong & Zhou, Changgeng & Zhai, Wenwen & Chen, Zhe & Wu, Hao & Niu, Lanqiang, 2022. "Energy consumption optimization of building air conditioning system via combining the parallel temporal convolutional neural network and adaptive opposition-learning chimp algorithm," Energy, Elsevier, vol. 259(C).
    2. Zhang, Tianhao & Dong, Zhe & Huang, Xiaojin, 2024. "Multi-objective optimization of thermal power and outlet steam temperature for a nuclear steam supply system with deep reinforcement learning," Energy, Elsevier, vol. 286(C).
    3. Li, Yanxue & Wang, Zixuan & Xu, Wenya & Gao, Weijun & Xu, Yang & Xiao, Fu, 2023. "Modeling and energy dynamic control for a ZEH via hybrid model-based deep reinforcement learning," Energy, Elsevier, vol. 277(C).
    4. Harrold, Daniel J.B. & Cao, Jun & Fan, Zhong, 2022. "Data-driven battery operation for energy arbitrage using rainbow deep reinforcement learning," Energy, Elsevier, vol. 238(PC).
    5. Geng Wu & Yi-Chung Hu & Yu-Jing Chiu & Shu-Ju Tsao, 2023. "A new multivariate grey prediction model for forecasting China’s regional energy consumption," Environment, Development and Sustainability: A Multidisciplinary Approach to the Theory and Practice of Sustainable Development, Springer, vol. 25(5), pages 4173-4193, May.
    6. Zhang, Yunxin & Guo, Huan & Sun, Ming & Liu, Sifeng & Forrest, Jeffrey, 2023. "A novel grey Lotka–Volterra model driven by the mechanism of competition and cooperation for energy consumption forecasting," Energy, Elsevier, vol. 264(C).
    7. Ding, Yuanping & Dang, Yaoguo, 2023. "Forecasting renewable energy generation with a novel flexible nonlinear multivariable discrete grey prediction model," Energy, Elsevier, vol. 277(C).
    8. Haisheng Hu & Wanhao Dong, 2022. "The Goal of Carbon Peaking, Carbon Emissions, and the Economic Effects of China’s Energy Planning Policy: Analysis Using a CGE Model," IJERPH, MDPI, vol. 20(1), pages 1-20, December.
    9. Jing, Gang & Cai, Wenjian & Zhang, Xin & Cui, Can & Yin, Xiaohong & Xian, Huacai, 2019. "An energy-saving oriented air balancing strategy for multi-zone demand-controlled ventilation system," Energy, Elsevier, vol. 172(C), pages 1053-1065.
    10. Tulika Saha & Sriparna Saha & Pushpak Bhattacharyya, 2020. "Towards sentiment aided dialogue policy learning for multi-intent conversations using hierarchical reinforcement learning," PLOS ONE, Public Library of Science, vol. 15(7), pages 1-28, July.
    11. Emrani-Rahaghi, Pouria & Hashemi-Dezaki, Hamed & Ketabi, Abbas, 2023. "Efficient voltage control of low voltage distribution networks using integrated optimized energy management of networked residential multi-energy microgrids," Applied Energy, Elsevier, vol. 349(C).
    12. Mahmoud Mahfouz & Angelos Filos & Cyrine Chtourou & Joshua Lockhart & Samuel Assefa & Manuela Veloso & Danilo Mandic & Tucker Balch, 2019. "On the Importance of Opponent Modeling in Auction Markets," Papers 1911.12816, arXiv.org.
    13. Pin Li & Jinsuo Zhang, 2019. "Is China’s Energy Supply Sustainable? New Research Model Based on the Exponential Smoothing and GM(1,1) Methods," Energies, MDPI, vol. 12(2), pages 1-30, January.
    14. Imen Azzouz & Wiem Fekih Hassen, 2023. "Optimization of Electric Vehicles Charging Scheduling Based on Deep Reinforcement Learning: A Decentralized Approach," Energies, MDPI, vol. 16(24), pages 1-18, December.
    15. Zhong, Fangliang & Calautit, John Kaiser & Wu, Yupeng, 2022. "Assessment of HVAC system operational fault impacts and multiple faults interactions under climate change," Energy, Elsevier, vol. 258(C).
    16. Jacob W. Crandall & Mayada Oudah & Tennom & Fatimah Ishowo-Oloko & Sherief Abdallah & Jean-François Bonnefon & Manuel Cebrian & Azim Shariff & Michael A. Goodrich & Iyad Rahwan, 2018. "Cooperating with machines," Nature Communications, Nature, vol. 9(1), pages 1-12, December.
      • Abdallah, Sherief & Bonnefon, Jean-François & Cebrian, Manuel & Crandall, Jacob W. & Ishowo-Oloko, Fatimah & Oudah, Mayada & Rahwan, Iyad & Shariff, Azim & Tennom,, 2017. "Cooperating with Machines," TSE Working Papers 17-806, Toulouse School of Economics (TSE).
      • Abdallah, Sherief & Bonnefon, Jean-François & Cebrian, Manuel & Crandall, Jacob W. & Ishowo-Oloko, Fatimah & Oudah, Mayada & Rahwan, Iyad & Shariff, Azim & Tennom,, 2017. "Cooperating with Machines," IAST Working Papers 17-68, Institute for Advanced Study in Toulouse (IAST).
      • Jacob Crandall & Mayada Oudah & Fatimah Ishowo-Oloko Tennom & Fatimah Ishowo-Oloko & Sherief Abdallah & Jean-François Bonnefon & Manuel Cebrian & Azim Shariff & Michael Goodrich & Iyad Rahwan, 2018. "Cooperating with machines," Post-Print hal-01897802, HAL.
    17. Sun, Alexander Y., 2020. "Optimal carbon storage reservoir management through deep reinforcement learning," Applied Energy, Elsevier, vol. 278(C).
    18. Yassine Chemingui & Adel Gastli & Omar Ellabban, 2020. "Reinforcement Learning-Based School Energy Management System," Energies, MDPI, vol. 13(23), pages 1-21, December.
    19. Woo Jae Byun & Bumkyu Choi & Seongmin Kim & Joohyun Jo, 2023. "Practical Application of Deep Reinforcement Learning to Optimal Trade Execution," FinTech, MDPI, vol. 2(3), pages 1-16, June.
    20. Lu, Yu & Xiang, Yue & Huang, Yuan & Yu, Bin & Weng, Liguo & Liu, Junyong, 2023. "Deep reinforcement learning based optimal scheduling of active distribution system considering distributed generation, energy storage and flexible load," Energy, Elsevier, vol. 271(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:259:y:2022:i:c:s0360544222017601. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.