IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v14y2021i4p997-d499227.html
   My bibliography  Save this article

Online Implementation of a Soft Actor-Critic Agent to Enhance Indoor Temperature Control and Energy Efficiency in Buildings

Author

Listed:
  • Davide Coraci

    (TEBE Research Group, BAEDA Lab, Department of Energy “Galileo Ferraris”, Politecnico di Torino, Corso Duca degli Abruzzi 24, 10129 Turin, Italy)

  • Silvio Brandi

    (TEBE Research Group, BAEDA Lab, Department of Energy “Galileo Ferraris”, Politecnico di Torino, Corso Duca degli Abruzzi 24, 10129 Turin, Italy)

  • Marco Savino Piscitelli

    (TEBE Research Group, BAEDA Lab, Department of Energy “Galileo Ferraris”, Politecnico di Torino, Corso Duca degli Abruzzi 24, 10129 Turin, Italy)

  • Alfonso Capozzoli

    (TEBE Research Group, BAEDA Lab, Department of Energy “Galileo Ferraris”, Politecnico di Torino, Corso Duca degli Abruzzi 24, 10129 Turin, Italy)

Abstract

Recently, a growing interest has been observed in HVAC control systems based on Artificial Intelligence, to improve comfort conditions while avoiding unnecessary energy consumption. In this work, a model-free algorithm belonging to the Deep Reinforcement Learning (DRL) class, Soft Actor-Critic, was implemented to control the supply water temperature to radiant terminal units of a heating system serving an office building. The controller was trained online, and a preliminary sensitivity analysis on hyperparameters was performed to assess their influence on the agent performance. The DRL agent with the best performance was compared to a rule-based controller assumed as a baseline during a three-month heating season. The DRL controller outperformed the baseline after two weeks of deployment, with an overall performance improvement related to control of indoor temperature conditions. Moreover, the adaptability of the DRL agent was tested for various control scenarios, simulating changes of external weather conditions, indoor temperature setpoint, building envelope features and occupancy patterns. The agent dynamically deployed, despite a slight increase in energy consumption, led to an improvement of indoor temperature control, reducing the cumulative sum of temperature violations on average for all scenarios by 75% and 48% compared to the baseline and statically deployed agent respectively.

Suggested Citation

  • Davide Coraci & Silvio Brandi & Marco Savino Piscitelli & Alfonso Capozzoli, 2021. "Online Implementation of a Soft Actor-Critic Agent to Enhance Indoor Temperature Control and Energy Efficiency in Buildings," Energies, MDPI, vol. 14(4), pages 1-26, February.
    • Handle: RePEc:gam:jeners:v:14:y:2021:i:4:p:997-:d:499227
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/14/4/997/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1996-1073/14/4/997/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Yang, Lei & Nagy, Zoltan & Goffin, Philippe & Schlueter, Arno, 2015. "Reinforcement learning for optimal control of low exergy buildings," Applied Energy, Elsevier, vol. 156(C), pages 577-586.
    2. Anand, Prashant & Cheong, David & Sekhar, Chandra & Santamouris, Mattheos & Kondepudi, Sekhar, 2019. "Energy saving estimation for plug and lighting load using occupancy analysis," Renewable Energy, Elsevier, vol. 143(C), pages 1143-1161.
    3. Wang, Zhe & Hong, Tianzhen, 2020. "Reinforcement learning for building controls: The opportunities and challenges," Applied Energy, Elsevier, vol. 269(C).
    4. Domenico Palladino & Iole Nardi & Cinzia Buratti, 2020. "Artificial Neural Network for the Thermal Comfort Index Prediction: Development of a New Simplified Algorithm," Energies, MDPI, vol. 13(17), pages 1-27, September.
    5. Martinopoulos, Georgios & Papakostas, Konstantinos T. & Papadopoulos, Agis M., 2018. "A comparative review of heating systems in EU countries, based on efficiency and fuel cost," Renewable and Sustainable Energy Reviews, Elsevier, vol. 90(C), pages 687-699.
    6. Mechri, Houcem Eddine & Capozzoli, Alfonso & Corrado, Vincenzo, 2010. "USE of the ANOVA approach for sensitive building energy design," Applied Energy, Elsevier, vol. 87(10), pages 3073-3083, October.
    7. Georgios D. Kontes & Georgios I. Giannakis & Víctor Sánchez & Pablo De Agustin-Camacho & Ander Romero-Amorrortu & Natalia Panagiotidou & Dimitrios V. Rovas & Simone Steiger & Christopher Mutschler & G, 2018. "Simulation-Based Evaluation and Optimization of Control Strategies in Buildings," Energies, MDPI, vol. 11(12), pages 1-23, December.
    8. Antonopoulos, Ioannis & Robu, Valentin & Couraud, Benoit & Kirli, Desen & Norbu, Sonam & Kiprakis, Aristides & Flynn, David & Elizondo-Gonzalez, Sergio & Wattam, Steve, 2020. "Artificial intelligence and machine learning approaches to energy demand-side response: A systematic review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 130(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Praveen Cheekatamarla & Kashif Nawaz, 2022. "Global Building Decarbonization Trends and Strategies," Energies, MDPI, vol. 15(22), pages 1-3, November.
    2. Pinto, Giuseppe & Kathirgamanathan, Anjukan & Mangina, Eleni & Finn, Donal P. & Capozzoli, Alfonso, 2022. "Enhancing energy management in grid-interactive buildings: A comparison among cooperative and coordinated architectures," Applied Energy, Elsevier, vol. 310(C).
    3. Ayas Shaqour & Aya Hagishima, 2022. "Systematic Review on Deep Reinforcement Learning-Based Energy Management for Different Building Types," Energies, MDPI, vol. 15(22), pages 1-27, November.
    4. Ebrahim Morady & Madjid Soltani & Farshad Moradi Kashkooli & Masoud Ziabasharhagh & Armughan Al-Haq & Jatin Nathwani, 2022. "Improving Energy Efficiency by Utilizing Wetted Cellulose Pads in Passive Cooling Systems," Energies, MDPI, vol. 15(1), pages 1-17, January.
    5. Davide Deltetto & Davide Coraci & Giuseppe Pinto & Marco Savino Piscitelli & Alfonso Capozzoli, 2021. "Exploring the Potentialities of Deep Reinforcement Learning for Incentive-Based Demand Response in a Cluster of Small Commercial Buildings," Energies, MDPI, vol. 14(10), pages 1-25, May.
    6. Seppo Sierla & Heikki Ihasalo & Valeriy Vyatkin, 2022. "A Review of Reinforcement Learning Applications to Control of Heating, Ventilation and Air Conditioning Systems," Energies, MDPI, vol. 15(10), pages 1-25, May.
    7. Pinto, Giuseppe & Deltetto, Davide & Capozzoli, Alfonso, 2021. "Data-driven district energy management with surrogate models and deep reinforcement learning," Applied Energy, Elsevier, vol. 304(C).
    8. Fernando Cassola & Leonel Morgado & António Coelho & Hugo Paredes & António Barbosa & Helga Tavares & Filipe Soares, 2022. "Using Virtual Choreographies to Identify Office Users’ Behaviors to Target Behavior Change Based on Their Potential to Impact Energy Consumption," Energies, MDPI, vol. 15(12), pages 1-21, June.
    9. Rima Aridi & Jalal Faraj & Samer Ali & Mostafa Gad El-Rab & Thierry Lemenand & Mahmoud Khaled, 2021. "Energy Recovery in Air Conditioning Systems: Comprehensive Review, Classifications, Critical Analysis, and Potential Recommendations," Energies, MDPI, vol. 14(18), pages 1-31, September.
    10. Coraci, Davide & Brandi, Silvio & Hong, Tianzhen & Capozzoli, Alfonso, 2023. "Online transfer learning strategy for enhancing the scalability and deployment of deep reinforcement learning control in smart buildings," Applied Energy, Elsevier, vol. 333(C).
    11. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    12. Ali Hamza & Muhammad Uneeb & Iftikhar Ahmad & Komal Saleem & Zunaib Ali, 2023. "Variable Structure-Based Control for Dynamic Temperature Setpoint Regulation in Hospital Extreme Healthcare Zones," Energies, MDPI, vol. 16(10), pages 1-27, May.
    13. Silvestri, Alberto & Coraci, Davide & Brandi, Silvio & Capozzoli, Alfonso & Borkowski, Esther & Köhler, Johannes & Wu, Duan & Zeilinger, Melanie N. & Schlueter, Arno, 2024. "Real building implementation of a deep reinforcement learning controller to enhance energy efficiency and indoor temperature control," Applied Energy, Elsevier, vol. 368(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Coraci, Davide & Brandi, Silvio & Hong, Tianzhen & Capozzoli, Alfonso, 2023. "Online transfer learning strategy for enhancing the scalability and deployment of deep reinforcement learning control in smart buildings," Applied Energy, Elsevier, vol. 333(C).
    2. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    3. Silvestri, Alberto & Coraci, Davide & Brandi, Silvio & Capozzoli, Alfonso & Borkowski, Esther & Köhler, Johannes & Wu, Duan & Zeilinger, Melanie N. & Schlueter, Arno, 2024. "Real building implementation of a deep reinforcement learning controller to enhance energy efficiency and indoor temperature control," Applied Energy, Elsevier, vol. 368(C).
    4. Yassine Chemingui & Adel Gastli & Omar Ellabban, 2020. "Reinforcement Learning-Based School Energy Management System," Energies, MDPI, vol. 13(23), pages 1-21, December.
    5. Gokhale, Gargya & Claessens, Bert & Develder, Chris, 2022. "Physics informed neural networks for control oriented thermal modeling of buildings," Applied Energy, Elsevier, vol. 314(C).
    6. Pinto, Giuseppe & Piscitelli, Marco Savino & Vázquez-Canteli, José Ramón & Nagy, Zoltán & Capozzoli, Alfonso, 2021. "Coordinated energy management for a cluster of buildings through deep reinforcement learning," Energy, Elsevier, vol. 229(C).
    7. Pinto, Giuseppe & Deltetto, Davide & Capozzoli, Alfonso, 2021. "Data-driven district energy management with surrogate models and deep reinforcement learning," Applied Energy, Elsevier, vol. 304(C).
    8. Homod, Raad Z. & Togun, Hussein & Kadhim Hussein, Ahmed & Noraldeen Al-Mousawi, Fadhel & Yaseen, Zaher Mundher & Al-Kouz, Wael & Abd, Haider J. & Alawi, Omer A. & Goodarzi, Marjan & Hussein, Omar A., 2022. "Dynamics analysis of a novel hybrid deep clustering for unsupervised learning by reinforcement of multi-agent to energy saving in intelligent buildings," Applied Energy, Elsevier, vol. 313(C).
    9. Davide Deltetto & Davide Coraci & Giuseppe Pinto & Marco Savino Piscitelli & Alfonso Capozzoli, 2021. "Exploring the Potentialities of Deep Reinforcement Learning for Incentive-Based Demand Response in a Cluster of Small Commercial Buildings," Energies, MDPI, vol. 14(10), pages 1-25, May.
    10. Clara Ceccolini & Roozbeh Sangi, 2022. "Benchmarking Approaches for Assessing the Performance of Building Control Strategies: A Review," Energies, MDPI, vol. 15(4), pages 1-30, February.
    11. Wang, Zhe & Hong, Tianzhen, 2020. "Reinforcement learning for building controls: The opportunities and challenges," Applied Energy, Elsevier, vol. 269(C).
    12. Han, Gwangwoo & Joo, Hong-Jin & Lim, Hee-Won & An, Young-Sub & Lee, Wang-Je & Lee, Kyoung-Ho, 2023. "Data-driven heat pump operation strategy using rainbow deep reinforcement learning for significant reduction of electricity cost," Energy, Elsevier, vol. 270(C).
    13. Gao, Yuan & Matsunami, Yuki & Miyata, Shohei & Akashi, Yasunori, 2022. "Multi-agent reinforcement learning dealing with hybrid action spaces: A case study for off-grid oriented renewable building energy system," Applied Energy, Elsevier, vol. 326(C).
    14. Lei, Yue & Zhan, Sicheng & Ono, Eikichi & Peng, Yuzhen & Zhang, Zhiang & Hasama, Takamasa & Chong, Adrian, 2022. "A practical deep reinforcement learning framework for multivariate occupant-centric control in buildings," Applied Energy, Elsevier, vol. 324(C).
    15. Zini, Marco & Carcasci, Carlo, 2023. "Machine learning-based monitoring method for the electricity consumption of a healthcare facility in Italy," Energy, Elsevier, vol. 262(PB).
    16. Gao, Yuan & Matsunami, Yuki & Miyata, Shohei & Akashi, Yasunori, 2022. "Operational optimization for off-grid renewable building energy system using deep reinforcement learning," Applied Energy, Elsevier, vol. 325(C).
    17. Du, Kun & Calautit, John & Eames, Philip & Wu, Yupeng, 2021. "A state-of-the-art review of the application of phase change materials (PCM) in Mobilized-Thermal Energy Storage (M-TES) for recovering low-temperature industrial waste heat (IWH) for distributed heat," Renewable Energy, Elsevier, vol. 168(C), pages 1040-1057.
    18. Zhang, Chengyu & Ma, Liangdong & Luo, Zhiwen & Han, Xing & Zhao, Tianyi, 2024. "Forecasting building plug load electricity consumption employing occupant-building interaction input features and bidirectional LSTM with improved swarm intelligent algorithms," Energy, Elsevier, vol. 288(C).
    19. Ferenc Kalmár & Tünde Kalmár, 2020. "Thermal Comfort Aspects of Solar Gains during the Heating Season," Energies, MDPI, vol. 13(7), pages 1-15, April.
    20. Mikhail Tokarev, 2019. "A Double-Bed Adsorptive Heat Transformer for Upgrading Ambient Heat: Design and First Tests," Energies, MDPI, vol. 12(21), pages 1-14, October.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:14:y:2021:i:4:p:997-:d:499227. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.