Scalable energy management approach of residential hybrid energy system using multi-agent deep reinforcement learning

Deploying renewable energy and implementing smart energy management strategies are crucial for decarbonizing Building Energy Systems (BES). Despite recent advancements in data-driven Deep Reinforcement Learning (DRL) for BES optimization, significant challenges still exist, such as the time-consuming and data-intensive nature of training DRL controllers and the complexity of environment dynamics in Multi-Agent Reinforcement Learning (MARL). Consequently, these obstacles impede the synchronization and coordination of multiple agent control, leading to slow DRL convergence performance. To address these issues. This paper proposes a novel approach to optimize hybrid building energy systems. We introduce an integrated system combining a multi-stage Proximal Policy Optimization (PPO) on-policy framework with Imitation Learning (IL), interacting with the model environment. To improve scalability and robustness of Multi-agent Systems (MAS), this approach is designed to enhance training efficiency with centralized training and decentralized execution. Simulation results of case studies demonstrate the effectiveness of the Multi-agent Deep Reinforcement Learning (MADRL) model in optimizing the operations of hybrid building energy systems in terms of indoor thermal comfort and energy efficiency. Results show the proposed framework significantly improve performance in achieving convergence in just 50 episodes for dynamic decision-making. The scalability and robustness of the proposed model have been validated across various scenarios. Compared with the baseline during cold and warm weeks, the proposed control approach achieved improvements of 34.86% and 46.10% in energy self-sufficiency ratio, respectively. Additionally, the developed MADRL effectively improved solar photovoltaic (PV) self-consumption and reduced household energy costs. Notably, it increased the average indoor temperature closer to the desired set-point by 1.33 °C, and improved the self-consumption ratio by 15.78% in the colder week and 18.47% in the warmer week, compared to baseline measurements. These findings highlight the advantages of the multi-stage PPO on-policy framework, enabling faster learning and reduced training time, resulting in cost-effective solutions and enhanced solar PV self-consumption.