Multiagent deep reinforcement learning-based cooperative optimal operation with strong scalability for residential microgrid clusters

With the rapid development of smart home technology, residential microgrid (RM) clusters have become an important way to utilize the demand-side resources of large-scale housing. However, there are some key problems in existing RM cluster optimization methods, such as difficult in adapting to the local observable environment and with poor privacy and scalability. Therefore, this paper proposes a multi-agent deep reinforcement learning (MADRL)-based RM cluster optimization operation method. First, with the aim of minimizing the energy cost of each residence while satisfying the comfort level of residents and avoiding transformer overload, the optimization scheduling problem of an RM cluster is described as a Markov game with an unknown state transition probability function. Then, a novel MADRL method is proposed to determine the optimal operation strategy of multiple RMs in this game paradigm. Each agent in the proposed method contains a collective strategy model and an independent learner. The collective strategy model can simulate the energy consumption of other RMs in the system and reflect its operating behavior. In addition, an independent learner based on a soft actor-critic (SAC) framework is used to learn the optimal scheduling strategy interactively with the environment. The proposed method has a completely decentralized and scalable structure, which can deal with continuous high-dimensional state and action spaces only requires local observations and approximations during training. Finally, a numerical example is given to verify that the proposed method can not only learn a stable cooperative energy management strategy but can also be extended to large-scale RM cluster problems. This gives the strong scalability and a high potential for practical application.