Integrated three-stage decentralized scheduling for virtual power plants: A model-assisted multi-agent reinforcement learning method

Virtual power plant (VPP) emerges as a promising integration and aggregation technology that facilitates the utilization of massive flexible demand-side resources (DSRs). However, non-negligible modeling errors and high-dimensional uncertainties involved in DSR aggregation threaten the delivery reliability and cost-effectiveness of VPP operation. To address this problem, this study proposes an integrated three-stage scheduling framework for VPPs and develops a model-assisted multi-agent reinforcement learning (MARL) approach. In the proposed framework, the VPP scheduling problem is formulated as a decentralized partially observable Markov Decision Process (Dec-POMDP), which depicts the complex interaction process among the three stages (bidding, re-dispatching and disaggregation). The interactions are evaluated by a comprehensive reward function, incorporating the trading and operation costs, as well as imbalance penalties. To enable decentralized decision-making, a model-assisted multi-agent proximal policy optimization (MA2PPO) algorithm is proposed, which trains a separate actor network for each aggregator. Additionally, the MA2PPO is augmented with a model-assisted safety decision-making method to accelerate the training process. Numerical simulation results verify that the proposed method enhances the delivery reliability and cost-effectiveness of the VPP, while achieving faster convergence time compared with purely model-free MARL methods.