Robust federated deep reinforcement learning for optimal control in multiple virtual power plants with electric vehicles

The deployment of virtual power plants (VPPs) with electric vehicles (EVs) is crucial for the successful integration of renewable energy sources and efficient management of EV charging and discharging while maintaining sustainability and cost-effectiveness. Deep reinforcement learning (DRL) is a highly promising method that uses historical data to learn optimal control strategies and adapts to a wide range of real-time scenarios. To address data privacy concerns in VPPs, federated DRL, which trains models across multiple VPPs, is necessary. However, existing federated DRL methods are prone to disturbance, which can severely impact system performance. This paper proposes a robust federated DRL method to ensure the robustness and reliability of VPP control strategies. Firstly, we formulate the control strategies of multiple VPPs as a Markov decision process that takes into account disturbances, aiming to achieve self-balance as much as possible. Secondly, we employ the stochastically controlled stochastic gradient method to increase training speed. Additionally, we introduce the robust gradient filter to develop a robust federated DRL method based on policy-based DRL. Finally, we validate the effectiveness and robustness of the proposed robust federated DRL method, which maintains balance in internal VPP power.