Adaptive personalized federated reinforcement learning for multiple-ESS optimal market dispatch strategy with electric vehicles and photovoltaic power generations

The state-of-the-art centralized computing framework applied to the optimal market dispatch of energy storage systems (ESS) aggregates data from local ESS units for training on the cloud server due to the limited computing resources on edge. However, this approach poses several challenges, including the lack of joint optimization for multiple ESS units, susceptibility to single point of failure and attacks, and inadequate data privacy protection for ESS owners. This study proposes an adaptive personalized federated reinforcement learning (FRL) for multiple-ESS optimal dispatch in various electricity markets with electric vehicle and renewable energy, achieving both the joint optimization of multiple ESSs and avoiding the degraded performance of FRL's local model. Under an adaptively ESS-related differential privacy protection, local devices and the cloud server are specialized to form multiagent deep reinforcement learning (DRL) model for bidding energy, regulation, and third-party services and update the global models, respectively. Given the adaptability of personalization layer to different agents and clients, an adaptive personalization method is developed by calculating the number of personalization layers with the relative loss of each agent and client in the iteration process. The case study shows that the adaptive personalized FRL outperforms conventional FRL, DRL and optimization algorithms.