Energy management in integrated energy system with electric vehicles as mobile energy storage: An approach using bi-level deep reinforcement learning

The integrated energy system with electric vehicle charging station via vehicle-to-grid aims to offer a proactive solution for low-carbon development of both energy and transportation sectors. However, achieving optimal energy efficiency with minimal operational costs in such a complex system is challenging due to the high randomness of electric vehicle travel patterns. This work proposes a reinforcement learning-based energy management framework to optimize the coordinated scheduling of integrated energy and vehicle-to-grid. The problem is modeled as a Markov Decision Process and optimized using the bi-level soft actor-critic algorithm, where the upper-level agent supervises the scheduling of the integrated energy system, and lower-level agent sets the electric vehicles’ charging and discharging strategies based on upper-level decision. Numerical simulations demonstrated that by adopting a bi-level reinforcement learning approach, the proposed algorithm effectively enhances energy exchange between integrated energy and electric vehicle charging station, reducing operational costs by 8 % compared to other multi-agent algorithms. Furthermore, when compared to the Mixed-Integer Linear Programming model, the proposed algorithm achieves similar optimality with significantly less computation time, maintaining a margin of error of approximately 4 %. Additionally, integrating electric vehicles as mobile energy storage within this framework can lead to a further 10 % reduction in operating costs.