Hierarchical reinforcement learning based energy management strategy of plug-in hybrid electric vehicle for ecological car-following process

The economy-oriented automated hybrid eclectic vehicles (HEV) provide great potential to save energy by optimizing both driving behaviors and power distribution. Recent advances in the ecological car following issue of HEV focus on fusing adaptive cruise control (ACC) and energy management system (EMS) by collaborative optimization. However, series control frameworks ACC+EMS breaks the internal coupling relation between motion control and energy distribution, leading to the natural limitation of its optimization. On the opposite, integrated ACC-EMS promises energy-saving improvement but brings complex optimization problems with multi-scale objectives and large exploration space. The huge computation load restricts the online application of ACC-EMS. To address these problems, a hierarchical reinforcement learning based ACC-EMS strategy is proposed with a hierarchical policy and non-hierarchical execution. The upper layer learns to plan state-of-charge and time-headway trajectories, while the low layer policy learns to achieve the expected goals by outputting control variables executed by the host vehicle. The proposed ACC-EMS strategy were self-learning by interaction in car-following scenario constructed with GPS data on I-880 highway. Comprehensive simulations show the proposed strategy has significantly improved the training speed and stability, compared to the offline global optimum, achieving the energy consumption difference of less than 3% and computational load of less than 600 times.