Reinforcement learning-based intelligent energy management architecture for hybrid construction machinery

Power allocation is of crucial significance to energy management system in the hybrid construction machinery (HCM). Most of the existing HCM energy management strategies are only formulated based on the predefined rules, which causes the system unable to adapt to the changeable and complicated working conditions, thus seriously limiting the energy saving potential of hybrid technology. In this paper, we build a reinforcement learning-based intelligent energy management architecture for HCM. Given the working conditions and operating characteristics of HCM, a Q-function updating method combining direct learning and indirect learning is proposed to enhance the performance and practicability of reinforcement learning. A virtual world model (VWM) is introduced to approximate the real-world environment and facilitate the identification of data-driven environment, so as to enhance the real-time performance and adaptability of the architecture. Based on the characteristics of HCM working conditions, the load cycle is subdivided, and the stationary Markov chain is employed to yield real-time transfer probability matrices of required power to accelerate the updating of the environment model. An HCM experiment platform is built, in which the typical signal of working condition is sampled for simulation. The results indicate that DYNA-Q based architecture outperforms Q-learning and rule-based strategy (RBS) in terms of adaptivity, real-time performance and optimality. The results also demonstrate that with the proposed architecture, the working condition of internal combustion engine (ICE) and the charge-discharge of ultracapacitor are more rational and efficient.