IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v378y2025ipas0306261924022153.html
   My bibliography  Save this article

Imitation reinforcement learning energy management for electric vehicles with hybrid energy storage system

Author

Listed:
  • Liu, Weirong
  • Yao, Pengfei
  • Wu, Yue
  • Duan, Lijun
  • Li, Heng
  • Peng, Jun

Abstract

Deep reinforcement learning has become a promising method for the energy management of electric vehicles. However, deep reinforcement learning relies on a large amount of trial-and-error training to acquire near-optimal performance. An adversarial imitation reinforcement learning energy management strategy is proposed for electric vehicles with hybrid energy storage system to minimize the cost of battery capacity loss. Firstly, the reinforcement learning exploration is guided by expert knowledge, which is generated by dynamic programming under various standard driving conditions. The expert knowledge is represented as the optimal power allocation mapping. Secondly, at the early imitation stage, the action of the reinforcement learning agent approaches the optimal power allocation mapping rapidly by using adversarial networks. Thirdly, a dynamic imitation weight is developed according to the Discriminator of adversarial networks, making the agent transit to self-explore the near-optimal power allocation under online driving conditions. Results demonstrate that the proposed strategy can accelerate the training by 42.60% while enhancing the reward by 15.79% compared with traditional reinforcement learning. Under different test driving cycles, the proposed method can further reduce the battery capacity loss cost by 5.1%–12.4%.

Suggested Citation

  • Liu, Weirong & Yao, Pengfei & Wu, Yue & Duan, Lijun & Li, Heng & Peng, Jun, 2025. "Imitation reinforcement learning energy management for electric vehicles with hybrid energy storage system," Applied Energy, Elsevier, vol. 378(PA).
  • Handle: RePEc:eee:appene:v:378:y:2025:i:pa:s0306261924022153
    DOI: 10.1016/j.apenergy.2024.124832
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261924022153
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.apenergy.2024.124832?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Song, Ziyou & Hofmann, Heath & Li, Jianqiu & Han, Xuebing & Ouyang, Minggao, 2015. "Optimization for a hybrid energy storage system in electric vehicles using dynamic programing approach," Applied Energy, Elsevier, vol. 139(C), pages 151-162.
    2. Song, Ziyou & Li, Jianqiu & Hou, Jun & Hofmann, Heath & Ouyang, Minggao & Du, Jiuyu, 2018. "The battery-supercapacitor hybrid energy storage system in electric vehicle applications: A case study," Energy, Elsevier, vol. 154(C), pages 433-441.
    3. Ganesh, Akhil Hannegudda & Xu, Bin, 2022. "A review of reinforcement learning based energy management systems for electrified powertrains: Progress, challenge, and potential solution," Renewable and Sustainable Energy Reviews, Elsevier, vol. 154(C).
    4. Song, Ziyou & Hofmann, Heath & Li, Jianqiu & Hou, Jun & Zhang, Xiaowu & Ouyang, Minggao, 2015. "The optimization of a hybrid energy storage system at subzero temperatures: Energy management strategy design and battery heating requirement analysis," Applied Energy, Elsevier, vol. 159(C), pages 576-588.
    5. Zhang, Lei & Hu, Xiaosong & Wang, Zhenpo & Ruan, Jiageng & Ma, Chengbin & Song, Ziyou & Dorrell, David G. & Pecht, Michael G., 2021. "Hybrid electrochemical energy storage systems: An overview for smart grid and electrified vehicle applications," Renewable and Sustainable Energy Reviews, Elsevier, vol. 139(C).
    6. Lian, Renzong & Peng, Jiankun & Wu, Yuankai & Tan, Huachun & Zhang, Hailong, 2020. "Rule-interposing deep reinforcement learning based energy management strategy for power-split hybrid electric vehicle," Energy, Elsevier, vol. 197(C).
    7. Liu, Yonggang & Wu, Yitao & Wang, Xiangyu & Li, Liang & Zhang, Yuanjian & Chen, Zheng, 2023. "Energy management for hybrid electric vehicles based on imitation reinforcement learning," Energy, Elsevier, vol. 263(PC).
    8. Wu, Yue & Huang, Zhiwu & Liao, Hongtao & Chen, Bin & Zhang, Xiaoyong & Zhou, Yanhui & Liu, Yongjie & Li, Heng & Peng, Jun, 2020. "Adaptive power allocation using artificial potential field with compensator for hybrid energy storage systems in electric vehicles," Applied Energy, Elsevier, vol. 257(C).
    9. Hua, Min & Zhang, Cetengfei & Zhang, Fanggang & Li, Zhi & Yu, Xiaoli & Xu, Hongming & Zhou, Quan, 2023. "Energy management of multi-mode plug-in hybrid electric vehicle using multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 348(C).
    10. Zou, Yuan & Liu, Teng & Liu, Dexing & Sun, Fengchun, 2016. "Reinforcement learning-based real-time energy management for a hybrid tracked vehicle," Applied Energy, Elsevier, vol. 171(C), pages 372-382.
    11. Wu, Jingda & He, Hongwen & Peng, Jiankun & Li, Yuecheng & Li, Zhanjiang, 2018. "Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus," Applied Energy, Elsevier, vol. 222(C), pages 799-811.
    12. Song, Ziyou & Li, Jianqiu & Han, Xuebing & Xu, Liangfei & Lu, Languang & Ouyang, Minggao & Hofmann, Heath, 2014. "Multi-objective optimization of a semi-active battery/supercapacitor energy storage system for electric vehicles," Applied Energy, Elsevier, vol. 135(C), pages 212-224.
    13. Xiong, Rui & Cao, Jiayi & Yu, Quanqing, 2018. "Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle," Applied Energy, Elsevier, vol. 211(C), pages 538-548.
    14. Wang, Yujie & Sun, Zhendong & Chen, Zonghai, 2019. "Development of energy management system based on a rule-based power distribution strategy for hybrid power sources," Energy, Elsevier, vol. 175(C), pages 1055-1066.
    15. Wang, Bin & Xu, Jun & Cao, Binggang & Ning, Bo, 2017. "Adaptive mode switch strategy based on simulated annealing optimization of a multi-mode hybrid energy storage system for electric vehicles," Applied Energy, Elsevier, vol. 194(C), pages 596-608.
    16. Wu, Yue & Huang, Zhiwu & Li, Dongjun & Li, Heng & Peng, Jun & Stroe, Daniel & Song, Ziyou, 2024. "Optimal battery thermal management for electric vehicles with battery degradation minimization," Applied Energy, Elsevier, vol. 353(PA).
    17. Ye, Yiming & Wang, Hanchen & Xu, Bin & Zhang, Jiangfeng, 2023. "An imitation learning-based energy management strategy for electric vehicles considering battery aging," Energy, Elsevier, vol. 283(C).
    18. Xu, Bin & Shi, Junzhe & Li, Sixu & Li, Huayi & Wang, Zhe, 2021. "Energy consumption and battery aging minimization using a Q-learning strategy for a battery/ultracapacitor electric vehicle," Energy, Elsevier, vol. 229(C).
    19. Li, Yuecheng & He, Hongwen & Khajepour, Amir & Wang, Hong & Peng, Jiankun, 2019. "Energy management for a power-split hybrid electric bus via deep reinforcement learning with terrain information," Applied Energy, Elsevier, vol. 255(C).
    20. Wu, Yue & Huang, Zhiwu & Hofmann, Heath & Liu, Yongjie & Huang, Jiahao & Hu, Xiaosong & Peng, Jun & Song, Ziyou, 2022. "Hierarchical predictive control for electric vehicles with hybrid energy storage system under vehicle-following scenarios," Energy, Elsevier, vol. 251(C).
    21. Zhou, Jianhao & Xue, Siwu & Xue, Yuan & Liao, Yuhui & Liu, Jun & Zhao, Wanzhong, 2021. "A novel energy management strategy of hybrid electric vehicle via an improved TD3 deep reinforcement learning," Energy, Elsevier, vol. 224(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Daniel Egan & Qilun Zhu & Robert Prucka, 2023. "A Review of Reinforcement Learning-Based Powertrain Controllers: Effects of Agent Selection for Mixed-Continuity Control and Reward Formulation," Energies, MDPI, vol. 16(8), pages 1-31, April.
    2. Xiao, B. & Ruan, J. & Yang, W. & Walker, P.D. & Zhang, N., 2021. "A review of pivotal energy management strategies for extended range electric vehicles," Renewable and Sustainable Energy Reviews, Elsevier, vol. 149(C).
    3. Dong, Peng & Zhao, Junwei & Liu, Xuewu & Wu, Jian & Xu, Xiangyang & Liu, Yanfang & Wang, Shuhan & Guo, Wei, 2022. "Practical application of energy management strategy for hybrid electric vehicles based on intelligent and connected technologies: Development stages, challenges, and future trends," Renewable and Sustainable Energy Reviews, Elsevier, vol. 170(C).
    4. Zhu, Tao & Wills, Richard G.A. & Lot, Roberto & Ruan, Haijun & Jiang, Zhihao, 2021. "Adaptive energy management of a battery-supercapacitor energy storage system for electric vehicles based on flexible perception and neural network fitting," Applied Energy, Elsevier, vol. 292(C).
    5. Zhu, Tao & Lot, Roberto & Wills, Richard G.A. & Yan, Xingda, 2020. "Sizing a battery-supercapacitor energy storage system with battery degradation consideration for high-performance electric vehicles," Energy, Elsevier, vol. 208(C).
    6. Yang, Weiwei & Ruan, Jiageng & Yang, Jue & Zhang, Nong, 2020. "Investigation of integrated uninterrupted dual input transmission and hybrid energy storage system for electric vehicles," Applied Energy, Elsevier, vol. 262(C).
    7. Yang, Ningkang & Han, Lijin & Xiang, Changle & Liu, Hui & Li, Xunmin, 2021. "An indirect reinforcement learning based real-time energy management strategy via high-order Markov Chain model for a hybrid electric vehicle," Energy, Elsevier, vol. 236(C).
    8. Tang, Tianfeng & Peng, Qianlong & Shi, Qing & Peng, Qingguo & Zhao, Jin & Chen, Chaoyi & Wang, Guangwei, 2024. "Energy management of fuel cell hybrid electric bus in mountainous regions: A deep reinforcement learning approach considering terrain characteristics," Energy, Elsevier, vol. 311(C).
    9. Wu, Yuankai & Tan, Huachun & Peng, Jiankun & Zhang, Hailong & He, Hongwen, 2019. "Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus," Applied Energy, Elsevier, vol. 247(C), pages 454-466.
    10. Hu, Lin & Tian, Qingtao & Zou, Changfu & Huang, Jing & Ye, Yao & Wu, Xianhui, 2022. "A study on energy distribution strategy of electric vehicle hybrid energy storage system considering driving style based on real urban driving data," Renewable and Sustainable Energy Reviews, Elsevier, vol. 162(C).
    11. Li, Guidan & Yang, Zhe & Li, Bin & Bi, Huakun, 2019. "Power allocation smoothing strategy for hybrid energy storage system based on Markov decision process," Applied Energy, Elsevier, vol. 241(C), pages 152-163.
    12. Hu, Dong & Huang, Chao & Yin, Guodong & Li, Yangmin & Huang, Yue & Huang, Hailong & Wu, Jingda & Li, Wenfei & Xie, Hui, 2024. "A transfer-based reinforcement learning collaborative energy management strategy for extended-range electric buses with cabin temperature comfort consideration," Energy, Elsevier, vol. 290(C).
    13. Liu, Zemin Eitan & Li, Yong & Zhou, Quan & Shuai, Bin & Hua, Min & Xu, Hongming & Xu, Lubing & Tan, Guikun & Li, Yanfei, 2025. "Real-time energy management for HEV combining naturalistic driving data and deep reinforcement learning with high generalization," Applied Energy, Elsevier, vol. 377(PA).
    14. Chi T. P. Nguyen & Bảo-Huy Nguyễn & Minh C. Ta & João Pedro F. Trovão, 2023. "Dual-Motor Dual-Source High Performance EV: A Comprehensive Review," Energies, MDPI, vol. 16(20), pages 1-28, October.
    15. Zhou, Jianhao & Xue, Yuan & Xu, Da & Li, Chaoxiong & Zhao, Wanzhong, 2022. "Self-learning energy management strategy for hybrid electric vehicle via curiosity-inspired asynchronous deep reinforcement learning," Energy, Elsevier, vol. 242(C).
    16. Huang, Ruchen & He, Hongwen & Su, Qicong, 2024. "Smart energy management for hybrid electric bus via improved soft actor-critic algorithm in a heuristic learning framework," Energy, Elsevier, vol. 309(C).
    17. Fuwu Yan & Jinhai Wang & Changqing Du & Min Hua, 2022. "Multi-Objective Energy Management Strategy for Hybrid Electric Vehicles Based on TD3 with Non-Parametric Reward Function," Energies, MDPI, vol. 16(1), pages 1-17, December.
    18. Song, Ziyou & Feng, Shuo & Zhang, Lei & Hu, Zunyan & Hu, Xiaosong & Yao, Rui, 2019. "Economy analysis of second-life battery in wind power systems considering battery degradation in dynamic processes: Real case scenarios," Applied Energy, Elsevier, vol. 251(C), pages 1-1.
    19. Wang, Yue & Li, Keqiang & Zeng, Xiaohua & Gao, Bolin & Hong, Jichao, 2023. "Investigation of novel intelligent energy management strategies for connected HEB considering global planning of fixed-route information," Energy, Elsevier, vol. 263(PB).
    20. Wu, Yue & Huang, Zhiwu & Li, Dongjun & Li, Heng & Peng, Jun & Stroe, Daniel & Song, Ziyou, 2024. "Optimal battery thermal management for electric vehicles with battery degradation minimization," Applied Energy, Elsevier, vol. 353(PA).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:378:y:2025:i:pa:s0306261924022153. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.