IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v18y2025i5p1280-d1606027.html
   My bibliography  Save this article

A Comparative Study of Energy Management Strategies for Battery-Ultracapacitor Electric Vehicles Based on Different Deep Reinforcement Learning Methods

Author

Listed:
  • Wenna Xu

    (School of Mechanical Engineering, Sichuan University of Science and Engineering, Yibin 644000, China)

  • Hao Huang

    (School of Mechanical Engineering, Sichuan University of Science and Engineering, Yibin 644000, China)

  • Chun Wang

    (School of Mechanical Engineering, Sichuan University of Science and Engineering, Yibin 644000, China
    Sichuan Provincial Key Lab of Process Equipment and Control, Sichuan University of Science and Engineering, Yibin 644000, China)

  • Shuai Xia

    (School of Mechanical Engineering, Sichuan University of Science and Engineering, Yibin 644000, China)

  • Xinmei Gao

    (School of Mechanical Engineering, Sichuan University of Science and Engineering, Yibin 644000, China
    Sichuan Provincial Key Lab of Process Equipment and Control, Sichuan University of Science and Engineering, Yibin 644000, China)

Abstract

An efficient energy management strategy (EMS) is crucial for the energy-saving and emission-reduction effects of electric vehicles. Research on deep reinforcement learning (DRL)-driven energy management systems (EMSs) has made significant strides in the global automotive industry. However, most scholars study only the impact of a single DRL algorithm on EMS performance, ignoring the potential improvement in optimization objectives that different DRL algorithms can offer under the same benchmark. This paper focuses on the control strategy of hybrid energy storage systems (HESSs) comprising lithium-ion batteries and ultracapacitors. Firstly, an equivalent model of the HESS is established based on dynamic experiments. Secondly, a regulated decision-making framework is constructed by uniformly setting the action space, state space, reward function, and hyperparameters of the agent for different DRL algorithms. To compare the control performances of the HESS under various EMSs, the regulation properties are analyzed with the standard driving cycle condition. Finally, the simulation results indicate that the EMS powered by a deep Q network (DQN) markedly diminishes the detrimental impact of peak current on the battery. Furthermore, the EMS based on a deep deterministic policy gradient (DDPG) reduces energy loss by 28.3%, and the economic efficiency of the EMS based on dynamic programming (DP) is improved to 0.7%.

Suggested Citation

  • Wenna Xu & Hao Huang & Chun Wang & Shuai Xia & Xinmei Gao, 2025. "A Comparative Study of Energy Management Strategies for Battery-Ultracapacitor Electric Vehicles Based on Different Deep Reinforcement Learning Methods," Energies, MDPI, vol. 18(5), pages 1-18, March.
    • Handle: RePEc:gam:jeners:v:18:y:2025:i:5:p:1280-:d:1606027
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/18/5/1280/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1996-1073/18/5/1280/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Liu, Yonggang & Liu, Junjun & Zhang, Yuanjian & Wu, Yitao & Chen, Zheng & Ye, Ming, 2020. "Rule learning based energy management strategy of fuel cell hybrid vehicles considering multi-objective optimization," Energy, Elsevier, vol. 207(C).
    2. He, Hongwen & Wang, Yunlong & Han, Ruoyan & Han, Mo & Bai, Yunfei & Liu, Qingwu, 2021. "An improved MPC-based energy management strategy for hybrid vehicles using V2V and V2I communications," Energy, Elsevier, vol. 225(C).
    3. Qi, Chunyang & Zhu, Yiwen & Song, Chuanxue & Yan, Guangfu & Xiao, Feng & Da wang, & Zhang, Xu & Cao, Jingwei & Song, Shixin, 2022. "Hierarchical reinforcement learning based energy management strategy for hybrid electric vehicle," Energy, Elsevier, vol. 238(PA).
    4. Wang, Hao & He, Hongwen & Bai, Yunfei & Yue, Hongwei, 2022. "Parameterized deep Q-network based energy management with balanced energy economy and battery life for hybrid electric vehicles," Applied Energy, Elsevier, vol. 320(C).
    5. Brunelli, Lorenzo & Capancioni, Alessandro & Canè, Stella & Cecchini, Giammarco & Perazzo, Alessandro & Brusa, Alessandro & Cavina, Nicolò, 2023. "A predictive control strategy based on A-ECMS to handle Zero-Emission Zones: Performance assessment and testing using an HiL equipped with vehicular connectivity," Applied Energy, Elsevier, vol. 340(C).
    6. Zou, Yuan & Liu, Teng & Liu, Dexing & Sun, Fengchun, 2016. "Reinforcement learning-based real-time energy management for a hybrid tracked vehicle," Applied Energy, Elsevier, vol. 171(C), pages 372-382.
    7. Tian, Xiang & Cai, Yingfeng & Sun, Xiaodong & Zhu, Zhen & Xu, Yiqiang, 2019. "An adaptive ECMS with driving style recognition for energy optimization of parallel hybrid electric buses," Energy, Elsevier, vol. 189(C).
    8. Xiong, Rui & Cao, Jiayi & Yu, Quanqing, 2018. "Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle," Applied Energy, Elsevier, vol. 211(C), pages 538-548.
    9. Sarvaiya, Shradhdha & Ganesh, Sachin & Xu, Bin, 2021. "Comparative analysis of hybrid vehicle energy management strategies with optimization of fuel economy and battery life," Energy, Elsevier, vol. 228(C).
    10. Ye, Yiming & Wang, Hanchen & Xu, Bin & Zhang, Jiangfeng, 2023. "An imitation learning-based energy management strategy for electric vehicles considering battery aging," Energy, Elsevier, vol. 283(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zhang, Hao & Chen, Boli & Lei, Nuo & Li, Bingbing & Chen, Chaoyi & Wang, Zhi, 2024. "Coupled velocity and energy management optimization of connected hybrid electric vehicles for maximum collective efficiency," Applied Energy, Elsevier, vol. 360(C).
    2. Liu, Weirong & Yao, Pengfei & Wu, Yue & Duan, Lijun & Li, Heng & Peng, Jun, 2025. "Imitation reinforcement learning energy management for electric vehicles with hybrid energy storage system," Applied Energy, Elsevier, vol. 378(PA).
    3. Dong, Peng & Zhao, Junwei & Liu, Xuewu & Wu, Jian & Xu, Xiangyang & Liu, Yanfang & Wang, Shuhan & Guo, Wei, 2022. "Practical application of energy management strategy for hybrid electric vehicles based on intelligent and connected technologies: Development stages, challenges, and future trends," Renewable and Sustainable Energy Reviews, Elsevier, vol. 170(C).
    4. Liu, Yonggang & Wu, Yitao & Wang, Xiangyu & Li, Liang & Zhang, Yuanjian & Chen, Zheng, 2023. "Energy management for hybrid electric vehicles based on imitation reinforcement learning," Energy, Elsevier, vol. 263(PC).
    5. Wei, Zhengchao & Ma, Yue & Yang, Ningkang & Ruan, Shumin & Xiang, Changle, 2023. "Reinforcement learning based power management integrating economic rotational speed of turboshaft engine and safety constraints of battery for hybrid electric power system," Energy, Elsevier, vol. 263(PB).
    6. Yang, Ningkang & Han, Lijin & Xiang, Changle & Liu, Hui & Li, Xunmin, 2021. "An indirect reinforcement learning based real-time energy management strategy via high-order Markov Chain model for a hybrid electric vehicle," Energy, Elsevier, vol. 236(C).
    7. Lian, Renzong & Peng, Jiankun & Wu, Yuankai & Tan, Huachun & Zhang, Hailong, 2020. "Rule-interposing deep reinforcement learning based energy management strategy for power-split hybrid electric vehicle," Energy, Elsevier, vol. 197(C).
    8. Daniel Egan & Qilun Zhu & Robert Prucka, 2023. "A Review of Reinforcement Learning-Based Powertrain Controllers: Effects of Agent Selection for Mixed-Continuity Control and Reward Formulation," Energies, MDPI, vol. 16(8), pages 1-31, April.
    9. Wu, Yuankai & Tan, Huachun & Peng, Jiankun & Zhang, Hailong & He, Hongwen, 2019. "Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus," Applied Energy, Elsevier, vol. 247(C), pages 454-466.
    10. da Silva, Samuel Filgueira & Eckert, Jony Javorski & Corrêa, Fernanda Cristina & Silva, Fabrício Leonardo & Silva, Ludmila C.A. & Dedini, Franco Giuseppe, 2022. "Dual HESS electric vehicle powertrain design and fuzzy control based on multi-objective optimization to increase driving range and battery life cycle," Applied Energy, Elsevier, vol. 324(C).
    11. Li, Guidan & Yang, Zhe & Li, Bin & Bi, Huakun, 2019. "Power allocation smoothing strategy for hybrid energy storage system based on Markov decision process," Applied Energy, Elsevier, vol. 241(C), pages 152-163.
    12. Hu, Dong & Huang, Chao & Yin, Guodong & Li, Yangmin & Huang, Yue & Huang, Hailong & Wu, Jingda & Li, Wenfei & Xie, Hui, 2024. "A transfer-based reinforcement learning collaborative energy management strategy for extended-range electric buses with cabin temperature comfort consideration," Energy, Elsevier, vol. 290(C).
    13. Zhang, Cetengfei & Zhou, Quan & Hua, Min & Xu, Hongming & Bassett, Mike & Zhang, Fanggang, 2023. "Cuboid equivalent consumption minimization strategy for energy management of multi-mode plug-in hybrid vehicles considering diverse time scale objectives," Applied Energy, Elsevier, vol. 351(C).
    14. Shi, Wenzhuo & Huangfu, Yigeng & Xu, Liangcai & Pang, Shengzhao, 2022. "Online energy management strategy considering fuel cell fault for multi-stack fuel cell hybrid vehicle based on multi-agent reinforcement learning," Applied Energy, Elsevier, vol. 328(C).
    15. Chen, Zheng & Gu, Hongji & Shen, Shiquan & Shen, Jiangwei, 2022. "Energy management strategy for power-split plug-in hybrid electric vehicle based on MPC and double Q-learning," Energy, Elsevier, vol. 245(C).
    16. Liu, Yonggang & Liu, Junjun & Zhang, Yuanjian & Wu, Yitao & Chen, Zheng & Ye, Ming, 2020. "Rule learning based energy management strategy of fuel cell hybrid vehicles considering multi-objective optimization," Energy, Elsevier, vol. 207(C).
    17. Zhou, Jianhao & Xue, Yuan & Xu, Da & Li, Chaoxiong & Zhao, Wanzhong, 2022. "Self-learning energy management strategy for hybrid electric vehicle via curiosity-inspired asynchronous deep reinforcement learning," Energy, Elsevier, vol. 242(C).
    18. Guo, Xiaokai & Yan, Xianguo & Chen, Zhi & Meng, Zhiyu, 2022. "Research on energy management strategy of heavy-duty fuel cell hybrid vehicles based on dueling-double-deep Q-network," Energy, Elsevier, vol. 260(C).
    19. Marouane Adnane & Ahmed Khoumsi & João Pedro F. Trovão, 2023. "Efficient Management of Energy Consumption of Electric Vehicles Using Machine Learning—A Systematic and Comprehensive Survey," Energies, MDPI, vol. 16(13), pages 1-39, June.
    20. Vázquez-Canteli, José R. & Nagy, Zoltán, 2019. "Reinforcement learning for demand response: A review of algorithms and modeling techniques," Applied Energy, Elsevier, vol. 235(C), pages 1072-1089.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:18:y:2025:i:5:p:1280-:d:1606027. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.