IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v305y2024ics0360544224021765.html
   My bibliography  Save this article

Deep reinforcement learning based adaptive energy management for plug-in hybrid electric vehicle with double deep Q-network

Author

Listed:
  • Shi, Dehua
  • Xu, Han
  • Wang, Shaohua
  • Hu, Jia
  • Chen, Long
  • Yin, Chunfang

Abstract

The equivalent consumption minimization strategy (ECMS) with pre-calibrated constant equivalence factor (EF) can ensure near global optimal solution for certain driving cycle and enable good real-time capability, but it is difficult to adapt to a wide range of driving conditions. To this end, aiming at the optimal energy management problem of a plug-in hybrid electric vehicle (PHEV), this paper proposes a deep reinforcement learning (DRL) based adaptive ECMS by combing the double deep Q-network (DDQN) and the driving cycle information. The DDQN is applied to correct the EF of the ECMS in a feed-forward manner with the battery state-of-charge (SOC) and the periodic predicted driving cycle information as inputs, and the ECMS is utilized to calculate the engine torque and gear ratio of the powertrain. The driving cycle information is represented by the average velocity, which is predicted by the historical velocity sequence based on the back-propagation (BP) neural network, and the difference of the average velocity between two continuous time windows. The hardware-in-the-loop (HIL) platform is constructed to test the performance of the proposed strategy. It is shown that the future average velocity can be well predicted by the historic velocity sequence. Both simulation and HIL test results demonstrate that the proposed adaptive ECMS based on DDQN exhibits superior performance in improving the vehicle fuel economy.

Suggested Citation

  • Shi, Dehua & Xu, Han & Wang, Shaohua & Hu, Jia & Chen, Long & Yin, Chunfang, 2024. "Deep reinforcement learning based adaptive energy management for plug-in hybrid electric vehicle with double deep Q-network," Energy, Elsevier, vol. 305(C).
    • Handle: RePEc:eee:energy:v:305:y:2024:i:c:s0360544224021765
    DOI: 10.1016/j.energy.2024.132402
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0360544224021765
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.energy.2024.132402?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Lian, Renzong & Peng, Jiankun & Wu, Yuankai & Tan, Huachun & Zhang, Hailong, 2020. "Rule-interposing deep reinforcement learning based energy management strategy for power-split hybrid electric vehicle," Energy, Elsevier, vol. 197(C).
    2. Yuping Zeng & Yang Cai & Guiyue Kou & Wei Gao & Datong Qin, 2018. "Energy Management for Plug-In Hybrid Electric Vehicle Based on Adaptive Simplified-ECMS," Sustainability, MDPI, vol. 10(6), pages 1-24, June.
    3. Shuxian Li & Minghui Hu & Changchao Gong & Sen Zhan & Datong Qin, 2018. "Energy Management Strategy for Hybrid Electric Vehicle Based on Driving Condition Identification Using KGA-Means," Energies, MDPI, vol. 11(6), pages 1-16, June.
    4. Baak, M. & Koopman, R. & Snoek, H. & Klous, S., 2020. "A new correlation coefficient between categorical, ordinal and interval variables with Pearson characteristics," Computational Statistics & Data Analysis, Elsevier, vol. 152(C).
    5. Guo, Qiuyi & Zhao, Zhiguo & Shen, Peihong & Zhan, Xiaowen & Li, Jingwei, 2019. "Adaptive optimal control based on driving style recognition for plug-in hybrid electric vehicle," Energy, Elsevier, vol. 186(C).
    6. Lei, Zhenzhen & Qin, Datong & Hou, Liliang & Peng, Jingyu & Liu, Yonggang & Chen, Zheng, 2020. "An adaptive equivalent consumption minimization strategy for plug-in hybrid electric vehicles based on traffic information," Energy, Elsevier, vol. 190(C).
    7. Tang, Wenbin & Wang, Yaqian & Jiao, Xiaohong & Ren, Lina, 2023. "Hierarchical energy management strategy based on adaptive dynamic programming for hybrid electric vehicles in car-following scenarios," Energy, Elsevier, vol. 265(C).
    8. Mahmoodi-k, Mehdi & Montazeri, Morteza & Madanipour, Vahid, 2021. "Simultaneous multi-objective optimization of a PHEV power management system and component sizing in real world traffic condition," Energy, Elsevier, vol. 233(C).
    9. Wu, Yuankai & Tan, Huachun & Peng, Jiankun & Zhang, Hailong & He, Hongwen, 2019. "Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus," Applied Energy, Elsevier, vol. 247(C), pages 454-466.
    10. Liu, Yonggang & Huang, Bin & Yang, Yang & Lei, Zhenzhen & Zhang, Yuanjian & Chen, Zheng, 2022. "Hierarchical speed planning and energy management for autonomous plug-in hybrid electric vehicle in vehicle-following environment," Energy, Elsevier, vol. 260(C).
    11. Xu, Bin & Rathod, Dhruvang & Zhang, Darui & Yebi, Adamu & Zhang, Xueyu & Li, Xiaoya & Filipi, Zoran, 2020. "Parametric study on reinforcement learning optimized energy management strategy for a hybrid electric vehicle," Applied Energy, Elsevier, vol. 259(C).
    12. Shi, Dehua & Liu, Sheng & Cai, Yingfeng & Wang, Shaohua & Li, Haoran & Chen, Long, 2021. "Pontryagin’s minimum principle based fuzzy adaptive energy management for hybrid electric vehicle using real-time traffic information," Applied Energy, Elsevier, vol. 286(C).
    13. Chen, Zheng & Gu, Hongji & Shen, Shiquan & Shen, Jiangwei, 2022. "Energy management strategy for power-split plug-in hybrid electric vehicle based on MPC and double Q-learning," Energy, Elsevier, vol. 245(C).
    14. Zou, Runnan & Fan, Likang & Dong, Yanrui & Zheng, Siyu & Hu, Chenxing, 2021. "DQL energy management: An online-updated algorithm and its application in fix-line hybrid electric vehicle," Energy, Elsevier, vol. 225(C).
    15. Li, Jie & Wu, Xiaodong & Fan, Jiawei & Liu, Yonggang & Xu, Min, 2023. "Overcoming driving challenges in complex urban traffic: A multi-objective eco-driving strategy via safety model based reinforcement learning," Energy, Elsevier, vol. 284(C).
    16. Sun, Xilei & Fu, Jianqin & Yang, Huiyong & Xie, Mingke & Liu, Jingping, 2023. "An energy management strategy for plug-in hybrid electric vehicles based on deep learning and improved model predictive control," Energy, Elsevier, vol. 269(C).
    17. Zhang, Zhen & Zhang, Tiezhu & Hong, Jichao & Zhang, Hongxin & Yang, Jian & Jia, Qingxiao, 2023. "Double deep Q-network guided energy management strategy of a novel electric-hydraulic hybrid electric vehicle," Energy, Elsevier, vol. 269(C).
    18. Xu, Nan & Kong, Yan & Yan, Jinyue & Zhang, Yuanjian & Sui, Yan & Ju, Hao & Liu, Heng & Xu, Zhe, 2022. "Global optimization energy management for multi-energy source vehicles based on “Information layer - Physical layer - Energy layer - Dynamic programming” (IPE-DP)," Applied Energy, Elsevier, vol. 312(C).
    19. Shen, Peihong & Zhao, Zhiguo & Zhan, Xiaowen & Li, Jingwei & Guo, Qiuyi, 2018. "Optimal energy management strategy for a plug-in hybrid electric commercial vehicle based on velocity prediction," Energy, Elsevier, vol. 155(C), pages 838-852.
    20. Xiaohong Jiao & Tielong Shen, 2014. "SDP Policy Iteration-Based Energy Management Strategy Using Traffic Information for Commuter Hybrid Electric Vehicles," Energies, MDPI, vol. 7(7), pages 1-28, July.
    21. Wang, Hanchen & Ye, Yiming & Zhang, Jiangfeng & Xu, Bin, 2023. "A comparative study of 13 deep reinforcement learning based energy management methods for a hybrid electric vehicle," Energy, Elsevier, vol. 266(C).
    22. Yang, Chao & Wang, Muyao & Wang, Weida & Pu, Zesong & Ma, Mingyue, 2021. "An efficient vehicle-following predictive energy management strategy for PHEV based on improved sequential quadratic programming algorithm," Energy, Elsevier, vol. 219(C).
    23. Lunz, Benedikt & Yan, Zexiong & Gerschler, Jochen Bernhard & Sauer, Dirk Uwe, 2012. "Influence of plug-in hybrid electric vehicle charging strategies on charging and battery degradation costs," Energy Policy, Elsevier, vol. 46(C), pages 511-519.
    24. Zhang, LiPeng & Liu, Wei & Qi, BingNan, 2020. "Energy optimization of multi-mode coupling drive plug-in hybrid electric vehicles based on speed prediction," Energy, Elsevier, vol. 206(C).
    25. Wang, Shaohua & Zhang, Kaimei & Shi, Dehua & Li, Meng & Yin, Chunfang, 2024. "Research on economical shifting strategy for multi-gear and multi-mode parallel plug-in HEV based on DIRECT algorithm," Energy, Elsevier, vol. 286(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dong, Peng & Zhao, Junwei & Liu, Xuewu & Wu, Jian & Xu, Xiangyang & Liu, Yanfang & Wang, Shuhan & Guo, Wei, 2022. "Practical application of energy management strategy for hybrid electric vehicles based on intelligent and connected technologies: Development stages, challenges, and future trends," Renewable and Sustainable Energy Reviews, Elsevier, vol. 170(C).
    2. Daniel Egan & Qilun Zhu & Robert Prucka, 2023. "A Review of Reinforcement Learning-Based Powertrain Controllers: Effects of Agent Selection for Mixed-Continuity Control and Reward Formulation," Energies, MDPI, vol. 16(8), pages 1-31, April.
    3. Penghui Qiang & Peng Wu & Tao Pan & Huaiquan Zang, 2021. "Real-Time Approximate Equivalent Consumption Minimization Strategy Based on the Single-Shaft Parallel Hybrid Powertrain," Energies, MDPI, vol. 14(23), pages 1-22, November.
    4. Shi, Wenzhuo & Huangfu, Yigeng & Xu, Liangcai & Pang, Shengzhao, 2022. "Online energy management strategy considering fuel cell fault for multi-stack fuel cell hybrid vehicle based on multi-agent reinforcement learning," Applied Energy, Elsevier, vol. 328(C).
    5. Qi, Chunyang & Song, Chuanxue & Xiao, Feng & Song, Shixin, 2022. "Generalization ability of hybrid electric vehicle energy management strategy based on reinforcement learning method," Energy, Elsevier, vol. 250(C).
    6. Nie, Zhigen & Jia, Yuan & Wang, Wanqiong & Chen, Zheng & Outbib, Rachid, 2022. "Co-optimization of speed planning and energy management for intelligent fuel cell hybrid vehicle considering complex traffic conditions," Energy, Elsevier, vol. 247(C).
    7. Gao, Kai & Luo, Pan & Xie, Jin & Chen, Bin & Wu, Yue & Du, Ronghua, 2023. "Energy management of plug-in hybrid electric vehicles based on speed prediction fused driving intention and LIDAR," Energy, Elsevier, vol. 284(C).
    8. Tan, Yingqi & Xu, Jingyi & Ma, Junyi & Li, Zirui & Chen, Huiyan & Xi, Junqiang & Liu, Haiou, 2024. "A transferable perception-guided EMS for series hybrid electric unmanned tracked vehicles," Energy, Elsevier, vol. 306(C).
    9. Iqbal, Najam & Wang, Hu & Zheng, Zunqing & Yao, Mingfa, 2024. "Reinforcement learning-based heuristic planning for optimized energy management in power-split hybrid electric heavy duty vehicles," Energy, Elsevier, vol. 302(C).
    10. Han, Lijin & You, Congwen & Yang, Ningkang & Liu, Hui & Chen, Ke & Xiang, Changle, 2024. "Adaptive real-time energy management strategy using heuristic search for off-road hybrid electric vehicles," Energy, Elsevier, vol. 304(C).
    11. Miranda, Matheus H.R. & Silva, Fabrício L. & Lourenço, Maria A.M. & Eckert, Jony J. & Silva, Ludmila C.A., 2022. "Vehicle drivetrain and fuzzy controller optimization using a planar dynamics simulation based on a real-world driving cycle," Energy, Elsevier, vol. 257(C).
    12. Chen, Z. & Liu, Y. & Ye, M. & Zhang, Y. & Chen, Z. & Li, G., 2021. "A survey on key techniques and development perspectives of equivalent consumption minimisation strategy for hybrid electric vehicles," Renewable and Sustainable Energy Reviews, Elsevier, vol. 151(C).
    13. Chen, Jiaxin & Shu, Hong & Tang, Xiaolin & Liu, Teng & Wang, Weida, 2022. "Deep reinforcement learning-based multi-objective control of hybrid power system combined with road recognition under time-varying environment," Energy, Elsevier, vol. 239(PC).
    14. Guo, Xiaokai & Yan, Xianguo & Chen, Zhi & Meng, Zhiyu, 2022. "Research on energy management strategy of heavy-duty fuel cell hybrid vehicles based on dueling-double-deep Q-network," Energy, Elsevier, vol. 260(C).
    15. Marouane Adnane & Ahmed Khoumsi & João Pedro F. Trovão, 2023. "Efficient Management of Energy Consumption of Electric Vehicles Using Machine Learning—A Systematic and Comprehensive Survey," Energies, MDPI, vol. 16(13), pages 1-39, June.
    16. Cui, Wei & Cui, Naxin & Li, Tao & Cui, Zhongrui & Du, Yi & Zhang, Chenghui, 2022. "An efficient multi-objective hierarchical energy management strategy for plug-in hybrid electric vehicle in connected scenario," Energy, Elsevier, vol. 257(C).
    17. Hou, Zhuoran & Guo, Jianhua & Chu, Liang & Hu, Jincheng & Chen, Zheng & Zhang, Yuanjian, 2023. "Exploration the route of information integration for vehicle design: A knowledge-enhanced energy management strategy," Energy, Elsevier, vol. 282(C).
    18. Yaqian Wang & Xiaohong Jiao, 2022. "Dual Heuristic Dynamic Programming Based Energy Management Control for Hybrid Electric Vehicles," Energies, MDPI, vol. 15(9), pages 1-19, April.
    19. Alessia Musa & Pier Giuseppe Anselma & Giovanni Belingardi & Daniela Anna Misul, 2023. "Energy Management in Hybrid Electric Vehicles: A Q-Learning Solution for Enhanced Drivability and Energy Efficiency," Energies, MDPI, vol. 17(1), pages 1-20, December.
    20. Baodi Zhang & Sheng Guo & Xin Zhang & Qicheng Xue & Lan Teng, 2020. "Adaptive Smoothing Power Following Control Strategy Based on an Optimal Efficiency Map for a Hybrid Electric Tracked Vehicle," Energies, MDPI, vol. 13(8), pages 1-25, April.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:305:y:2024:i:c:s0360544224021765. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.