IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v320y2022ics0306261922006274.html
   My bibliography  Save this article

Parameterized deep Q-network based energy management with balanced energy economy and battery life for hybrid electric vehicles

Author

Listed:
  • Wang, Hao
  • He, Hongwen
  • Bai, Yunfei
  • Yue, Hongwei

Abstract

Energy management strategy (EMS) is an essential technique to ensure the long- term driving economy of hybrid electric vehicles (HEVs). The complicated discrete–continuous hybrid action space lying in HEV’s driving system presents a challenge to achieve high-performance EMSs. Thus, this paper proposes a novel improved deep Q-network (DQN)-based EMS to reduce the HEV’s driving costs, with lithium-ion battery (LIB) life and energy economy considered. Firstly, a data-driven battery life map reflecting the non-linear decaying trajectory of battery state of health (SOH) is proposed to quantify the real-time battery aging. Secondly, in the proposed EMS incorporating the battery aging model, an enhanced parameterized DQN (PDQN) algorithm is applied to particularly provide a hybrid solution discriminating between discrete and continuous actions. Finally, with the dynamic programming (DP) method employed as the benchmark, the effectiveness and optimality of the proposed EMS are validated. Without the prior knowledge of testing driving conditions, the proposed EMS effectively achieves 99.5% performance of the DP method, reducing the vehicle’s driving costs by 3.1% and extending battery life effectively. The EMS converges quickly during training and a hardware-in-loop test validates its real application potential.

Suggested Citation

  • Wang, Hao & He, Hongwen & Bai, Yunfei & Yue, Hongwei, 2022. "Parameterized deep Q-network based energy management with balanced energy economy and battery life for hybrid electric vehicles," Applied Energy, Elsevier, vol. 320(C).
    • Handle: RePEc:eee:appene:v:320:y:2022:i:c:s0306261922006274
    DOI: 10.1016/j.apenergy.2022.119270
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261922006274
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2022.119270?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Xu, Bin & Rathod, Dhruvang & Zhang, Darui & Yebi, Adamu & Zhang, Xueyu & Li, Xiaoya & Filipi, Zoran, 2020. "Parametric study on reinforcement learning optimized energy management strategy for a hybrid electric vehicle," Applied Energy, Elsevier, vol. 259(C).
    2. Zhongbao Wei & Feng Leng & Zhongjie He & Wenyu Zhang & Kaiyuan Li, 2018. "Online State of Charge and State of Health Estimation for a Lithium-Ion Battery Based on a Data–Model Fusion Method," Energies, MDPI, vol. 11(7), pages 1-16, July.
    3. Kristen A. Severson & Peter M. Attia & Norman Jin & Nicholas Perkins & Benben Jiang & Zi Yang & Michael H. Chen & Muratahan Aykol & Patrick K. Herring & Dimitrios Fraggedakis & Martin Z. Bazant & Step, 2019. "Data-driven prediction of battery cycle life before capacity degradation," Nature Energy, Nature, vol. 4(5), pages 383-391, May.
    4. Wu, Jian & Wang, Xiangyu & Li, Liang & Qin, Cun'an & Du, Yongchang, 2018. "Hierarchical control strategy with battery aging consideration for hybrid electric vehicle regenerative braking control," Energy, Elsevier, vol. 145(C), pages 301-312.
    5. Tian, He & Lu, Ziwang & Wang, Xu & Zhang, Xinlong & Huang, Yong & Tian, Guangyu, 2016. "A length ratio based neural network energy management strategy for online control of plug-in hybrid electric city bus," Applied Energy, Elsevier, vol. 177(C), pages 71-80.
    6. Guo, Ningyuan & Zhang, Xudong & Zou, Yuan & Guo, Lingxiong & Du, Guodong, 2021. "Real-time predictive energy management of plug-in hybrid electric vehicles for coordination of fuel economy and battery degradation," Energy, Elsevier, vol. 214(C).
    7. Peng, Jiankun & He, Hongwen & Xiong, Rui, 2017. "Rule based energy management strategy for a series–parallel plug-in hybrid electric bus optimized by dynamic programming," Applied Energy, Elsevier, vol. 185(P2), pages 1633-1643.
    8. Omar, Noshin & Monem, Mohamed Abdel & Firouz, Yousef & Salminen, Justin & Smekens, Jelle & Hegazy, Omar & Gaulous, Hamid & Mulder, Grietus & Van den Bossche, Peter & Coosemans, Thierry & Van Mierlo, J, 2014. "Lithium iron phosphate based battery – Assessment of the aging parameters and development of cycle life model," Applied Energy, Elsevier, vol. 113(C), pages 1575-1585.
    9. Li, Shuangqi & He, Hongwen & Su, Chang & Zhao, Pengfei, 2020. "Data driven battery modeling and management method with aging phenomenon considered," Applied Energy, Elsevier, vol. 275(C).
    10. Wei, Zhongbao & Zhao, Jiyun & Ji, Dongxu & Tseng, King Jet, 2017. "A multi-timescale estimator for battery state of charge and capacity dual estimation based on an online identified model," Applied Energy, Elsevier, vol. 204(C), pages 1264-1274.
    11. Xixue Liu & Datong Qin & Shaoqian Wang, 2019. "Minimum Energy Management Strategy of Equivalent Fuel Consumption of Hybrid Electric Vehicle Based on Improved Global Optimization Equivalent Factor," Energies, MDPI, vol. 12(11), pages 1-17, May.
    12. Xie, Shanshan & He, Hongwen & Peng, Jiankun, 2017. "An energy management strategy based on stochastic model predictive control for plug-in hybrid electric buses," Applied Energy, Elsevier, vol. 196(C), pages 279-288.
    13. Xiang, Changle & Ding, Feng & Wang, Weida & He, Wei, 2017. "Energy management of a dual-mode power-split hybrid electric vehicle based on velocity prediction and nonlinear model predictive control," Applied Energy, Elsevier, vol. 189(C), pages 640-653.
    14. Du, Jiuyu & Zhang, Xiaobin & Wang, Tianze & Song, Ziyou & Yang, Xueqing & Wang, Hewu & Ouyang, Minggao & Wu, Xiaogang, 2018. "Battery degradation minimization oriented energy management strategy for plug-in hybrid electric bus with multi-energy storage system," Energy, Elsevier, vol. 165(PA), pages 153-163.
    15. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    16. Wu, Yuankai & Tan, Huachun & Peng, Jiankun & Zhang, Hailong & He, Hongwen, 2019. "Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus," Applied Energy, Elsevier, vol. 247(C), pages 454-466.
    17. Lyu, Chao & Lai, Qingzhi & Ge, Tengfei & Yu, Honghai & Wang, Lixin & Ma, Na, 2017. "A lead-acid battery's remaining useful life prediction by using electrochemical model in the Particle Filtering framework," Energy, Elsevier, vol. 120(C), pages 975-984.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jinming Xu & Yuan Lin, 2024. "Energy Management for Hybrid Electric Vehicles Using Safe Hybrid-Action Reinforcement Learning," Mathematics, MDPI, vol. 12(5), pages 1-20, February.
    2. Hu, Dong & Xie, Hui & Song, Kang & Zhang, Yuanyuan & Yan, Long, 2023. "An apprenticeship-reinforcement learning scheme based on expert demonstrations for energy management strategy of hybrid electric vehicles," Applied Energy, Elsevier, vol. 342(C).
    3. Zhang, Hao & Chen, Boli & Lei, Nuo & Li, Bingbing & Chen, Chaoyi & Wang, Zhi, 2024. "Coupled velocity and energy management optimization of connected hybrid electric vehicles for maximum collective efficiency," Applied Energy, Elsevier, vol. 360(C).
    4. Huang, Ruchen & He, Hongwen & Gao, Miaojue, 2023. "Training-efficient and cost-optimal energy management for fuel cell hybrid electric bus based on a novel distributed deep reinforcement learning framework," Applied Energy, Elsevier, vol. 346(C).
    5. Ji, Jie & Zhou, Mengxiong & Guo, Renwei & Tang, Jiankang & Su, Jiaoyue & Huang, Hui & Sun, Na & Nazir, Muhammad Shahzad & Wang, Yaodong, 2023. "A electric power optimal scheduling study of hybrid energy storage system integrated load prediction technology considering ageing mechanism," Renewable Energy, Elsevier, vol. 215(C).
    6. Hu, Dong & Huang, Chao & Yin, Guodong & Li, Yangmin & Huang, Yue & Huang, Hailong & Wu, Jingda & Li, Wenfei & Xie, Hui, 2024. "A transfer-based reinforcement learning collaborative energy management strategy for extended-range electric buses with cabin temperature comfort consideration," Energy, Elsevier, vol. 290(C).
    7. Zhang, Hao & Liu, Shang & Lei, Nuo & Fan, Qinhao & Wang, Zhi, 2022. "Leveraging the benefits of ethanol-fueled advanced combustion and supervisory control optimization in hybrid biofuel-electric vehicles," Applied Energy, Elsevier, vol. 326(C).
    8. He, Hongwen & Meng, Xiangfei & Wang, Yong & Khajepour, Amir & An, Xiaowen & Wang, Renguang & Sun, Fengchun, 2024. "Deep reinforcement learning based energy management strategies for electrified vehicles: Recent advances and perspectives," Renewable and Sustainable Energy Reviews, Elsevier, vol. 192(C).
    9. Ren, Xiaoxia & Ye, Jinze & Xie, Liping & Lin, Xinyou, 2024. "Battery longevity-conscious energy management predictive control strategy optimized by using deep reinforcement learning algorithm for a fuel cell hybrid electric vehicle," Energy, Elsevier, vol. 286(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dong, Peng & Zhao, Junwei & Liu, Xuewu & Wu, Jian & Xu, Xiangyang & Liu, Yanfang & Wang, Shuhan & Guo, Wei, 2022. "Practical application of energy management strategy for hybrid electric vehicles based on intelligent and connected technologies: Development stages, challenges, and future trends," Renewable and Sustainable Energy Reviews, Elsevier, vol. 170(C).
    2. Kong, Yan & Xu, Nan & Zhang, Yuanjian & Sui, Yan & Ju, Hao & Liu, Heng & Xu, Zhe, 2021. "Acquisition of full-factor trip information for global optimization energy management in multi-energy source vehicles and the measure of the amount of information to be transmitted," Energy, Elsevier, vol. 236(C).
    3. Manzolli, Jônatas Augusto & Trovão, João Pedro & Antunes, Carlos Henggeler, 2022. "A review of electric bus vehicles research topics – Methods and trends," Renewable and Sustainable Energy Reviews, Elsevier, vol. 159(C).
    4. Zhu, Tao & Wills, Richard G.A. & Lot, Roberto & Ruan, Haijun & Jiang, Zhihao, 2021. "Adaptive energy management of a battery-supercapacitor energy storage system for electric vehicles based on flexible perception and neural network fitting," Applied Energy, Elsevier, vol. 292(C).
    5. Chen, Jiaxin & Shu, Hong & Tang, Xiaolin & Liu, Teng & Wang, Weida, 2022. "Deep reinforcement learning-based multi-objective control of hybrid power system combined with road recognition under time-varying environment," Energy, Elsevier, vol. 239(PC).
    6. Li, Guozhen & Zhang, Zhenyu & Shi, Wankai & Li, Wenyong, 2023. "Energy management strategy and simulation analysis of a hybrid train based on a comprehensive efficiency optimization," Applied Energy, Elsevier, vol. 349(C).
    7. Guo, Xiaokai & Yan, Xianguo & Chen, Zhi & Meng, Zhiyu, 2022. "Research on energy management strategy of heavy-duty fuel cell hybrid vehicles based on dueling-double-deep Q-network," Energy, Elsevier, vol. 260(C).
    8. Wu, Yitao & Zhang, Yuanjian & Li, Guang & Shen, Jiangwei & Chen, Zheng & Liu, Yonggang, 2020. "A predictive energy management strategy for multi-mode plug-in hybrid electric vehicles based on multi neural networks," Energy, Elsevier, vol. 208(C).
    9. Liu, Teng & Tan, Wenhao & Tang, Xiaolin & Zhang, Jinwei & Xing, Yang & Cao, Dongpu, 2021. "Driving conditions-driven energy management strategies for hybrid electric vehicles: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 151(C).
    10. Du, Guodong & Zou, Yuan & Zhang, Xudong & Kong, Zehui & Wu, Jinlong & He, Dingbo, 2019. "Intelligent energy management for hybrid electric tracked vehicles using online reinforcement learning," Applied Energy, Elsevier, vol. 251(C), pages 1-1.
    11. Daniel Egan & Qilun Zhu & Robert Prucka, 2023. "A Review of Reinforcement Learning-Based Powertrain Controllers: Effects of Agent Selection for Mixed-Continuity Control and Reward Formulation," Energies, MDPI, vol. 16(8), pages 1-31, April.
    12. Chen, Z. & Liu, Y. & Ye, M. & Zhang, Y. & Chen, Z. & Li, G., 2021. "A survey on key techniques and development perspectives of equivalent consumption minimisation strategy for hybrid electric vehicles," Renewable and Sustainable Energy Reviews, Elsevier, vol. 151(C).
    13. Fengqi Zhang & Lihua Wang & Serdar Coskun & Hui Pang & Yahui Cui & Junqiang Xi, 2020. "Energy Management Strategies for Hybrid Electric Vehicles: Review, Classification, Comparison, and Outlook," Energies, MDPI, vol. 13(13), pages 1-35, June.
    14. da Silva, Samuel Filgueira & Eckert, Jony Javorski & Corrêa, Fernanda Cristina & Silva, Fabrício Leonardo & Silva, Ludmila C.A. & Dedini, Franco Giuseppe, 2022. "Dual HESS electric vehicle powertrain design and fuzzy control based on multi-objective optimization to increase driving range and battery life cycle," Applied Energy, Elsevier, vol. 324(C).
    15. López-Ibarra, Jon Ander & Gaztañaga, Haizea & Saez-de-Ibarra, Andoni & Camblong, Haritza, 2020. "Plug-in hybrid electric buses total cost of ownership optimization at fleet level based on battery aging," Applied Energy, Elsevier, vol. 280(C).
    16. Chen, Zheng & Gu, Hongji & Shen, Shiquan & Shen, Jiangwei, 2022. "Energy management strategy for power-split plug-in hybrid electric vehicle based on MPC and double Q-learning," Energy, Elsevier, vol. 245(C).
    17. Hu, Dong & Xie, Hui & Song, Kang & Zhang, Yuanyuan & Yan, Long, 2023. "An apprenticeship-reinforcement learning scheme based on expert demonstrations for energy management strategy of hybrid electric vehicles," Applied Energy, Elsevier, vol. 342(C).
    18. Alessia Musa & Pier Giuseppe Anselma & Giovanni Belingardi & Daniela Anna Misul, 2023. "Energy Management in Hybrid Electric Vehicles: A Q-Learning Solution for Enhanced Drivability and Energy Efficiency," Energies, MDPI, vol. 17(1), pages 1-20, December.
    19. Baodi Zhang & Sheng Guo & Xin Zhang & Qicheng Xue & Lan Teng, 2020. "Adaptive Smoothing Power Following Control Strategy Based on an Optimal Efficiency Map for a Hybrid Electric Tracked Vehicle," Energies, MDPI, vol. 13(8), pages 1-25, April.
    20. Qi, Chunyang & Song, Chuanxue & Xiao, Feng & Song, Shixin, 2022. "Generalization ability of hybrid electric vehicle energy management strategy based on reinforcement learning method," Energy, Elsevier, vol. 250(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:320:y:2022:i:c:s0306261922006274. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.