IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v258y2022ics0360544222017091.html
   My bibliography  Save this article

High robustness energy management strategy of hybrid electric vehicle based on improved soft actor-critic deep reinforcement learning

Author

Listed:
  • Sun, Wenjing
  • Zou, Yuan
  • Zhang, Xudong
  • Guo, Ningyuan
  • Zhang, Bin
  • Du, Guodong

Abstract

As a hybrid electric vehicle (HEV) key control technology, intelligent energy management strategies (EMSs) directly affect fuel consumption. Investigating the robustness of EMSs to maximize the advantages of energy savings and emission reduction in different driving environments is necessary. This article proposes a soft actor-critic (SAC) deep reinforcement learning (DRL) EMS for hybrid electric tracked vehicles (HETVs). Munchausen reinforcement learning (MRL) is adopted in the SAC algorithm, and the Munchausen SAC (MSAC) algorithm is constructed to achieve lower fuel consumption than the traditional SAC method. The prioritized experience replay (PER) is proposed to achieve more reasonable experience sampling and improve the optimization effect. To enhance the “cold start” performance, a dynamic programming (DP)-assisted training method is proposed that substantially improves the training efficiency. The proposed method optimization result is compared with the traditional SAC and deep deterministic policy gradient (DDPG) with PER through the simulation. The result shows that the proposed strategy improves both fuel consumption and possesses excellent robustness under different driving cycles.

Suggested Citation

  • Sun, Wenjing & Zou, Yuan & Zhang, Xudong & Guo, Ningyuan & Zhang, Bin & Du, Guodong, 2022. "High robustness energy management strategy of hybrid electric vehicle based on improved soft actor-critic deep reinforcement learning," Energy, Elsevier, vol. 258(C).
    • Handle: RePEc:eee:energy:v:258:y:2022:i:c:s0360544222017091
    DOI: 10.1016/j.energy.2022.124806
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0360544222017091
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.energy.2022.124806?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Xie, Shaobo & Hu, Xiaosong & Xin, Zongke & Brighton, James, 2019. "Pontryagin’s Minimum Principle based model predictive control of energy management for a plug-in hybrid electric bus," Applied Energy, Elsevier, vol. 236(C), pages 893-905.
    2. Guo, Ningyuan & Zhang, Xudong & Zou, Yuan & Guo, Lingxiong & Du, Guodong, 2021. "Real-time predictive energy management of plug-in hybrid electric vehicles for coordination of fuel economy and battery degradation," Energy, Elsevier, vol. 214(C).
    3. Lei, Zhenzhen & Qin, Datong & Hou, Liliang & Peng, Jingyu & Liu, Yonggang & Chen, Zheng, 2020. "An adaptive equivalent consumption minimization strategy for plug-in hybrid electric vehicles based on traffic information," Energy, Elsevier, vol. 190(C).
    4. Wu, Yuankai & Tan, Huachun & Peng, Jiankun & Zhang, Hailong & He, Hongwen, 2019. "Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus," Applied Energy, Elsevier, vol. 247(C), pages 454-466.
    5. Tribioli, Laura & Cozzolino, Raffaello & Chiappini, Daniele & Iora, Paolo, 2016. "Energy management of a plug-in fuel cell/battery hybrid vehicle with on-board fuel processing," Applied Energy, Elsevier, vol. 184(C), pages 140-154.
    6. Wu, Jingda & He, Hongwen & Peng, Jiankun & Li, Yuecheng & Li, Zhanjiang, 2018. "Continuous reinforcement learning of energy management with deep Q network for a power split hybrid electric bus," Applied Energy, Elsevier, vol. 222(C), pages 799-811.
    7. Zhang, Shuo & Hu, Xiaosong & Xie, Shaobo & Song, Ziyou & Hu, Lin & Hou, Cong, 2019. "Adaptively coordinated optimization of battery aging and energy management in plug-in hybrid electric buses," Applied Energy, Elsevier, vol. 256(C).
    8. Teng Liu & Yuan Zou & Dexing Liu & Fengchun Sun, 2015. "Reinforcement Learning–Based Energy Management Strategy for a Hybrid Electric Tracked Vehicle," Energies, MDPI, vol. 8(7), pages 1-18, July.
    9. Zhou, Jianhao & Xue, Siwu & Xue, Yuan & Liao, Yuhui & Liu, Jun & Zhao, Wanzhong, 2021. "A novel energy management strategy of hybrid electric vehicle via an improved TD3 deep reinforcement learning," Energy, Elsevier, vol. 224(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Chen, Fujun & Wang, Bowen & Ni, Meng & Gong, Zhichao & Jiao, Kui, 2024. "Online energy management strategy for ammonia-hydrogen hybrid electric vehicles harnessing deep reinforcement learning," Energy, Elsevier, vol. 301(C).
    2. Lee, Junhyeok & Shin, Youngchul & Moon, Ilkyeong, 2024. "A hybrid deep reinforcement learning approach for a proactive transshipment of fresh food in the online–offline channel system," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 187(C).
    3. Gao, Qinxiang & Lei, Tao & Yao, Wenli & Zhang, Xingyu & Zhang, Xiaobin, 2023. "A health-aware energy management strategy for fuel cell hybrid electric UAVs based on safe reinforcement learning," Energy, Elsevier, vol. 283(C).
    4. Wang, Hanchen & Arjmandzadeh, Ziba & Ye, Yiming & Zhang, Jiangfeng & Xu, Bin, 2024. "FlexNet: A warm start method for deep reinforcement learning in hybrid electric vehicle energy management applications," Energy, Elsevier, vol. 288(C).
    5. Liu, Yonggang & Wu, Yitao & Wang, Xiangyu & Li, Liang & Zhang, Yuanjian & Chen, Zheng, 2023. "Energy management for hybrid electric vehicles based on imitation reinforcement learning," Energy, Elsevier, vol. 263(PC).
    6. Chang, Chengcheng & Zhao, Wanzhong & Wang, Chunyan & Luan, Zhongkai, 2023. "An energy management strategy of deep reinforcement learning based on multi-agent architecture under self-generating conditions," Energy, Elsevier, vol. 283(C).
    7. He, Hongwen & Su, Qicong & Huang, Ruchen & Niu, Zegong, 2024. "Enabling intelligent transferable energy management of series hybrid electric tracked vehicle across motion dimensions via soft actor-critic algorithm," Energy, Elsevier, vol. 294(C).
    8. Yang, Xiaofeng & He, Hongwen & Wei, Zhongbao & Wang, Rui & Xu, Ke & Zhang, Dong, 2023. "Enabling Safety-Enhanced fast charging of electric vehicles via soft actor Critic-Lagrange DRL algorithm in a Cyber-Physical system," Applied Energy, Elsevier, vol. 329(C).
    9. Kang, Hyuna & Jung, Seunghoon & Kim, Hakpyeong & Jeoung, Jaewon & Hong, Taehoon, 2024. "Reinforcement learning-based optimal scheduling model of battery energy storage system at the building level," Renewable and Sustainable Energy Reviews, Elsevier, vol. 190(PA).
    10. Mudhafar Al-Saadi & Maher Al-Greer & Michael Short, 2023. "Reinforcement Learning-Based Intelligent Control Strategies for Optimal Power Management in Advanced Power Distribution Systems: A Survey," Energies, MDPI, vol. 16(4), pages 1-38, February.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dong, Peng & Zhao, Junwei & Liu, Xuewu & Wu, Jian & Xu, Xiangyang & Liu, Yanfang & Wang, Shuhan & Guo, Wei, 2022. "Practical application of energy management strategy for hybrid electric vehicles based on intelligent and connected technologies: Development stages, challenges, and future trends," Renewable and Sustainable Energy Reviews, Elsevier, vol. 170(C).
    2. Qi, Chunyang & Song, Chuanxue & Xiao, Feng & Song, Shixin, 2022. "Generalization ability of hybrid electric vehicle energy management strategy based on reinforcement learning method," Energy, Elsevier, vol. 250(C).
    3. Daniel Egan & Qilun Zhu & Robert Prucka, 2023. "A Review of Reinforcement Learning-Based Powertrain Controllers: Effects of Agent Selection for Mixed-Continuity Control and Reward Formulation," Energies, MDPI, vol. 16(8), pages 1-31, April.
    4. Iqbal, Najam & Wang, Hu & Zheng, Zunqing & Yao, Mingfa, 2024. "Reinforcement learning-based heuristic planning for optimized energy management in power-split hybrid electric heavy duty vehicles," Energy, Elsevier, vol. 302(C).
    5. Du, Guodong & Zou, Yuan & Zhang, Xudong & Guo, Lingxiong & Guo, Ningyuan, 2022. "Energy management for a hybrid electric vehicle based on prioritized deep reinforcement learning framework," Energy, Elsevier, vol. 241(C).
    6. Geng, Wenran & Lou, Diming & Wang, Chen & Zhang, Tong, 2020. "A cascaded energy management optimization method of multimode power-split hybrid electric vehicles," Energy, Elsevier, vol. 199(C).
    7. Du, Guodong & Zou, Yuan & Zhang, Xudong & Liu, Teng & Wu, Jinlong & He, Dingbo, 2020. "Deep reinforcement learning based energy management for a hybrid electric vehicle," Energy, Elsevier, vol. 201(C).
    8. Chen, Zheng & Hu, Hengjie & Wu, Yitao & Zhang, Yuanjian & Li, Guang & Liu, Yonggang, 2020. "Stochastic model predictive control for energy management of power-split plug-in hybrid electric vehicles based on reinforcement learning," Energy, Elsevier, vol. 211(C).
    9. Chen, Jiaxin & Shu, Hong & Tang, Xiaolin & Liu, Teng & Wang, Weida, 2022. "Deep reinforcement learning-based multi-objective control of hybrid power system combined with road recognition under time-varying environment," Energy, Elsevier, vol. 239(PC).
    10. Huang, Ruchen & He, Hongwen & Zhao, Xuyang & Wang, Yunlong & Li, Menglin, 2022. "Battery health-aware and naturalistic data-driven energy management for hybrid electric bus based on TD3 deep reinforcement learning algorithm," Applied Energy, Elsevier, vol. 321(C).
    11. Chen, Zheng & Gu, Hongji & Shen, Shiquan & Shen, Jiangwei, 2022. "Energy management strategy for power-split plug-in hybrid electric vehicle based on MPC and double Q-learning," Energy, Elsevier, vol. 245(C).
    12. Zhou, Jianhao & Xue, Yuan & Xu, Da & Li, Chaoxiong & Zhao, Wanzhong, 2022. "Self-learning energy management strategy for hybrid electric vehicle via curiosity-inspired asynchronous deep reinforcement learning," Energy, Elsevier, vol. 242(C).
    13. Li, Cheng & Xu, Xiangyang & Zhu, Helong & Gan, Jiongpeng & Chen, Zhige & Tang, Xiaolin, 2024. "Research on car-following control and energy management strategy of hybrid electric vehicles in connected scene," Energy, Elsevier, vol. 293(C).
    14. Cui, Wei & Cui, Naxin & Li, Tao & Cui, Zhongrui & Du, Yi & Zhang, Chenghui, 2022. "An efficient multi-objective hierarchical energy management strategy for plug-in hybrid electric vehicle in connected scenario," Energy, Elsevier, vol. 257(C).
    15. Zhou, Jianhao & Xue, Siwu & Xue, Yuan & Liao, Yuhui & Liu, Jun & Zhao, Wanzhong, 2021. "A novel energy management strategy of hybrid electric vehicle via an improved TD3 deep reinforcement learning," Energy, Elsevier, vol. 224(C).
    16. Yaqian Wang & Xiaohong Jiao, 2022. "Dual Heuristic Dynamic Programming Based Energy Management Control for Hybrid Electric Vehicles," Energies, MDPI, vol. 15(9), pages 1-19, April.
    17. Hu, Dong & Xie, Hui & Song, Kang & Zhang, Yuanyuan & Yan, Long, 2023. "An apprenticeship-reinforcement learning scheme based on expert demonstrations for energy management strategy of hybrid electric vehicles," Applied Energy, Elsevier, vol. 342(C).
    18. Hongzhe Li & Jinsong Kang & Cheng Li, 2024. "Energy Management Strategy Based on Reinforcement Learning and Frequency Decoupling for Fuel Cell Hybrid Powertrain," Energies, MDPI, vol. 17(8), pages 1-21, April.
    19. Peng, Hujun & Chen, Zhu & Li, Jianxiang & Deng, Kai & Dirkes, Steffen & Gottschalk, Jonas & Ünlübayir, Cem & Thul, Andreas & Löwenstein, Lars & Pischinger, Stefan & Hameyer, Kay, 2021. "Offline optimal energy management strategies considering high dynamics in batteries and constraints on fuel cell system power rate: From analytical derivation to validation on test bench," Applied Energy, Elsevier, vol. 282(PA).
    20. Alessia Musa & Pier Giuseppe Anselma & Giovanni Belingardi & Daniela Anna Misul, 2023. "Energy Management in Hybrid Electric Vehicles: A Q-Learning Solution for Enhanced Drivability and Energy Efficiency," Energies, MDPI, vol. 17(1), pages 1-20, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:258:y:2022:i:c:s0360544222017091. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.