IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v293y2024ics0360544224003360.html
   My bibliography  Save this article

Research on energy management strategy of fuel cell hybrid power via an improved TD3 deep reinforcement learning

Author

Listed:
  • Zhou, Yujie
  • Huang, Yin
  • Mao, Xuping
  • Kang, Zehao
  • Huang, Xuejin
  • Xuan, Dongji

Abstract

Fuel cell hybrid electric vehicles (FCHEV) are helping to advance the cause of environmental protection as a sustainable form of transportation. An effective energy management strategy (EMS) is crucial to reduce the usage cost of FCHEV and enhance SOC maintenance ability. This study establishes separate degradation models for fuel cells and lithium batteries, incorporating the decay factor of energy sources’ lifetime into the EMS. To address the sparse reward problem during training, a novel energy management strategy algorithm based on deep reinforcement learning is proposed, which combines the twin delayed deep deterministic policy gradient (TD3) algorithm framework with learning rate annealing (AL) and hindsight prioritized experience replay (HPER) optimization methods, resulting in strong training performance. Experimental results demonstrate significant advantages of the EMS based on the HPER_AL_TD3 algorithm over traditional TD3-based approaches. The proposed EMS exhibits superior adaptability to various driving cycles, ensuring stable SOC levels and reducing the overall usage cost. This research aims to enhance the learning capability of EMS based on deep reinforcement learning and contribute to the promotion of FCHEV.

Suggested Citation

  • Zhou, Yujie & Huang, Yin & Mao, Xuping & Kang, Zehao & Huang, Xuejin & Xuan, Dongji, 2024. "Research on energy management strategy of fuel cell hybrid power via an improved TD3 deep reinforcement learning," Energy, Elsevier, vol. 293(C).
    • Handle: RePEc:eee:energy:v:293:y:2024:i:c:s0360544224003360
    DOI: 10.1016/j.energy.2024.130564
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0360544224003360
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.energy.2024.130564?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Chen, Huicui & Pei, Pucheng & Song, Mancun, 2015. "Lifetime prediction and the economic lifetime of Proton Exchange Membrane fuel cells," Applied Energy, Elsevier, vol. 142(C), pages 154-163.
    2. Li, Weihan & Cui, Han & Nemeth, Thomas & Jansen, Jonathan & Ünlübayir, Cem & Wei, Zhongbao & Feng, Xuning & Han, Xuebing & Ouyang, Minggao & Dai, Haifeng & Wei, Xuezhe & Sauer, Dirk Uwe, 2021. "Cloud-based health-conscious energy management of hybrid battery systems in electric vehicles with deep reinforcement learning," Applied Energy, Elsevier, vol. 293(C).
    3. Yuping Zeng & Yang Cai & Guiyue Kou & Wei Gao & Datong Qin, 2018. "Energy Management for Plug-In Hybrid Electric Vehicle Based on Adaptive Simplified-ECMS," Sustainability, MDPI, vol. 10(6), pages 1-24, June.
    4. Qi, Chunyang & Zhu, Yiwen & Song, Chuanxue & Yan, Guangfu & Xiao, Feng & Da wang, & Zhang, Xu & Cao, Jingwei & Song, Shixin, 2022. "Hierarchical reinforcement learning based energy management strategy for hybrid electric vehicle," Energy, Elsevier, vol. 238(PA).
    5. Song, Ke & Ding, Yuhang & Hu, Xiao & Xu, Hongjie & Wang, Yimin & Cao, Jing, 2021. "Degradation adaptive energy management strategy using fuel cell state-of-health for fuel economy improvement of hybrid electric vehicle," Applied Energy, Elsevier, vol. 285(C).
    6. Wang, Hanchen & Ye, Yiming & Zhang, Jiangfeng & Xu, Bin, 2023. "A comparative study of 13 deep reinforcement learning based energy management methods for a hybrid electric vehicle," Energy, Elsevier, vol. 266(C).
    7. Xu, Liangfei & Mueller, Clemens David & Li, Jianqiu & Ouyang, Minggao & Hu, Zunyan, 2015. "Multi-objective component sizing based on optimal energy management strategy of fuel cell electric vehicles," Applied Energy, Elsevier, vol. 157(C), pages 664-674.
    8. David Silver & Aja Huang & Chris J. Maddison & Arthur Guez & Laurent Sifre & George van den Driessche & Julian Schrittwieser & Ioannis Antonoglou & Veda Panneershelvam & Marc Lanctot & Sander Dieleman, 2016. "Mastering the game of Go with deep neural networks and tree search," Nature, Nature, vol. 529(7587), pages 484-489, January.
    9. Huang, Ruchen & He, Hongwen & Zhao, Xuyang & Wang, Yunlong & Li, Menglin, 2022. "Battery health-aware and naturalistic data-driven energy management for hybrid electric bus based on TD3 deep reinforcement learning algorithm," Applied Energy, Elsevier, vol. 321(C).
    10. Justin M. Bracci & Evan D. Sherwin & Naomi L. Boness & Adam R. Brandt, 2023. "A cost comparison of various hourly-reliable and net-zero hydrogen production pathways in the United States," Nature Communications, Nature, vol. 14(1), pages 1-13, December.
    11. Ruan, Jiageng & Wu, Changcheng & Liang, Zhaowen & Liu, Kai & Li, Bin & Li, Weihan & Li, Tongyang, 2023. "The application of machine learning-based energy management strategy in a multi-mode plug-in hybrid electric vehicle, part II: Deep deterministic policy gradient algorithm design for electric mode," Energy, Elsevier, vol. 269(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. He, Hongwen & Meng, Xiangfei & Wang, Yong & Khajepour, Amir & An, Xiaowen & Wang, Renguang & Sun, Fengchun, 2024. "Deep reinforcement learning based energy management strategies for electrified vehicles: Recent advances and perspectives," Renewable and Sustainable Energy Reviews, Elsevier, vol. 192(C).
    2. Liang, Zhaowen & Ruan, Jiageng & Wang, Zhenpo & Liu, Kai & Li, Bin, 2024. "Soft actor-critic-based EMS design for dual motor battery electric bus," Energy, Elsevier, vol. 288(C).
    3. Chen, Jiaxin & Shu, Hong & Tang, Xiaolin & Liu, Teng & Wang, Weida, 2022. "Deep reinforcement learning-based multi-objective control of hybrid power system combined with road recognition under time-varying environment," Energy, Elsevier, vol. 239(PC).
    4. Chen, Dongfang & Pei, Pucheng & Meng, Yining & Ren, Peng & Li, Yuehua & Wang, Mingkai & Wang, Xizhong, 2022. "Novel extraction method of working condition spectrum for the lifetime prediction and energy management strategy evaluation of automotive fuel cells," Energy, Elsevier, vol. 255(C).
    5. Zhou, Jianhao & Xue, Yuan & Xu, Da & Li, Chaoxiong & Zhao, Wanzhong, 2022. "Self-learning energy management strategy for hybrid electric vehicle via curiosity-inspired asynchronous deep reinforcement learning," Energy, Elsevier, vol. 242(C).
    6. Feng, Yanbiao & Dong, Zuomin, 2020. "Integrated design and control optimization of fuel cell hybrid mining truck with minimized lifecycle cost," Applied Energy, Elsevier, vol. 270(C).
    7. Li, Tianyu & Huang, Lingtao & Liu, Huiying, 2019. "Energy management and economic analysis for a fuel cell supercapacitor excavator," Energy, Elsevier, vol. 172(C), pages 840-851.
    8. Song, Ke & Huang, Xing & Huang, Pengyu & Sun, Hui & Chen, Yuhui & Huang, Dongya, 2024. "Data-driven health state estimation and remaining useful life prediction of fuel cells," Renewable Energy, Elsevier, vol. 227(C).
    9. Iqbal, Mehroze & Becherif, Mohamed & Ramadan, Haitham S. & Badji, Abderrezak, 2021. "Dual-layer approach for systematic sizing and online energy management of fuel cell hybrid vehicles," Applied Energy, Elsevier, vol. 300(C).
    10. Huang, Ruchen & He, Hongwen & Gao, Miaojue, 2023. "Training-efficient and cost-optimal energy management for fuel cell hybrid electric bus based on a novel distributed deep reinforcement learning framework," Applied Energy, Elsevier, vol. 346(C).
    11. Menglin Li & Haoran Liu & Mei Yan & Hongyang Xu & Hongwen He, 2022. "A Novel Multi-Objective Energy Management Strategy for Fuel Cell Buses Quantifying Fuel Cell Degradation as Operating Cost," Sustainability, MDPI, vol. 14(23), pages 1-16, December.
    12. Elsisi, Mahmoud & Amer, Mohammed & Dababat, Alya’ & Su, Chun-Lien, 2023. "A comprehensive review of machine learning and IoT solutions for demand side energy management, conservation, and resilient operation," Energy, Elsevier, vol. 281(C).
    13. He, Hongwen & Su, Qicong & Huang, Ruchen & Niu, Zegong, 2024. "Enabling intelligent transferable energy management of series hybrid electric tracked vehicle across motion dimensions via soft actor-critic algorithm," Energy, Elsevier, vol. 294(C).
    14. Xiaodong Liu & Hongqiang Guo & Xingqun Cheng & Juan Du & Jian Ma, 2022. "A Robust Design of the Model-Free-Adaptive-Control-Based Energy Management for Plug-In Hybrid Electric Vehicle," Energies, MDPI, vol. 15(20), pages 1-24, October.
    15. Anselma, Pier Giuseppe & Belingardi, Giovanni, 2022. "Fuel cell electrified propulsion systems for long-haul heavy-duty trucks: present and future cost-oriented sizing," Applied Energy, Elsevier, vol. 321(C).
    16. Peng, Jiankun & Shen, Yang & Wu, ChangCheng & Wang, Chunhai & Yi, Fengyan & Ma, Chunye, 2023. "Research on energy-saving driving control of hydrogen fuel bus based on deep reinforcement learning in freeway ramp weaving area," Energy, Elsevier, vol. 285(C).
    17. Liu, Yonggang & Wu, Yitao & Wang, Xiangyu & Li, Liang & Zhang, Yuanjian & Chen, Zheng, 2023. "Energy management for hybrid electric vehicles based on imitation reinforcement learning," Energy, Elsevier, vol. 263(PC).
    18. Huang, Yin & Kang, Zehao & Mao, Xuping & Hu, Haoqin & Tan, Jiaqi & Xuan, Dongji, 2023. "Deep reinforcement learning based energymanagement strategy considering running costs and energy source aging for fuel cell hybrid electric vehicle," Energy, Elsevier, vol. 283(C).
    19. Xiaoyue Li & John M. Mulvey, 2023. "Optimal Portfolio Execution in a Regime-switching Market with Non-linear Impact Costs: Combining Dynamic Program and Neural Network," Papers 2306.08809, arXiv.org.
    20. Pregelj, Boštjan & Micor, Michał & Dolanc, Gregor & Petrovčič, Janko & Jovan, Vladimir, 2016. "Impact of fuel cell and battery size to overall system performance – A diesel fuel-cell APU case study," Applied Energy, Elsevier, vol. 182(C), pages 365-375.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:293:y:2024:i:c:s0360544224003360. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.