IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v304y2024ics0360544224015421.html
   My bibliography  Save this article

Deep reinforcement learning-based health-conscious energy management for fuel cell hybrid electric vehicles in model predictive control framework

Author

Listed:
  • Huang, Xuejin
  • Zhang, Jingyi
  • Ou, Kai
  • Huang, Yin
  • Kang, Zehao
  • Mao, Xuping
  • Zhou, Yujie
  • Xuan, Dongji

Abstract

The main contribution of this study is to introduce deep reinforcement learning (DRL) within the model prediction control (MPC) framework, and consider comprehensive economic objectives including fuel cell degradation costs, lithium battery aging costs, hydrogen consumption costs, etc. This approach successfully mitigated the inherent shortcomings of deep reinforcement learning, namely poor generalization and lack of adaptability, thereby significantly enhancing the robustness of economic driving decision in unknown scenarios. In this study, an MPC framework was developed for the energy management problem of fuel cell vehicles, and Bi-directional Long Short-Term Memory (Bi-LSTM) neural network was used to construct a vehicle speed predictor The accuracy of its prediction was verified through comparative analysis, and then it was regarded as a DRL model. Different from the overall strategy of the entire driving cycle, the model based DRL agent can learn the optimal action for each vehicle state. The simulation evaluated the impact of different predictors and prediction ranges on hydrogen economy, and verified the adaptability of the proposed strategy in different driving environments, the stability of battery state maintenance, and the advantages of delaying energy system degradation through comprehensive comparative analysis.

Suggested Citation

  • Huang, Xuejin & Zhang, Jingyi & Ou, Kai & Huang, Yin & Kang, Zehao & Mao, Xuping & Zhou, Yujie & Xuan, Dongji, 2024. "Deep reinforcement learning-based health-conscious energy management for fuel cell hybrid electric vehicles in model predictive control framework," Energy, Elsevier, vol. 304(C).
  • Handle: RePEc:eee:energy:v:304:y:2024:i:c:s0360544224015421
    DOI: 10.1016/j.energy.2024.131769
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0360544224015421
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.energy.2024.131769?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Hussain, Syed Asad & Razi, Faran & Hewage, Kasun & Sadiq, Rehan, 2023. "The perspective of energy poverty and 1st energy crisis of green transition," Energy, Elsevier, vol. 275(C).
    2. Li, Jianwei & Yan, Chonghao & Yang, Qingqing & Hao, Dong & Zou, Weitao & Gao, Lei & Zhao, Xuan, 2023. "Quantitative diagnosis of PEMFC membrane humidity with a vector-distance based characteristic mapping approach," Applied Energy, Elsevier, vol. 335(C).
    3. Wu, Yuankai & Tan, Huachun & Peng, Jiankun & Zhang, Hailong & He, Hongwen, 2019. "Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus," Applied Energy, Elsevier, vol. 247(C), pages 454-466.
    4. Chen, Huicui & Pei, Pucheng & Song, Mancun, 2015. "Lifetime prediction and the economic lifetime of Proton Exchange Membrane fuel cells," Applied Energy, Elsevier, vol. 142(C), pages 154-163.
    5. Song, Zhen & Pan, Yue & Chen, Huicui & Zhang, Tong, 2021. "Effects of temperature on the performance of fuel cell hybrid electric vehicles: A review," Applied Energy, Elsevier, vol. 302(C).
    6. Huang, Yin & Kang, Zehao & Mao, Xuping & Hu, Haoqin & Tan, Jiaqi & Xuan, Dongji, 2023. "Deep reinforcement learning based energymanagement strategy considering running costs and energy source aging for fuel cell hybrid electric vehicle," Energy, Elsevier, vol. 283(C).
    7. Sun, Xilei & Fu, Jianqin & Yang, Huiyong & Xie, Mingke & Liu, Jingping, 2023. "An energy management strategy for plug-in hybrid electric vehicles based on deep learning and improved model predictive control," Energy, Elsevier, vol. 269(C).
    8. Peng, Jiankun & He, Hongwen & Xiong, Rui, 2017. "Rule based energy management strategy for a series–parallel plug-in hybrid electric bus optimized by dynamic programming," Applied Energy, Elsevier, vol. 185(P2), pages 1633-1643.
    9. Shi, Wenzhuo & Huangfu, Yigeng & Xu, Liangcai & Pang, Shengzhao, 2022. "Online energy management strategy considering fuel cell fault for multi-stack fuel cell hybrid vehicle based on multi-agent reinforcement learning," Applied Energy, Elsevier, vol. 328(C).
    10. Ximing Wang & Hongwen He & Fengchun Sun & Jieli Zhang, 2015. "Application Study on the Dynamic Programming Algorithm for Energy Management of Plug-in Hybrid Electric Vehicles," Energies, MDPI, vol. 8(4), pages 1-20, April.
    11. Yang, Ningkang & Ruan, Shumin & Han, Lijin & Liu, Hui & Guo, Lingxiong & Xiang, Changle, 2023. "Reinforcement learning-based real-time intelligent energy management for hybrid electric vehicles in a model predictive control framework," Energy, Elsevier, vol. 270(C).
    12. Xiang, Changle & Ding, Feng & Wang, Weida & He, Wei, 2017. "Energy management of a dual-mode power-split hybrid electric vehicle based on velocity prediction and nonlinear model predictive control," Applied Energy, Elsevier, vol. 189(C), pages 640-653.
    13. Zou, Weitao & Li, Jianwei & Yang, Qingqing & Wan, Xinming & He, Yuntang & Lan, Hao, 2023. "A real-time energy management approach with fuel cell and battery competition-synergy control for the fuel cell vehicle," Applied Energy, Elsevier, vol. 334(C).
    14. Huang, Ruchen & He, Hongwen & Zhao, Xuyang & Wang, Yunlong & Li, Menglin, 2022. "Battery health-aware and naturalistic data-driven energy management for hybrid electric bus based on TD3 deep reinforcement learning algorithm," Applied Energy, Elsevier, vol. 321(C).
    15. Jia, Chunchun & Li, Kunang & He, Hongwen & Zhou, Jiaming & Li, Jianwei & Wei, Zhongbao, 2023. "Health-aware energy management strategy for fuel cell hybrid bus considering air-conditioning control based on TD3 algorithm," Energy, Elsevier, vol. 283(C).
    16. Sun, Zhendong & Wang, Yujie & Chen, Zonghai & Li, Xiyun, 2020. "Min-max game based energy management strategy for fuel cell/supercapacitor hybrid electric vehicles," Applied Energy, Elsevier, vol. 267(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jia, Chunchun & Li, Kunang & He, Hongwen & Zhou, Jiaming & Li, Jianwei & Wei, Zhongbao, 2023. "Health-aware energy management strategy for fuel cell hybrid bus considering air-conditioning control based on TD3 algorithm," Energy, Elsevier, vol. 283(C).
    2. Jia, Chunchun & Zhou, Jiaming & He, Hongwen & Li, Jianwei & Wei, Zhongbao & Li, Kunang & Shi, Man, 2023. "A novel energy management strategy for hybrid electric bus with fuel cell health and battery thermal- and health-constrained awareness," Energy, Elsevier, vol. 271(C).
    3. Huang, Yin & Kang, Zehao & Mao, Xuping & Hu, Haoqin & Tan, Jiaqi & Xuan, Dongji, 2023. "Deep reinforcement learning based energymanagement strategy considering running costs and energy source aging for fuel cell hybrid electric vehicle," Energy, Elsevier, vol. 283(C).
    4. Du, Guodong & Zou, Yuan & Zhang, Xudong & Kong, Zehui & Wu, Jinlong & He, Dingbo, 2019. "Intelligent energy management for hybrid electric tracked vehicles using online reinforcement learning," Applied Energy, Elsevier, vol. 251(C), pages 1-1.
    5. Jia, Chunchun & He, Hongwen & Zhou, Jiaming & Li, Jianwei & Wei, Zhongbao & Li, Kunang, 2023. "A novel health-aware deep reinforcement learning energy management for fuel cell bus incorporating offline high-quality experience," Energy, Elsevier, vol. 282(C).
    6. Zou, Weitao & Li, Jianwei & Yang, Qingqing & Wan, Xinming & He, Yuntang & Lan, Hao, 2023. "A real-time energy management approach with fuel cell and battery competition-synergy control for the fuel cell vehicle," Applied Energy, Elsevier, vol. 334(C).
    7. Dong, Peng & Zhao, Junwei & Liu, Xuewu & Wu, Jian & Xu, Xiangyang & Liu, Yanfang & Wang, Shuhan & Guo, Wei, 2022. "Practical application of energy management strategy for hybrid electric vehicles based on intelligent and connected technologies: Development stages, challenges, and future trends," Renewable and Sustainable Energy Reviews, Elsevier, vol. 170(C).
    8. Tan, Yingqi & Xu, Jingyi & Ma, Junyi & Li, Zirui & Chen, Huiyan & Xi, Junqiang & Liu, Haiou, 2024. "A transferable perception-guided EMS for series hybrid electric unmanned tracked vehicles," Energy, Elsevier, vol. 306(C).
    9. Kong, Yan & Xu, Nan & Zhang, Yuanjian & Sui, Yan & Ju, Hao & Liu, Heng & Xu, Zhe, 2021. "Acquisition of full-factor trip information for global optimization energy management in multi-energy source vehicles and the measure of the amount of information to be transmitted," Energy, Elsevier, vol. 236(C).
    10. Niu, Zegong & He, Hongwen, 2024. "A data-driven solution for intelligent power allocation of connected hybrid electric vehicles inspired by offline deep reinforcement learning in V2X scenario," Applied Energy, Elsevier, vol. 372(C).
    11. Geng, Wenran & Lou, Diming & Wang, Chen & Zhang, Tong, 2020. "A cascaded energy management optimization method of multimode power-split hybrid electric vehicles," Energy, Elsevier, vol. 199(C).
    12. He, Hongwen & Meng, Xiangfei & Wang, Yong & Khajepour, Amir & An, Xiaowen & Wang, Renguang & Sun, Fengchun, 2024. "Deep reinforcement learning based energy management strategies for electrified vehicles: Recent advances and perspectives," Renewable and Sustainable Energy Reviews, Elsevier, vol. 192(C).
    13. Wang, Hao & He, Hongwen & Bai, Yunfei & Yue, Hongwei, 2022. "Parameterized deep Q-network based energy management with balanced energy economy and battery life for hybrid electric vehicles," Applied Energy, Elsevier, vol. 320(C).
    14. Zhu, Tao & Wills, Richard G.A. & Lot, Roberto & Ruan, Haijun & Jiang, Zhihao, 2021. "Adaptive energy management of a battery-supercapacitor energy storage system for electric vehicles based on flexible perception and neural network fitting," Applied Energy, Elsevier, vol. 292(C).
    15. Du, Guodong & Zou, Yuan & Zhang, Xudong & Liu, Teng & Wu, Jinlong & He, Dingbo, 2020. "Deep reinforcement learning based energy management for a hybrid electric vehicle," Energy, Elsevier, vol. 201(C).
    16. Lian, Renzong & Peng, Jiankun & Wu, Yuankai & Tan, Huachun & Zhang, Hailong, 2020. "Rule-interposing deep reinforcement learning based energy management strategy for power-split hybrid electric vehicle," Energy, Elsevier, vol. 197(C).
    17. Chen, Z. & Liu, Y. & Ye, M. & Zhang, Y. & Chen, Z. & Li, G., 2021. "A survey on key techniques and development perspectives of equivalent consumption minimisation strategy for hybrid electric vehicles," Renewable and Sustainable Energy Reviews, Elsevier, vol. 151(C).
    18. Chen, Jiaxin & Shu, Hong & Tang, Xiaolin & Liu, Teng & Wang, Weida, 2022. "Deep reinforcement learning-based multi-objective control of hybrid power system combined with road recognition under time-varying environment," Energy, Elsevier, vol. 239(PC).
    19. Fengqi Zhang & Lihua Wang & Serdar Coskun & Hui Pang & Yahui Cui & Junqiang Xi, 2020. "Energy Management Strategies for Hybrid Electric Vehicles: Review, Classification, Comparison, and Outlook," Energies, MDPI, vol. 13(13), pages 1-35, June.
    20. Shi, Wenzhuo & Huangfu, Yigeng & Xu, Liangcai & Pang, Shengzhao, 2022. "Online energy management strategy considering fuel cell fault for multi-stack fuel cell hybrid vehicle based on multi-agent reinforcement learning," Applied Energy, Elsevier, vol. 328(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:304:y:2024:i:c:s0360544224015421. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.