IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v263y2023ipbs036054422202638x.html
   My bibliography  Save this article

Reinforcement learning based power management integrating economic rotational speed of turboshaft engine and safety constraints of battery for hybrid electric power system

Author

Listed:
  • Wei, Zhengchao
  • Ma, Yue
  • Yang, Ningkang
  • Ruan, Shumin
  • Xiang, Changle

Abstract

Hybrid electric power system (HEPS) with turboshaft engine is a promising solution for the land and air vehicle, and the power management strategy (PMS) is the key to obtaining better performance of HEPS. In this paper, a reinforcement learning (RL)-based PMS integrating economic rotational speed (ERS) of turboshaft engine and safety constraints-based variable action space (SC-VAS) approach is proposed. First, an efficient algorithm based on the turbine performance map for calculating ERS is proposed, with low complexity which is 5.5% of the conventional algorithm. Second, based on the ERS feature, the SC-VAS approach is presented to further optimize the action space to prevent the discharging/charging power and state of charge of the battery from violating the safety constraints. Comparison results show that with no violation of constraints of battery, the convergence speed of RL agent incorporating the SC-VAS approach increases by 5 times, and the size of the optimized Q table decreases to 21.9% of that of the basic Q table. The proposed PMS with the ERS feature and SC-VAS approach can bring a 4.29% reduction in the fuel consumption under an air-land driving condition. Moreover, the results of the hardware-in-the-loop experiment demonstrate the real-time performance of the proposed strategy.

Suggested Citation

  • Wei, Zhengchao & Ma, Yue & Yang, Ningkang & Ruan, Shumin & Xiang, Changle, 2023. "Reinforcement learning based power management integrating economic rotational speed of turboshaft engine and safety constraints of battery for hybrid electric power system," Energy, Elsevier, vol. 263(PB).
    • Handle: RePEc:eee:energy:v:263:y:2023:i:pb:s036054422202638x
    DOI: 10.1016/j.energy.2022.125752
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S036054422202638X
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.energy.2022.125752?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Li, Xunming & Han, Lijin & Liu, Hui & Wang, Weida & Xiang, Changle, 2019. "Real-time optimal energy management strategy for a dual-mode power-split hybrid electric vehicle based on an explicit model predictive control algorithm," Energy, Elsevier, vol. 172(C), pages 1161-1178.
    2. Omar Mohamed & Ashraf Khalil, 2020. "Progress in Modeling and Control of Gas Turbine Power Generation Systems: A Survey," Energies, MDPI, vol. 13(9), pages 1-26, May.
    3. Ceusters, Glenn & Rodríguez, Román Cantú & García, Alberte Bouso & Franke, Rüdiger & Deconinck, Geert & Helsen, Lieve & Nowé, Ann & Messagie, Maarten & Camargo, Luis Ramirez, 2021. "Model-predictive control and reinforcement learning in multi-energy system case studies," Applied Energy, Elsevier, vol. 303(C).
    4. Bao, Shuyue & Sun, Ping & Zhu, Jianxin & Ji, Qian & Liu, Junheng, 2022. "Improved multi-dimensional dynamic programming energy management strategy for a vehicle power-split hybrid powertrain," Energy, Elsevier, vol. 256(C).
    5. Lian, Renzong & Peng, Jiankun & Wu, Yuankai & Tan, Huachun & Zhang, Hailong, 2020. "Rule-interposing deep reinforcement learning based energy management strategy for power-split hybrid electric vehicle," Energy, Elsevier, vol. 197(C).
    6. Song, Ke & Wang, Xiaodi & Li, Feiqiang & Sorrentino, Marco & Zheng, Bailin, 2020. "Pontryagin’s minimum principle-based real-time energy management strategy for fuel cell hybrid electric vehicle considering both fuel economy and power source durability," Energy, Elsevier, vol. 205(C).
    7. Qi, Chunyang & Zhu, Yiwen & Song, Chuanxue & Yan, Guangfu & Xiao, Feng & Da wang, & Zhang, Xu & Cao, Jingwei & Song, Shixin, 2022. "Hierarchical reinforcement learning based energy management strategy for hybrid electric vehicle," Energy, Elsevier, vol. 238(PA).
    8. Collins, Jeffrey M. & McLarty, Dustin, 2020. "All-electric commercial aviation with solid oxide fuel cell-gas turbine-battery hybrids," Applied Energy, Elsevier, vol. 265(C).
    9. Yang, Ningkang & Han, Lijin & Xiang, Changle & Liu, Hui & Li, Xunmin, 2021. "An indirect reinforcement learning based real-time energy management strategy via high-order Markov Chain model for a hybrid electric vehicle," Energy, Elsevier, vol. 236(C).
    10. Tian, Xiang & Cai, Yingfeng & Sun, Xiaodong & Zhu, Zhen & Xu, Yiqiang, 2019. "An adaptive ECMS with driving style recognition for energy optimization of parallel hybrid electric buses," Energy, Elsevier, vol. 189(C).
    11. Zhengchao Wei & Yue Ma & Changle Xiang & Dabo Liu & Weixiang Zhou, 2021. "Power Prediction-Based Model Predictive Control for Energy Management in Land and Air Vehicle with Turboshaft Engine," Complexity, Hindawi, vol. 2021, pages 1-24, August.
    12. Xiong, Rui & Cao, Jiayi & Yu, Quanqing, 2018. "Reinforcement learning-based real-time power management for hybrid energy storage system in the plug-in hybrid electric vehicle," Applied Energy, Elsevier, vol. 211(C), pages 538-548.
    13. Maino, Claudio & Misul, Daniela & Musa, Alessia & Spessa, Ezio, 2021. "Optimal mesh discretization of the dynamic programming for hybrid electric vehicles," Applied Energy, Elsevier, vol. 292(C).
    14. Xiang, Changle & Ding, Feng & Wang, Weida & He, Wei, 2017. "Energy management of a dual-mode power-split hybrid electric vehicle based on velocity prediction and nonlinear model predictive control," Applied Energy, Elsevier, vol. 189(C), pages 640-653.
    15. Ezzat, M.F. & Dincer, I., 2019. "Development and exergetic assessment of a new hybrid vehicle incorporating gas turbine as powering option," Energy, Elsevier, vol. 170(C), pages 112-119.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dong, Peng & Zhao, Junwei & Liu, Xuewu & Wu, Jian & Xu, Xiangyang & Liu, Yanfang & Wang, Shuhan & Guo, Wei, 2022. "Practical application of energy management strategy for hybrid electric vehicles based on intelligent and connected technologies: Development stages, challenges, and future trends," Renewable and Sustainable Energy Reviews, Elsevier, vol. 170(C).
    2. Zhu, Tao & Wills, Richard G.A. & Lot, Roberto & Ruan, Haijun & Jiang, Zhihao, 2021. "Adaptive energy management of a battery-supercapacitor energy storage system for electric vehicles based on flexible perception and neural network fitting," Applied Energy, Elsevier, vol. 292(C).
    3. Marouane Adnane & Ahmed Khoumsi & João Pedro F. Trovão, 2023. "Efficient Management of Energy Consumption of Electric Vehicles Using Machine Learning—A Systematic and Comprehensive Survey," Energies, MDPI, vol. 16(13), pages 1-39, June.
    4. Fengqi Zhang & Lihua Wang & Serdar Coskun & Hui Pang & Yahui Cui & Junqiang Xi, 2020. "Energy Management Strategies for Hybrid Electric Vehicles: Review, Classification, Comparison, and Outlook," Energies, MDPI, vol. 13(13), pages 1-35, June.
    5. Yang, Ningkang & Ruan, Shumin & Han, Lijin & Liu, Hui & Guo, Lingxiong & Xiang, Changle, 2023. "Reinforcement learning-based real-time intelligent energy management for hybrid electric vehicles in a model predictive control framework," Energy, Elsevier, vol. 270(C).
    6. Guo, Lingxiong & Liu, Hui & Han, Lijin & Yang, Ningkang & Liu, Rui & Xiang, Changle, 2023. "Predictive energy management strategy of dual-mode hybrid electric vehicles combining dynamic coordination control and simultaneous power distribution," Energy, Elsevier, vol. 263(PA).
    7. Nyong-Bassey, Bassey Etim & Giaouris, Damian & Patsios, Charalampos & Papadopoulou, Simira & Papadopoulos, Athanasios I. & Walker, Sara & Voutetakis, Spyros & Seferlis, Panos & Gadoue, Shady, 2020. "Reinforcement learning based adaptive power pinch analysis for energy management of stand-alone hybrid energy storage systems considering uncertainty," Energy, Elsevier, vol. 193(C).
    8. Daniel Egan & Qilun Zhu & Robert Prucka, 2023. "A Review of Reinforcement Learning-Based Powertrain Controllers: Effects of Agent Selection for Mixed-Continuity Control and Reward Formulation," Energies, MDPI, vol. 16(8), pages 1-31, April.
    9. Miranda, Matheus H.R. & Silva, Fabrício L. & Lourenço, Maria A.M. & Eckert, Jony J. & Silva, Ludmila C.A., 2022. "Vehicle drivetrain and fuzzy controller optimization using a planar dynamics simulation based on a real-world driving cycle," Energy, Elsevier, vol. 257(C).
    10. Penghui Qiang & Peng Wu & Tao Pan & Huaiquan Zang, 2021. "Real-Time Approximate Equivalent Consumption Minimization Strategy Based on the Single-Shaft Parallel Hybrid Powertrain," Energies, MDPI, vol. 14(23), pages 1-22, November.
    11. Chen, Z. & Liu, Y. & Ye, M. & Zhang, Y. & Chen, Z. & Li, G., 2021. "A survey on key techniques and development perspectives of equivalent consumption minimisation strategy for hybrid electric vehicles," Renewable and Sustainable Energy Reviews, Elsevier, vol. 151(C).
    12. Hu, Dong & Huang, Chao & Yin, Guodong & Li, Yangmin & Huang, Yue & Huang, Hailong & Wu, Jingda & Li, Wenfei & Xie, Hui, 2024. "A transfer-based reinforcement learning collaborative energy management strategy for extended-range electric buses with cabin temperature comfort consideration," Energy, Elsevier, vol. 290(C).
    13. Shi, Wenzhuo & Huangfu, Yigeng & Xu, Liangcai & Pang, Shengzhao, 2022. "Online energy management strategy considering fuel cell fault for multi-stack fuel cell hybrid vehicle based on multi-agent reinforcement learning," Applied Energy, Elsevier, vol. 328(C).
    14. Huang, Ruchen & He, Hongwen & Zhao, Xuyang & Wang, Yunlong & Li, Menglin, 2022. "Battery health-aware and naturalistic data-driven energy management for hybrid electric bus based on TD3 deep reinforcement learning algorithm," Applied Energy, Elsevier, vol. 321(C).
    15. Zhou, Jianhao & Xue, Yuan & Xu, Da & Li, Chaoxiong & Zhao, Wanzhong, 2022. "Self-learning energy management strategy for hybrid electric vehicle via curiosity-inspired asynchronous deep reinforcement learning," Energy, Elsevier, vol. 242(C).
    16. Ma, Shuai & Lin, Meng & Lin, Tzu-En & Lan, Tian & Liao, Xun & Maréchal, François & Van herle, Jan & Yang, Yongping & Dong, Changqing & Wang, Ligang, 2021. "Fuel cell-battery hybrid systems for mobility and off-grid applications: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 135(C).
    17. Bo, Lin & Han, Lijin & Xiang, Changle & Liu, Hui & Ma, Tian, 2022. "A Q-learning fuzzy inference system based online energy management strategy for off-road hybrid electric vehicles," Energy, Elsevier, vol. 252(C).
    18. Xiao, B. & Ruan, J. & Yang, W. & Walker, P.D. & Zhang, N., 2021. "A review of pivotal energy management strategies for extended range electric vehicles," Renewable and Sustainable Energy Reviews, Elsevier, vol. 149(C).
    19. Fabrizio Donatantonio & Alessandro Ferrara & Pierpaolo Polverino & Ivan Arsie & Cesare Pianese, 2022. "Novel Approaches for Energy Management Strategies of Hybrid Electric Vehicles and Comparison with Conventional Solutions," Energies, MDPI, vol. 15(6), pages 1-22, March.
    20. Guo, Xiaokai & Yan, Xianguo & Chen, Zhi & Meng, Zhiyu, 2022. "Research on energy management strategy of heavy-duty fuel cell hybrid vehicles based on dueling-double-deep Q-network," Energy, Elsevier, vol. 260(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:263:y:2023:i:pb:s036054422202638x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.