IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v332y2023ics0306261922018207.html
   My bibliography  Save this article

Cooperative energy management and eco-driving of plug-in hybrid electric vehicle via multi-agent reinforcement learning

Author

Listed:
  • Wang, Yong
  • Wu, Yuankai
  • Tang, Yingjuan
  • Li, Qin
  • He, Hongwen

Abstract

The advanced cruise control system has expanded the energy-saving potential of the hybrid electric vehicle (HEV). Despite this, most energy-saving researches for HEV either only optimize the energy management strategy (EMS) or integrate eco-driving through a hierarchically optimized assumption that optimizes EMS and eco-driving separately. Such kinds of approaches may lead to sub-optimal results. To fill this gap, we design a multi-agent reinforcement learning (MARL) based optimal energy-saving strategy for HEV, achieving a cooperative control on the powertrain and car-following behaviors to minimize the energy consumption and keep a safe following distance simultaneously. Specifically, a plug-in HEV model is regarded as the research object in this paper. Firstly, the HEV energy management problem in the car-following scenario is decomposed into a multi-agent cooperative task into two subtasks, each of which can conduct interactive learning through cooperative optimization. Secondly, the energy-saving strategy is designed, called the independent soft actor–critic, which consists of a car-following agent and an energy management agent. Finally, the performance of velocity tracking and energy-saving are validated under different driving cycles. In comparison to the state-of-the-art hierarchical model predictive control (MPC) strategy, the proposed MARL method can reduce fuel consumption by 15.8% while ensuring safety and comfort.

Suggested Citation

  • Wang, Yong & Wu, Yuankai & Tang, Yingjuan & Li, Qin & He, Hongwen, 2023. "Cooperative energy management and eco-driving of plug-in hybrid electric vehicle via multi-agent reinforcement learning," Applied Energy, Elsevier, vol. 332(C).
    • Handle: RePEc:eee:appene:v:332:y:2023:i:c:s0306261922018207
    DOI: 10.1016/j.apenergy.2022.120563
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261922018207
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2022.120563?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Zhou, Yang & Ahn, Soyoung, 2019. "Robust local and string stability for a decentralized car following control strategy for connected automated vehicles," Transportation Research Part B: Methodological, Elsevier, vol. 125(C), pages 175-196.
    2. Tian, He & Li, Shengbo Eben & Wang, Xu & Huang, Yong & Tian, Guangyu, 2018. "Data-driven hierarchical control for online energy management of plug-in hybrid electric city bus," Energy, Elsevier, vol. 142(C), pages 55-67.
    3. Bellocchi, Sara & Klöckner, Kai & Manno, Michele & Noussan, Michel & Vellini, Michela, 2019. "On the role of electric vehicles towards low-carbon energy systems: Italy and Germany in comparison," Applied Energy, Elsevier, vol. 255(C).
    4. Ibrahim, Amier & Jiang, Fangming, 2021. "The electric vehicle energy management: An overview of the energy system and related modeling and simulation," Renewable and Sustainable Energy Reviews, Elsevier, vol. 144(C).
    5. Quan, Shengwei & Wang, Ya-Xiong & Xiao, Xuelian & He, Hongwen & Sun, Fengchun, 2021. "Real-time energy management for fuel cell electric vehicle using speed prediction-based model predictive control considering performance degradation," Applied Energy, Elsevier, vol. 304(C).
    6. Chen, Z. & Liu, Y. & Ye, M. & Zhang, Y. & Chen, Z. & Li, G., 2021. "A survey on key techniques and development perspectives of equivalent consumption minimisation strategy for hybrid electric vehicles," Renewable and Sustainable Energy Reviews, Elsevier, vol. 151(C).
    7. Zhang, Jian & Tang, Tie-Qiao & Yan, Yadan & Qu, Xiaobo, 2021. "Eco-driving control for connected and automated electric vehicles at signalized intersections with wireless charging," Applied Energy, Elsevier, vol. 282(PA).
    8. Gao, Zhiming & LaClair, Tim & Ou, Shiqi & Huff, Shean & Wu, Guoyuan & Hao, Peng & Boriboonsomsin, Kanok & Barth, Matthew, 2019. "Evaluation of electric vehicle component performance over eco-driving cycles," Energy, Elsevier, vol. 172(C), pages 823-839.
    9. Oriol Vinyals & Igor Babuschkin & Wojciech M. Czarnecki & Michaël Mathieu & Andrew Dudzik & Junyoung Chung & David H. Choi & Richard Powell & Timo Ewalds & Petko Georgiev & Junhyuk Oh & Dan Horgan & M, 2019. "Grandmaster level in StarCraft II using multi-agent reinforcement learning," Nature, Nature, vol. 575(7782), pages 350-354, November.
    10. Ruan, Shumin & Ma, Yue & Yang, Ningkang & Xiang, Changle & Li, Xunming, 2022. "Real-time energy-saving control for HEVs in car-following scenario with a double explicit MPC approach," Energy, Elsevier, vol. 247(C).
    11. Qu, Xiaobo & Yu, Yang & Zhou, Mofan & Lin, Chin-Teng & Wang, Xiangyu, 2020. "Jointly dampening traffic oscillations and improving energy consumption with electric, connected and automated vehicles: A reinforcement learning based approach," Applied Energy, Elsevier, vol. 257(C).
    12. Hongwen, He & Jinquan, Guo & Jiankun, Peng & Huachun, Tan & Chao, Sun, 2018. "Real-time global driving cycle construction and the application to economy driving pro system in plug-in hybrid electric vehicles," Energy, Elsevier, vol. 152(C), pages 95-107.
    13. Lee, Heeyun & Kim, Kyunghyun & Kim, Namwook & Cha, Suk Won, 2022. "Energy efficient speed planning of electric vehicles for car-following scenario using model-based reinforcement learning," Applied Energy, Elsevier, vol. 313(C).
    14. Peter R. Wurman & Samuel Barrett & Kenta Kawamoto & James MacGlashan & Kaushik Subramanian & Thomas J. Walsh & Roberto Capobianco & Alisa Devlic & Franziska Eckert & Florian Fuchs & Leilani Gilpin & P, 2022. "Outracing champion Gran Turismo drivers with deep reinforcement learning," Nature, Nature, vol. 602(7896), pages 223-228, February.
    15. Li, Yuecheng & He, Hongwen & Khajepour, Amir & Wang, Hong & Peng, Jiankun, 2019. "Energy management for a power-split hybrid electric bus via deep reinforcement learning with terrain information," Applied Energy, Elsevier, vol. 255(C).
    16. Huang, Yuhan & Ng, Elvin C.Y. & Zhou, John L. & Surawski, Nic C. & Chan, Edward F.C. & Hong, Guang, 2018. "Eco-driving technology for sustainable road transport: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 93(C), pages 596-609.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jiankai Gao & Yang Li & Bin Wang & Haibo Wu, 2023. "Multi-Microgrid Collaborative Optimization Scheduling Using an Improved Multi-Agent Soft Actor-Critic Algorithm," Energies, MDPI, vol. 16(7), pages 1-21, April.
    2. Hua, Min & Zhang, Cetengfei & Zhang, Fanggang & Li, Zhi & Yu, Xiaoli & Xu, Hongming & Zhou, Quan, 2023. "Energy management of multi-mode plug-in hybrid electric vehicle using multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 348(C).
    3. Li, Jie & Fotouhi, Abbas & Pan, Wenjun & Liu, Yonggang & Zhang, Yuanjian & Chen, Zheng, 2023. "Deep reinforcement learning-based eco-driving control for connected electric vehicles at signalized intersections considering traffic uncertainties," Energy, Elsevier, vol. 279(C).
    4. Tian, Weiyong & Liu, Li & Zhang, Xiaohui & Shao, Jiaqi, 2024. "Flight trajectory and energy management coupled optimization for hybrid electric UAVs with adaptive sequential convex programming method," Applied Energy, Elsevier, vol. 364(C).
    5. Hou, Zhuoran & Guo, Jianhua & Chu, Liang & Hu, Jincheng & Chen, Zheng & Zhang, Yuanjian, 2023. "Exploration the route of information integration for vehicle design: A knowledge-enhanced energy management strategy," Energy, Elsevier, vol. 282(C).
    6. Chen, Bin & Wang, Miaoben & Hu, Lin & He, Guo & Yan, Haoyang & Wen, Xinji & Du, Ronghua, 2024. "Data-driven Koopman model predictive control for hybrid energy storage system of electric vehicles under vehicle-following scenarios," Applied Energy, Elsevier, vol. 365(C).
    7. Li, Cheng & Xu, Xiangyang & Zhu, Helong & Gan, Jiongpeng & Chen, Zhige & Tang, Xiaolin, 2024. "Research on car-following control and energy management strategy of hybrid electric vehicles in connected scene," Energy, Elsevier, vol. 293(C).
    8. Obeida Farhat & Mahmoud Khaled & Jalal Faraj & Farouk Hachem & Cathy Castelain, 2024. "Multiple Heat Recovery System for an Industrial Thermal Peeling Press Machine—Experimental Study with Energy and Economic Analyses," Energies, MDPI, vol. 17(6), pages 1-30, March.
    9. Li, Jie & Fotouhi, Abbas & Liu, Yonggang & Zhang, Yuanjian & Chen, Zheng, 2024. "Review on eco-driving control for connected and automated vehicles," Renewable and Sustainable Energy Reviews, Elsevier, vol. 189(PB).
    10. Ma, Yan & Ma, Qian & Liu, Yongqin & Gao, Jinwu & Chen, Hong, 2024. "Two-level optimization strategy for vehicle speed and battery thermal management in connected and automated EVs," Applied Energy, Elsevier, vol. 361(C).
    11. Keerthana Sivamayil & Elakkiya Rajasekar & Belqasem Aljafari & Srete Nikolovski & Subramaniyaswamy Vairavasundaram & Indragandhi Vairavasundaram, 2023. "A Systematic Study on Reinforcement Learning Based Applications," Energies, MDPI, vol. 16(3), pages 1-23, February.
    12. Xu Wang & Ying Huang & Jian Wang, 2023. "Study on Driver-Oriented Energy Management Strategy for Hybrid Heavy-Duty Off-Road Vehicles under Aggressive Transient Operating Condition," Sustainability, MDPI, vol. 15(9), pages 1-25, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Huang, Ruchen & He, Hongwen & Gao, Miaojue, 2023. "Training-efficient and cost-optimal energy management for fuel cell hybrid electric bus based on a novel distributed deep reinforcement learning framework," Applied Energy, Elsevier, vol. 346(C).
    2. Zhang, Hao & Chen, Boli & Lei, Nuo & Li, Bingbing & Chen, Chaoyi & Wang, Zhi, 2024. "Coupled velocity and energy management optimization of connected hybrid electric vehicles for maximum collective efficiency," Applied Energy, Elsevier, vol. 360(C).
    3. Hou, Shengyan & Yin, Hai & Xu, Fuguo & Benjamín, Pla & Gao, Jinwu & Chen, Hong, 2023. "Multihorizon predictive energy optimization and lifetime management for connected fuel cell electric vehicles," Energy, Elsevier, vol. 266(C).
    4. Dong, Haoxuan & Zhuang, Weichao & Chen, Boli & Wang, Yan & Lu, Yanbo & Liu, Ying & Xu, Liwei & Yin, Guodong, 2022. "A comparative study of energy-efficient driving strategy for connected internal combustion engine and electric vehicles at signalized intersections," Applied Energy, Elsevier, vol. 310(C).
    5. Jinming Xu & Yuan Lin, 2024. "Energy Management for Hybrid Electric Vehicles Using Safe Hybrid-Action Reinforcement Learning," Mathematics, MDPI, vol. 12(5), pages 1-20, February.
    6. Li, Jie & Fotouhi, Abbas & Liu, Yonggang & Zhang, Yuanjian & Chen, Zheng, 2024. "Review on eco-driving control for connected and automated vehicles," Renewable and Sustainable Energy Reviews, Elsevier, vol. 189(PB).
    7. Daniel Egan & Qilun Zhu & Robert Prucka, 2023. "A Review of Reinforcement Learning-Based Powertrain Controllers: Effects of Agent Selection for Mixed-Continuity Control and Reward Formulation," Energies, MDPI, vol. 16(8), pages 1-31, April.
    8. Nweye, Kingsley & Sankaranarayanan, Siva & Nagy, Zoltan, 2023. "MERLIN: Multi-agent offline and transfer learning for occupant-centric operation of grid-interactive communities," Applied Energy, Elsevier, vol. 346(C).
    9. Guo, Ningyuan & Zhang, Wencan & Li, Junqiu & Chen, Zheng & Li, Jianwei & Sun, Chao, 2024. "Predictive energy management of fuel cell plug-in hybrid electric vehicles: A co-state boundaries-oriented PMP optimization approach," Applied Energy, Elsevier, vol. 362(C).
    10. Zhang, Hanyu & Du, Lili, 2023. "Platoon-centered control for eco-driving at signalized intersection built upon hybrid MPC system, online learning and distributed optimization part I: Modeling and solution algorithm design," Transportation Research Part B: Methodological, Elsevier, vol. 172(C), pages 174-198.
    11. Ahmed, Abdelsalam A. & Ramadan, Haitham S., 2020. "Prototype implementation of advanced electric vehicles drivetrain system: Verification and validation," Applied Energy, Elsevier, vol. 266(C).
    12. Amjad, Muhammad & Farooq-i-Azam, Muhammad & Ni, Qiang & Dong, Mianxiong & Ansari, Ejaz Ahmad, 2022. "Wireless charging systems for electric vehicles," Renewable and Sustainable Energy Reviews, Elsevier, vol. 167(C).
    13. Liu, Qingling & Xu, Xiaowen, 2024. "A platoon-based eco-driving control mechanism for low-density traffic flow," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 637(C).
    14. Zhang, Yuanjian & Liu, Yonggang & Huang, Yanjun & Chen, Zheng & Li, Guang & Hao, Wanming & Cunningham, Geoff & Early, Juliana, 2021. "An optimal control strategy design for plug-in hybrid electric vehicles based on internet of vehicles," Energy, Elsevier, vol. 228(C).
    15. Sun, Xilei & Fu, Jianqin & Yang, Huiyong & Xie, Mingke & Liu, Jingping, 2023. "An energy management strategy for plug-in hybrid electric vehicles based on deep learning and improved model predictive control," Energy, Elsevier, vol. 269(C).
    16. Zhou, Yang & Zhong, Xinzhi & Chen, Qian & Ahn, Soyoung & Jiang, Jiwan & Jafarsalehi, Ghazaleh, 2023. "Data-driven analysis for disturbance amplification in car-following behavior of automated vehicles," Transportation Research Part B: Methodological, Elsevier, vol. 174(C).
    17. Tao, Fazhan & Fu, Zhigao & Gong, Huixian & Ji, Baofeng & Zhou, Yao, 2023. "Twin delayed deep deterministic policy gradient based energy management strategy for fuel cell/battery/ultracapacitor hybrid electric vehicles considering predicted terrain information," Energy, Elsevier, vol. 283(C).
    18. He, Hongwen & Meng, Xiangfei & Wang, Yong & Khajepour, Amir & An, Xiaowen & Wang, Renguang & Sun, Fengchun, 2024. "Deep reinforcement learning based energy management strategies for electrified vehicles: Recent advances and perspectives," Renewable and Sustainable Energy Reviews, Elsevier, vol. 192(C).
    19. Zhang, Yahui & Wang, Zimeng & Tian, Yang & Wang, Zhong & Kang, Mingxin & Xie, Fangxi & Wen, Guilin, 2024. "Pre-optimization-assisted deep reinforcement learning-based energy management strategy for a series–parallel hybrid electric truck," Energy, Elsevier, vol. 302(C).
    20. Huang, Ruchen & He, Hongwen & Su, Qicong, 2024. "Towards a fossil-free urban transport system: An intelligent cross-type transferable energy management framework based on deep transfer reinforcement learning," Applied Energy, Elsevier, vol. 363(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:332:y:2023:i:c:s0306261922018207. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.