IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v15y2022i17p6323-d901670.html
   My bibliography  Save this article

Hybrid Deep Reinforcement Learning Considering Discrete-Continuous Action Spaces for Real-Time Energy Management in More Electric Aircraft

Author

Listed:
  • Bing Liu

    (College of Information Engineering, Zhejiang University of Technology, Hangzhou 310023, China)

  • Bowen Xu

    (College of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China)

  • Tong He

    (College of Information Engineering, Zhejiang University of Technology, Hangzhou 310023, China)

  • Wei Yu

    (Green Rooftop Inc., Hangzhou 310032, China)

  • Fanghong Guo

    (College of Information Engineering, Zhejiang University of Technology, Hangzhou 310023, China)

Abstract

The increasing number and functional complexity of power electronics in more electric aircraft (MEA) power systems have led to a high degree of complexity in modelling and computation, making real-time energy management a formidable challenge, and the discrete-continuous action space of the MEA system under consideration also poses a challenge to existing DRL algorithms. Therefore, this paper proposes an optimisation strategy for real-time energy management based on hybrid deep reinforcement learning (HDRL). An energy management model of the MEA power system is constructed for the analysis of generators, buses, loads and energy storage system (ESS) characteristics, and the problem is described as a multi-objective optimisation problem with integer and continuous variables. The problem is solved by combining a duelling double deep Q network (D3QN) algorithm with a deep deterministic policy gradient (DDPG) algorithm, where the D3QN algorithm deals with the discrete action space and the DDPG algorithm with the continuous action space. These two algorithms are alternately trained and interact with each other to maximize the long-term payoff of MEA. Finally, the simulation results show that the effectiveness of the method is verified under different generator operating conditions. For different time lengths T , the method always obtains smaller objective function values compared to previous DRL algorithms, is several orders of magnitude faster than commercial solvers, and is always less than 0.2 s, despite a slight shortfall in solution accuracy. In addition, the method has been validated on a hardware-in-the-loop simulation platform.

Suggested Citation

  • Bing Liu & Bowen Xu & Tong He & Wei Yu & Fanghong Guo, 2022. "Hybrid Deep Reinforcement Learning Considering Discrete-Continuous Action Spaces for Real-Time Energy Management in More Electric Aircraft," Energies, MDPI, vol. 15(17), pages 1-21, August.
    • Handle: RePEc:gam:jeners:v:15:y:2022:i:17:p:6323-:d:901670
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/15/17/6323/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1996-1073/15/17/6323/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Lian, Renzong & Peng, Jiankun & Wu, Yuankai & Tan, Huachun & Zhang, Hailong, 2020. "Rule-interposing deep reinforcement learning based energy management strategy for power-split hybrid electric vehicle," Energy, Elsevier, vol. 197(C).
    2. Wu, Yuankai & Tan, Huachun & Peng, Jiankun & Zhang, Hailong & He, Hongwen, 2019. "Deep reinforcement learning of energy management with continuous control strategy and traffic information for a series-parallel plug-in hybrid electric bus," Applied Energy, Elsevier, vol. 247(C), pages 454-466.
    3. Du, Guodong & Zou, Yuan & Zhang, Xudong & Liu, Teng & Wu, Jinlong & He, Dingbo, 2020. "Deep reinforcement learning based energy management for a hybrid electric vehicle," Energy, Elsevier, vol. 201(C).
    4. Ying Ji & Jianhui Wang & Jiacan Xu & Xiaoke Fang & Huaguang Zhang, 2019. "Real-Time Energy Management of a Microgrid Using Deep Reinforcement Learning," Energies, MDPI, vol. 12(12), pages 1-21, June.
    5. Luqin Fan & Jing Zhang & Yu He & Ying Liu & Tao Hu & Heng Zhang, 2021. "Optimal Scheduling of Microgrid Based on Deep Deterministic Policy Gradient and Transfer Learning," Energies, MDPI, vol. 14(3), pages 1-15, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Yang, Dongpo & Liu, Tong & Song, Dafeng & Zhang, Xuanming & Zeng, Xiaohua, 2023. "A real time multi-objective optimization Guided-MPC strategy for power-split hybrid electric bus based on velocity prediction," Energy, Elsevier, vol. 276(C).
    2. Dong, Peng & Zhao, Junwei & Liu, Xuewu & Wu, Jian & Xu, Xiangyang & Liu, Yanfang & Wang, Shuhan & Guo, Wei, 2022. "Practical application of energy management strategy for hybrid electric vehicles based on intelligent and connected technologies: Development stages, challenges, and future trends," Renewable and Sustainable Energy Reviews, Elsevier, vol. 170(C).
    3. Qi, Chunyang & Song, Chuanxue & Xiao, Feng & Song, Shixin, 2022. "Generalization ability of hybrid electric vehicle energy management strategy based on reinforcement learning method," Energy, Elsevier, vol. 250(C).
    4. Liu, Teng & Tan, Wenhao & Tang, Xiaolin & Zhang, Jinwei & Xing, Yang & Cao, Dongpu, 2021. "Driving conditions-driven energy management strategies for hybrid electric vehicles: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 151(C).
    5. Zhu, Tao & Wills, Richard G.A. & Lot, Roberto & Ruan, Haijun & Jiang, Zhihao, 2021. "Adaptive energy management of a battery-supercapacitor energy storage system for electric vehicles based on flexible perception and neural network fitting," Applied Energy, Elsevier, vol. 292(C).
    6. Jiang, Yue & Meng, Hao & Chen, Guanpeng & Yang, Congnan & Xu, Xiaojun & Zhang, Lei & Xu, Haijun, 2022. "Differential-steering based path tracking control and energy-saving torque distribution strategy of 6WID unmanned ground vehicle," Energy, Elsevier, vol. 254(PA).
    7. Ju, Fei & Zhuang, Weichao & Wang, Liangmo & Zhang, Zhe, 2020. "Comparison of four-wheel-drive hybrid powertrain configurations," Energy, Elsevier, vol. 209(C).
    8. Daniel Egan & Qilun Zhu & Robert Prucka, 2023. "A Review of Reinforcement Learning-Based Powertrain Controllers: Effects of Agent Selection for Mixed-Continuity Control and Reward Formulation," Energies, MDPI, vol. 16(8), pages 1-31, April.
    9. Miranda, Matheus H.R. & Silva, Fabrício L. & Lourenço, Maria A.M. & Eckert, Jony J. & Silva, Ludmila C.A., 2022. "Vehicle drivetrain and fuzzy controller optimization using a planar dynamics simulation based on a real-world driving cycle," Energy, Elsevier, vol. 257(C).
    10. Penghui Qiang & Peng Wu & Tao Pan & Huaiquan Zang, 2021. "Real-Time Approximate Equivalent Consumption Minimization Strategy Based on the Single-Shaft Parallel Hybrid Powertrain," Energies, MDPI, vol. 14(23), pages 1-22, November.
    11. Wen, Lulu & Zhou, Kaile & Li, Jun & Wang, Shanyong, 2020. "Modified deep learning and reinforcement learning for an incentive-based demand response model," Energy, Elsevier, vol. 205(C).
    12. Chen, Jiaxin & Shu, Hong & Tang, Xiaolin & Liu, Teng & Wang, Weida, 2022. "Deep reinforcement learning-based multi-objective control of hybrid power system combined with road recognition under time-varying environment," Energy, Elsevier, vol. 239(PC).
    13. Qi, Chunyang & Zhu, Yiwen & Song, Chuanxue & Yan, Guangfu & Xiao, Feng & Da wang, & Zhang, Xu & Cao, Jingwei & Song, Shixin, 2022. "Hierarchical reinforcement learning based energy management strategy for hybrid electric vehicle," Energy, Elsevier, vol. 238(PA).
    14. Daeil Lee & Seoryong Koo & Inseok Jang & Jonghyun Kim, 2022. "Comparison of Deep Reinforcement Learning and PID Controllers for Automatic Cold Shutdown Operation," Energies, MDPI, vol. 15(8), pages 1-25, April.
    15. Shi, Wenzhuo & Huangfu, Yigeng & Xu, Liangcai & Pang, Shengzhao, 2022. "Online energy management strategy considering fuel cell fault for multi-stack fuel cell hybrid vehicle based on multi-agent reinforcement learning," Applied Energy, Elsevier, vol. 328(C).
    16. Robert Jane & Tae Young Kim & Samantha Rose & Emily Glass & Emilee Mossman & Corey James, 2022. "Developing AI/ML Based Predictive Capabilities for a Compression Ignition Engine Using Pseudo Dynamometer Data," Energies, MDPI, vol. 15(21), pages 1-49, October.
    17. Zhengyu Yao & Hwan-Sik Yoon & Yang-Ki Hong, 2023. "Control of Hybrid Electric Vehicle Powertrain Using Offline-Online Hybrid Reinforcement Learning," Energies, MDPI, vol. 16(2), pages 1-18, January.
    18. Wang, Yue & Li, Keqiang & Zeng, Xiaohua & Gao, Bolin & Hong, Jichao, 2023. "Investigation of novel intelligent energy management strategies for connected HEB considering global planning of fixed-route information," Energy, Elsevier, vol. 263(PB).
    19. Li, Shuangqi & He, Hongwen & Zhao, Pengfei, 2021. "Energy management for hybrid energy storage system in electric vehicle: A cyber-physical system perspective," Energy, Elsevier, vol. 230(C).
    20. Yao, Yongming & Wang, Jie & Zhou, Zhicong & Li, Hang & Liu, Huiying & Li, Tianyu, 2023. "Grey Markov prediction-based hierarchical model predictive control energy management for fuel cell/battery hybrid unmanned aerial vehicles," Energy, Elsevier, vol. 262(PA).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:15:y:2022:i:17:p:6323-:d:901670. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.