IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v324y2022ics0306261922010741.html
   My bibliography  Save this article

Hybrid multi-agent emotional deep Q network for generation control of multi-area integrated energy systems

Author

Listed:
  • Yin, Linfei
  • Li, Yu

Abstract

With the integration of renewable energy, pumped storage, and new energy storage into multi-area integrated energy systems, the generation control of multi-area integrated energy systems is facing serious challenges. A differential evolution variable parameter vector multi-agent emotional deep Q network is proposed to increase the frequency regulation accuracy and convergence speed of multi-area integrated energy systems. The proposed control framework enhances the performance of artificial emotion by differential evolution and adaptive to the environment; the learning rates and action values of two deep Q networks are emotionalized separately by adaptive artificial emotion based on differential evolution; the action values of two deep Q networks are employed to generate commands for smart generation control through vector control. The proposed control framework is calculated in two-area and four-area integrated energy systems with China Southern Power Grid as the background. The numerical calculation results verify the best control performance and fastest convergence speed of the proposed control framework. The frequency deviations of the two cases are reduced by at least 7.44 % and 8.37 %, respectively; the convergence speed of the control framework in the two cases is increased by at least 2.70 % and 0.84 %, respectively; the power generation costs of the two cases are reduced by at least 305,370.9 $ and 460,186.6 $, respectively; the carbon emissions of the two cases are reduced by at least 10,940 kg and 11,610 kg, respectively.

Suggested Citation

  • Yin, Linfei & Li, Yu, 2022. "Hybrid multi-agent emotional deep Q network for generation control of multi-area integrated energy systems," Applied Energy, Elsevier, vol. 324(C).
    • Handle: RePEc:eee:appene:v:324:y:2022:i:c:s0306261922010741
    DOI: 10.1016/j.apenergy.2022.119797
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261922010741
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2022.119797?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Yin, Linfei & Zhang, Bin, 2021. "Time series generative adversarial network controller for long-term smart generation control of microgrids," Applied Energy, Elsevier, vol. 281(C).
    2. Zhang, Xiaoshun & Chen, Yixuan & Yu, Tao & Yang, Bo & Qu, Kaiping & Mao, Senmao, 2017. "Equilibrium-inspired multiagent optimizer with extreme transfer learning for decentralized optimal carbon-energy combined-flow of large-scale power systems," Applied Energy, Elsevier, vol. 189(C), pages 157-176.
    3. Xi, Lei & Chen, Jianfeng & Huang, Yuehua & Xu, Yanchun & Liu, Lang & Zhou, Yimin & Li, Yudan, 2018. "Smart generation control based on multi-agent reinforcement learning with the idea of the time tunnel," Energy, Elsevier, vol. 153(C), pages 977-987.
    4. Zhao, Zhigao & Yang, Jiandong & Chung, C.Y. & Yang, Weijia & He, Xianghui & Chen, Man, 2021. "Performance enhancement of pumped storage units for system frequency support based on a novel small signal model," Energy, Elsevier, vol. 234(C).
    5. Yin, Linfei & Sun, Zhixiang, 2021. "Multi-layer distributed multi-objective consensus algorithm for multi-objective economic dispatch of large-scale multi-area interconnected power systems," Applied Energy, Elsevier, vol. 300(C).
    6. Li, Jiawen & Yu, Tao & Yang, Bo, 2021. "A data-driven output voltage control of solid oxide fuel cell using multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 304(C).
    7. Guo, Chenyu & Wang, Xin & Zheng, Yihui & Zhang, Feng, 2022. "Real-time optimal energy management of microgrid with uncertainties based on deep reinforcement learning," Energy, Elsevier, vol. 238(PC).
    8. Abu Hanifah, Rabiatuladawiyah & Toha, Siti Fauziah & Hassan, Mohd Khair & Ahmad, Salmiah, 2016. "Power reduction optimization with swarm based technique in electric power assist steering system," Energy, Elsevier, vol. 102(C), pages 444-452.
    9. Zhang, Xiaoshun & Tan, Tian & Yang, Bo & Wang, Jingbo & Li, Shengnan & He, Tingyi & Yang, Lei & Yu, Tao & Sun, Liming, 2020. "Greedy search based data-driven algorithm of centralized thermoelectric generation system under non-uniform temperature distribution," Applied Energy, Elsevier, vol. 260(C).
    10. Han, Ji & Miao, Shihong & Chen, Zhe & Liu, Zhou & Li, Yaowang & Yang, Weichen & Liu, Ziwen, 2021. "Multi-View clustering and discrete consensus based tri-level coordinated control of wind farm and adiabatic compressed air energy storage for providing frequency regulation service," Applied Energy, Elsevier, vol. 304(C).
    11. Zhang, Chongyu & Lu, Xi & Ren, Guo & Chen, Shi & Hu, Chengyu & Kong, Zhaoyang & Zhang, Ning & Foley, Aoife M., 2021. "Optimal allocation of onshore wind power in China based on cluster analysis," Applied Energy, Elsevier, vol. 285(C).
    12. Egido, I. & Sigrist, L. & Lobato, E. & Rouco, L. & Barrado, A., 2015. "An ultra-capacitor for frequency stability enhancement in small-isolated power systems: Models, simulation and field tests," Applied Energy, Elsevier, vol. 137(C), pages 670-676.
    13. Li, Jiawen & Yu, Tao & Zhang, Xiaoshun & Li, Fusheng & Lin, Dan & Zhu, Hanxin, 2021. "Efficient experience replay based deep deterministic policy gradient for AGC dispatch in integrated energy system," Applied Energy, Elsevier, vol. 285(C).
    14. Tang, Xiaolin & Zhou, Haitao & Wang, Feng & Wang, Weida & Lin, Xianke, 2022. "Longevity-conscious energy management strategy of fuel cell hybrid electric Vehicle Based on deep reinforcement learning," Energy, Elsevier, vol. 238(PA).
    15. Zhao, B.Y. & Zhao, Z.G. & Li, Y. & Wang, R.Z. & Taylor, R.A., 2019. "An adaptive PID control method to improve the power tracking performance of solar photovoltaic air-conditioning systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 113(C), pages 1-1.
    16. Latif, Abdul & Hussain, S.M. Suhail & Das, Dulal Chandra & Ustun, Taha Selim, 2020. "State-of-the-art of controllers and soft computing techniques for regulated load frequency management of single/multi-area traditional and renewable energy based power systems," Applied Energy, Elsevier, vol. 266(C).
    17. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    18. Yin, Linfei & Wu, Yunzhi, 2022. "Mode-decomposition memory reinforcement network strategy for smart generation control in multi-area power systems containing renewable energy," Applied Energy, Elsevier, vol. 307(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Li, Sichen & Hu, Weihao & Cao, Di & Chen, Zhe & Huang, Qi & Blaabjerg, Frede & Liao, Kaiji, 2023. "Physics-model-free heat-electricity energy management of multiple microgrids based on surrogate model-enabled multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 346(C).
    2. Fan, Wei & Fan, Ying & Yao, Xing & Yi, Bowen & Jiang, Dalin & Wu, Lin, 2024. "Distributed transaction optimization model of multi-integrated energy systems based on nash negotiation," Renewable Energy, Elsevier, vol. 225(C).
    3. Xue, Lin & Wang, Jianxue & Zhang, Yao & Yong, Weizhen & Qi, Jie & Li, Haotian, 2023. "Model-data-event based community integrated energy system low-carbon economic scheduling," Renewable and Sustainable Energy Reviews, Elsevier, vol. 182(C).
    4. Quan, Yue & Xi, Lei, 2024. "Smart generation system: A decentralized multi-agent control architecture based on improved consensus algorithm for generation command dispatch of sustainable energy systems," Applied Energy, Elsevier, vol. 365(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Li, Jiawen & Yu, Tao & Zhang, Xiaoshun & Li, Fusheng & Lin, Dan & Zhu, Hanxin, 2021. "Efficient experience replay based deep deterministic policy gradient for AGC dispatch in integrated energy system," Applied Energy, Elsevier, vol. 285(C).
    2. Yin, Linfei & Zhang, Bin, 2021. "Time series generative adversarial network controller for long-term smart generation control of microgrids," Applied Energy, Elsevier, vol. 281(C).
    3. Lu, Yu & Xiang, Yue & Huang, Yuan & Yu, Bin & Weng, Liguo & Liu, Junyong, 2023. "Deep reinforcement learning based optimal scheduling of active distribution system considering distributed generation, energy storage and flexible load," Energy, Elsevier, vol. 271(C).
    4. Wang, Yi & Qiu, Dawei & Sun, Mingyang & Strbac, Goran & Gao, Zhiwei, 2023. "Secure energy management of multi-energy microgrid: A physical-informed safe reinforcement learning approach," Applied Energy, Elsevier, vol. 335(C).
    5. Zhu, Ziqing & Hu, Ze & Chan, Ka Wing & Bu, Siqi & Zhou, Bin & Xia, Shiwei, 2023. "Reinforcement learning in deregulated energy market: A comprehensive review," Applied Energy, Elsevier, vol. 329(C).
    6. Pinciroli, Luca & Baraldi, Piero & Compare, Michele & Zio, Enrico, 2023. "Optimal operation and maintenance of energy storage systems in grid-connected microgrids by deep reinforcement learning," Applied Energy, Elsevier, vol. 352(C).
    7. Zhu, Dafeng & Yang, Bo & Liu, Yuxiang & Wang, Zhaojian & Ma, Kai & Guan, Xinping, 2022. "Energy management based on multi-agent deep reinforcement learning for a multi-energy industrial park," Applied Energy, Elsevier, vol. 311(C).
    8. Mazare, Mahmood, 2024. "Adaptive optimal secure wind power generation control for variable speed wind turbine systems via reinforcement learning," Applied Energy, Elsevier, vol. 353(PA).
    9. Yin, Linfei & Luo, Shikui & Ma, Chenxiao, 2021. "Expandable depth and width adaptive dynamic programming for economic smart generation control of smart grids," Energy, Elsevier, vol. 232(C).
    10. Han, Kunlun & Yang, Kai & Yin, Linfei, 2022. "Lightweight actor-critic generative adversarial networks for real-time smart generation control of microgrids," Applied Energy, Elsevier, vol. 317(C).
    11. Li, Jiawen & Yu, Tao & Yang, Bo, 2021. "A data-driven output voltage control of solid oxide fuel cell using multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 304(C).
    12. Yin, Linfei & Wu, Yunzhi, 2022. "Mode-decomposition memory reinforcement network strategy for smart generation control in multi-area power systems containing renewable energy," Applied Energy, Elsevier, vol. 307(C).
    13. Huang, Ruchen & He, Hongwen & Gao, Miaojue, 2023. "Training-efficient and cost-optimal energy management for fuel cell hybrid electric bus based on a novel distributed deep reinforcement learning framework," Applied Energy, Elsevier, vol. 346(C).
    14. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    15. Dimitrios Vamvakas & Panagiotis Michailidis & Christos Korkas & Elias Kosmatopoulos, 2023. "Review and Evaluation of Reinforcement Learning Frameworks on Smart Grid Applications," Energies, MDPI, vol. 16(14), pages 1-38, July.
    16. Li, Jiawen & Yu, Tao & Zhang, Xiaoshun, 2022. "Coordinated load frequency control of multi-area integrated energy system using multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 306(PA).
    17. Shen, Rendong & Zhong, Shengyuan & Wen, Xin & An, Qingsong & Zheng, Ruifan & Li, Yang & Zhao, Jun, 2022. "Multi-agent deep reinforcement learning optimization framework for building energy system with renewable energy," Applied Energy, Elsevier, vol. 312(C).
    18. Gao, Yuan & Matsunami, Yuki & Miyata, Shohei & Akashi, Yasunori, 2022. "Multi-agent reinforcement learning dealing with hybrid action spaces: A case study for off-grid oriented renewable building energy system," Applied Energy, Elsevier, vol. 326(C).
    19. Li, Bin & Wang, Shuai & Li, Botong & Li, Hongbo & Wu, Jianzhong, 2023. "Optimal performance evaluation of thermal AGC units based on multi-dimensional feature analysis," Applied Energy, Elsevier, vol. 339(C).
    20. Yin, Linfei & Zheng, Da, 2024. "Decomposition prediction fractional-order PID reinforcement learning for short-term smart generation control of integrated energy systems," Applied Energy, Elsevier, vol. 355(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:324:y:2022:i:c:s0306261922010741. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.