IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v376y2024ipas0306261924013680.html
   My bibliography  Save this article

Integrated three-stage decentralized scheduling for virtual power plants: A model-assisted multi-agent reinforcement learning method

Author

Listed:
  • Xu, Biao
  • Luan, Wenpeng
  • Yang, Jing
  • Zhao, Bochao
  • Long, Chao
  • Ai, Qian
  • Xiang, Jiani

Abstract

Virtual power plant (VPP) emerges as a promising integration and aggregation technology that facilitates the utilization of massive flexible demand-side resources (DSRs). However, non-negligible modeling errors and high-dimensional uncertainties involved in DSR aggregation threaten the delivery reliability and cost-effectiveness of VPP operation. To address this problem, this study proposes an integrated three-stage scheduling framework for VPPs and develops a model-assisted multi-agent reinforcement learning (MARL) approach. In the proposed framework, the VPP scheduling problem is formulated as a decentralized partially observable Markov Decision Process (Dec-POMDP), which depicts the complex interaction process among the three stages (bidding, re-dispatching and disaggregation). The interactions are evaluated by a comprehensive reward function, incorporating the trading and operation costs, as well as imbalance penalties. To enable decentralized decision-making, a model-assisted multi-agent proximal policy optimization (MA2PPO) algorithm is proposed, which trains a separate actor network for each aggregator. Additionally, the MA2PPO is augmented with a model-assisted safety decision-making method to accelerate the training process. Numerical simulation results verify that the proposed method enhances the delivery reliability and cost-effectiveness of the VPP, while achieving faster convergence time compared with purely model-free MARL methods.

Suggested Citation

  • Xu, Biao & Luan, Wenpeng & Yang, Jing & Zhao, Bochao & Long, Chao & Ai, Qian & Xiang, Jiani, 2024. "Integrated three-stage decentralized scheduling for virtual power plants: A model-assisted multi-agent reinforcement learning method," Applied Energy, Elsevier, vol. 376(PA).
    • Handle: RePEc:eee:appene:v:376:y:2024:i:pa:s0306261924013680
    DOI: 10.1016/j.apenergy.2024.123985
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261924013680
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2024.123985?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Shayegan-Rad, Ali & Badri, Ali & Zangeneh, Ali, 2017. "Day-ahead scheduling of virtual power plant in joint energy and regulation reserve markets under uncertainties," Energy, Elsevier, vol. 121(C), pages 114-125.
    2. Ding, Yixing & Xu, Qingshan & Hao, Lili & Xia, Yuanxing, 2023. "A Stackelberg Game-based robust optimization for user-side energy storage configuration and power pricing," Energy, Elsevier, vol. 283(C).
    3. Li, Xiangyu & Luo, Fengji & Li, Chaojie, 2024. "Multi-agent deep reinforcement learning-based autonomous decision-making framework for community virtual power plants," Applied Energy, Elsevier, vol. 360(C).
    4. Li, Xinchao & Lu, Shan & Li, Zhe & Wang, Yue & Zhu, Li, 2022. "Modeling and optimization of bioethanol production planning under hybrid uncertainty: A heuristic multi-stage stochastic programming approach," Energy, Elsevier, vol. 245(C).
    5. Hasankhani, Arezoo & Hakimi, Seyed Mehdi, 2021. "Stochastic energy management of smart microgrid with intermittent renewable energy resources in electricity market," Energy, Elsevier, vol. 219(C).
    6. Iria, José & Soares, Filipe, 2019. "Real-time provision of multiple electricity market products by an aggregator of prosumers," Applied Energy, Elsevier, vol. 255(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wang, Tonghe & Hua, Haochen & Shi, Tianying & Wang, Rui & Sun, Yizhong & Naidoo, Pathmanathan, 2024. "A bi-level dispatch optimization of multi-microgrid considering green electricity consumption willingness under renewable portfolio standard policy," Applied Energy, Elsevier, vol. 356(C).
    2. Jani, Ali & Karimi, Hamid & Jadid, Shahram, 2022. "Two-layer stochastic day-ahead and real-time energy management of networked microgrids considering integration of renewable energy resources," Applied Energy, Elsevier, vol. 323(C).
    3. Kaiyan Wang & Xueyan Wang & Rong Jia & Jian Dang & Yan Liang & Haodong Du, 2022. "Research on Coupled Cooperative Operation of Medium- and Long-Term and Spot Electricity Transaction for Multi-Energy System: A Case Study in China," Sustainability, MDPI, vol. 14(17), pages 1-20, August.
    4. Elkholy, M.H. & Metwally, Hamid & Farahat, M.A. & Senjyu, Tomonobu & Elsayed Lotfy, Mohammed, 2022. "Smart centralized energy management system for autonomous microgrid using FPGA," Applied Energy, Elsevier, vol. 317(C).
    5. Nemanja Mišljenović & Matej Žnidarec & Goran Knežević & Damir Šljivac & Andreas Sumper, 2023. "A Review of Energy Management Systems and Organizational Structures of Prosumers," Energies, MDPI, vol. 16(7), pages 1-32, March.
    6. Arul Rajagopalan & Karthik Nagarajan & Oscar Danilo Montoya & Seshathiri Dhanasekaran & Inayathullah Abdul Kareem & Angalaeswari Sendraya Perumal & Natrayan Lakshmaiya & Prabhu Paramasivam, 2022. "Multi-Objective Optimal Scheduling of a Microgrid Using Oppositional Gradient-Based Grey Wolf Optimizer," Energies, MDPI, vol. 15(23), pages 1-24, November.
    7. Guoqiang Sun & Weihang Qian & Wenjin Huang & Zheng Xu & Zhongxing Fu & Zhinong Wei & Sheng Chen, 2019. "Stochastic Adaptive Robust Dispatch for Virtual Power Plants Using the Binding Scenario Identification Approach," Energies, MDPI, vol. 12(10), pages 1-23, May.
    8. Iria, José & Scott, Paul & Attarha, Ahmad, 2020. "Network-constrained bidding optimization strategy for aggregators of prosumers," Energy, Elsevier, vol. 207(C).
    9. Fischer, Robert & Toffolo, Andrea, 2022. "Is total system cost minimization fair to all the actors of an energy system? Not according to game theory," Energy, Elsevier, vol. 239(PC).
    10. Mojtaba Farrokh & Ehsan Ahmadi & Minghe Sun, 2023. "A robust stochastic possibilistic programming model for dynamic supply chain network design with pricing and technology selection decisions," OPSEARCH, Springer;Operational Research Society of India, vol. 60(3), pages 1082-1120, September.
    11. Nicolai Lystbæk & Mikkel Gregersen & Hamid Reza Shaker, 2023. "Review of Energy Portfolio Optimization in Energy Markets Considering Flexibility of Power-to-X," Sustainability, MDPI, vol. 15(5), pages 1-17, March.
    12. Wu, Zhongqun & Yang, Chan & Zheng, Ruijin, 2022. "Developing a holistic fuzzy hierarchy-cloud assessment model for the connection risk of renewable energy microgrid," Energy, Elsevier, vol. 245(C).
    13. Arriet, Andrea & Matis, Timothy I. & Feijoo, Felipe, 2024. "Electricity sector impacts of water taxation for natural gas supply under high renewable generation," Energy, Elsevier, vol. 294(C).
    14. Ran, Cuiling & Zhang, Yanzi & Yin, Ying, 2021. "Demand response to improve the shared electric vehicle planning: Managerial insights, sustainable benefits," Applied Energy, Elsevier, vol. 292(C).
    15. M. Ebrahim Adabi & Bogdan Marinescu, 2022. "Direct Participation of Dynamic Virtual Power Plants in Secondary Frequency Control," Energies, MDPI, vol. 15(8), pages 1-15, April.
    16. Li, Ningning & Gao, Yan, 2023. "Real-time pricing based on convex hull method for smart grid with multiple generating units," Energy, Elsevier, vol. 285(C).
    17. Saeian, Hosein & Niknam, Taher & Zare, Mohsen & Aghaei, Jamshid, 2022. "Coordinated optimal bidding strategies methods of aggregated microgrids: A game theory-based demand side management under an electricity market environment," Energy, Elsevier, vol. 245(C).
    18. Jun Dong & Dongran Liu & Xihao Dou & Bo Li & Shiyao Lv & Yuzheng Jiang & Tongtao Ma, 2021. "Key Issues and Technical Applications in the Study of Power Markets as the System Adapts to the New Power System in China," Sustainability, MDPI, vol. 13(23), pages 1-29, December.
    19. Zhang, Chao & Yin, Wanjun & Wen, Tao, 2024. "An advanced multi-objective collaborative scheduling strategy for large scale EV charging and discharging connected to the predictable wind power grid," Energy, Elsevier, vol. 287(C).
    20. Khaloie, Hooman & Anvari-Moghaddam, Amjad & Contreras, Javier & Siano, Pierluigi, 2021. "Risk-involved optimal operating strategy of a hybrid power generation company: A mixed interval-CVaR model," Energy, Elsevier, vol. 232(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:376:y:2024:i:pa:s0306261924013680. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.