IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v306y2024ics0360544224021625.html
   My bibliography  Save this article

Optimal bidding strategy for the price-maker virtual power plant in the day-ahead market based on multi-agent twin delayed deep deterministic policy gradient algorithm

Author

Listed:
  • Jiang, Yuzheng
  • Dong, Jun
  • Huang, Hexiang

Abstract

In recent years, distributed energy resources (DERs) have developed greatly, and the total capacity of resources aggregated by virtual power plants (VPPs) has increased. The role VPP plays in the market is changing. It is necessary to explore VPP's bidding strategy in the electricity market based on the impact of VPP on the clearing price. At the same time, the strategy game of different market players will also impact VPP's strategy. The multi-agent twin delayed deep deterministic policy gradient (MATD3) algorithm is used to solve the problem of price-maker VPP participation in the day-ahead (DA) market. VPP is required to submit the power and prices of electricity in multi-segments at different times. In the market bidding process, the main market participants are considered as agents. Iterative refinement of each player's strategy is achieved through multi-agent reinforcement learning, with the primary aim of maximizing revenue. For the VPP, the MATD3 algorithm improved the reward by approximately 65 % and increased the convergence speed by 17 % compared to the multi-agent deep deterministic policy gradient and multi-agent proximal policy optimization algorithms. The effects of different influences on VPP's bidding strategy are analyzed, which can provide a decision-making reference for market participants.

Suggested Citation

  • Jiang, Yuzheng & Dong, Jun & Huang, Hexiang, 2024. "Optimal bidding strategy for the price-maker virtual power plant in the day-ahead market based on multi-agent twin delayed deep deterministic policy gradient algorithm," Energy, Elsevier, vol. 306(C).
    • Handle: RePEc:eee:energy:v:306:y:2024:i:c:s0360544224021625
    DOI: 10.1016/j.energy.2024.132388
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0360544224021625
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.energy.2024.132388?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Wang, Jian & Ilea, Valentin & Bovo, Cristian & Xie, Ning & Wang, Yong, 2023. "Optimal self-scheduling for a multi-energy virtual power plant providing energy and reserve services under a holistic market framework," Energy, Elsevier, vol. 278(PB).
    2. Zamani, Ali Ghahgharaee & Zakariazadeh, Alireza & Jadid, Shahram, 2016. "Day-ahead resource scheduling of a renewable energy based virtual power plant," Applied Energy, Elsevier, vol. 169(C), pages 324-340.
    3. Yang, Qing & Wang, Hao & Wang, Taotao & Zhang, Shengli & Wu, Xiaoxiao & Wang, Hui, 2021. "Blockchain-based decentralized energy management platform for residential distributed energy resources in a virtual power plant," Applied Energy, Elsevier, vol. 294(C).
    4. Sun, Qingkai & Wang, Xiaojun & Liu, Zhao & Mirsaeidi, Sohrab & He, Jinghan & Pei, Wei, 2022. "Multi-agent energy management optimization for integrated energy systems under the energy and carbon co-trading market," Applied Energy, Elsevier, vol. 324(C).
    5. Li, Qiang & Wei, Fanchao & Zhou, Yongcheng & Li, Jiajia & Zhou, Guowen & Wang, Zhonghao & Liu, Jinfu & Yan, Peigang & Yu, Daren, 2023. "A scheduling framework for VPP considering multiple uncertainties and flexible resources," Energy, Elsevier, vol. 282(C).
    6. Shabanzadeh, Morteza & Sheikh-El-Eslami, Mohammad-Kazem & Haghifam, Mahmoud-Reza, 2016. "A medium-term coalition-forming model of heterogeneous DERs for a commercial virtual power plant," Applied Energy, Elsevier, vol. 169(C), pages 663-681.
    7. Yu, Liying & Wang, Peng & Chen, Zhe & Li, Dewen & Li, Ning & Cherkaoui, Rachid, 2023. "Finding Nash equilibrium based on reinforcement learning for bidding strategy and distributed algorithm for ISO in imperfect electricity market," Applied Energy, Elsevier, vol. 350(C).
    8. Bjørndal, Endre & Bjørndal, Mette Helene & Coniglio, Stefano & Körner, Marc-Fabian & Leinauer, Christina & Weibelzahl, Martin, 2023. "Energy storage operation and electricity market design: On the market power of monopolistic storage operators," European Journal of Operational Research, Elsevier, vol. 307(2), pages 887-909.
    9. Loßner, Martin & Böttger, Diana & Bruckner, Thomas, 2017. "Economic assessment of virtual power plants in the German energy market — A scenario-based and model-supported analysis," Energy Economics, Elsevier, vol. 62(C), pages 125-138.
    10. Zeng, Yu & Wei, Xuan & Yao, Yuan & Xu, Yinliang & Sun, Hongbin & Kin Victor Chan, Wai & Feng, Wei, 2023. "Determining the pricing and deployment strategy for virtual power plants of peer-to-peer prosumers: A game-theoretic approach," Applied Energy, Elsevier, vol. 345(C).
    11. Wang, Zheng & Li, Chaojie & Zhou, Xiaojun & Xie, Renyou & Li, Xiangyu & Dong, Zhaoyang, 2023. "Stochastic bidding for VPPs enabled ancillary services: A case study," Applied Energy, Elsevier, vol. 352(C).
    12. Wozabal, David & Rameseder, Gunther, 2020. "Optimal bidding of a virtual power plant on the Spanish day-ahead and intraday market for electricity," European Journal of Operational Research, Elsevier, vol. 280(2), pages 639-655.
    13. Fusco, Andrea & Gioffrè, Domenico & Francesco Castelli, Alessandro & Bovo, Cristian & Martelli, Emanuele, 2023. "A multi-stage stochastic programming model for the unit commitment of conventional and virtual power plants bidding in the day-ahead and ancillary services markets," Applied Energy, Elsevier, vol. 336(C).
    14. Huang, Yin & Kang, Zehao & Mao, Xuping & Hu, Haoqin & Tan, Jiaqi & Xuan, Dongji, 2023. "Deep reinforcement learning based energymanagement strategy considering running costs and energy source aging for fuel cell hybrid electric vehicle," Energy, Elsevier, vol. 283(C).
    15. Shafiekhani, Morteza & Ahmadi, Abdollah & Homaee, Omid & Shafie-khah, Miadreza & Catalão, João P.S., 2022. "Optimal bidding strategy of a renewable-based virtual power plant including wind and solar units and dispatchable loads," Energy, Elsevier, vol. 239(PD).
    16. Jiakai Hu & Chuanwen Jiang & Yangyang Liu, 2019. "Short-Term Bidding Strategy for a Price-Maker Virtual Power Plant Based on Interval Optimization," Energies, MDPI, vol. 12(19), pages 1-22, September.
    17. Hadayeghparast, Shahrzad & SoltaniNejad Farsangi, Alireza & Shayanfar, Heidarali, 2019. "Day-ahead stochastic multi-objective economic/emission operational scheduling of a large scale virtual power plant," Energy, Elsevier, vol. 172(C), pages 630-646.
    18. Nosratabadi, Seyyed Mostafa & Hooshmand, Rahmat-Allah & Gholipour, Eskandar, 2017. "A comprehensive review on microgrid and virtual power plant concepts employed for distributed energy resources scheduling in power systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 67(C), pages 341-363.
    19. Wang, Han & Riaz, Shariq & Mancarella, Pierluigi, 2020. "Integrated techno-economic modeling, flexibility analysis, and business case assessment of an urban virtual power plant with multi-market co-optimization," Applied Energy, Elsevier, vol. 259(C).
    20. Esmaeili Aliabadi, Danial & Kaya, Murat & Sahin, Guvenc, 2017. "Competition, risk and learning in electricity markets: An agent-based simulation study," Applied Energy, Elsevier, vol. 195(C), pages 1000-1011.
    21. Naval, Natalia & Yusta, Jose M., 2021. "Virtual power plant models and electricity markets - A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 149(C).
    22. Zhang, Tianhan & Qiu, Weiqiang & Zhang, Zhi & Lin, Zhenzhi & Ding, Yi & Wang, Yiting & Wang, Lianfang & Yang, Li, 2023. "Optimal bidding strategy and profit allocation method for shared energy storage-assisted VPP in joint energy and regulation markets," Applied Energy, Elsevier, vol. 329(C).
    23. Wei, Xuan & Xu, Yinliang & Sun, Hongbin & Bai, Xiang & Chang, Xinyue & Xue, Yixun, 2024. "Day-ahead optimal dispatch of a virtual power plant in the joint energy-reserve-carbon market," Applied Energy, Elsevier, vol. 356(C).
    24. Yu, Songyuan & Fang, Fang & Liu, Yajuan & Liu, Jizhen, 2019. "Uncertainties of virtual power plant: Problems and countermeasures," Applied Energy, Elsevier, vol. 239(C), pages 454-470.
    25. Liang, Tao & Chai, Lulu & Tan, Jianxin & Jing, Yanwei & Lv, Liangnian, 2024. "Dynamic optimization of an integrated energy system with carbon capture and power-to-gas interconnection: A deep reinforcement learning-based scheduling strategy," Applied Energy, Elsevier, vol. 367(C).
    26. Jia, Chunchun & Li, Kunang & He, Hongwen & Zhou, Jiaming & Li, Jianwei & Wei, Zhongbao, 2023. "Health-aware energy management strategy for fuel cell hybrid bus considering air-conditioning control based on TD3 algorithm," Energy, Elsevier, vol. 283(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Naval, Natalia & Yusta, Jose M., 2021. "Virtual power plant models and electricity markets - A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 149(C).
    2. Mohammad Mohammadi Roozbehani & Ehsan Heydarian-Forushani & Saeed Hasanzadeh & Seifeddine Ben Elghali, 2022. "Virtual Power Plant Operational Strategies: Models, Markets, Optimization, Challenges, and Opportunities," Sustainability, MDPI, vol. 14(19), pages 1-23, September.
    3. Natalia Naval & Jose M. Yusta, 2020. "Water-Energy Management for Demand Charges and Energy Cost Optimization of a Pumping Stations System under a Renewable Virtual Power Plant Model," Energies, MDPI, vol. 13(11), pages 1-21, June.
    4. Wafa Nafkha-Tayari & Seifeddine Ben Elghali & Ehsan Heydarian-Forushani & Mohamed Benbouzid, 2022. "Virtual Power Plants Optimization Issue: A Comprehensive Review on Methods, Solutions, and Prospects," Energies, MDPI, vol. 15(10), pages 1-20, May.
    5. Jiaqi Liu & Hongji Hu & Samson S. Yu & Hieu Trinh, 2023. "Virtual Power Plant with Renewable Energy Sources and Energy Storage Systems for Sustainable Power Grid-Formation, Control Techniques and Demand Response," Energies, MDPI, vol. 16(9), pages 1-28, April.
    6. Mahmud, Khizir & Khan, Behram & Ravishankar, Jayashri & Ahmadi, Abdollah & Siano, Pierluigi, 2020. "An internet of energy framework with distributed energy resources, prosumers and small-scale virtual power plants: An overview," Renewable and Sustainable Energy Reviews, Elsevier, vol. 127(C).
    7. Reza Nadimi & Masahito Takahashi & Koji Tokimatsu & Mika Goto, 2024. "The Reliability and Profitability of Virtual Power Plant with Short-Term Power Market Trading and Non-Spinning Reserve Diesel Generator," Energies, MDPI, vol. 17(9), pages 1-19, April.
    8. Yu, Songyuan & Fang, Fang & Liu, Yajuan & Liu, Jizhen, 2019. "Uncertainties of virtual power plant: Problems and countermeasures," Applied Energy, Elsevier, vol. 239(C), pages 454-470.
    9. Naval, Natalia & Sánchez, Raul & Yusta, Jose M., 2020. "A virtual power plant optimal dispatch model with large and small-scale distributed renewable generation," Renewable Energy, Elsevier, vol. 151(C), pages 57-69.
    10. Mei, Shufan & Tan, Qinliang & Liu, Yuan & Trivedi, Anupam & Srinivasan, Dipti, 2023. "Optimal bidding strategy for virtual power plant participating in combined electricity and ancillary services market considering dynamic demand response price and integrated consumption satisfaction," Energy, Elsevier, vol. 284(C).
    11. Amit Kumer Podder & Sayemul Islam & Nallapaneni Manoj Kumar & Aneesh A. Chand & Pulivarthi Nageswara Rao & Kushal A. Prasad & T. Logeswaran & Kabir A. Mamun, 2020. "Systematic Categorization of Optimization Strategies for Virtual Power Plants," Energies, MDPI, vol. 13(23), pages 1-46, November.
    12. Cheng, Xiaoyuan & Yao, Ruiqiu & Postnikov, Andrey & Hu, Yukun & Varga, Liz, 2024. "Decentralized intelligent multi-party competitive aggregation framework for electricity prosumers," Applied Energy, Elsevier, vol. 373(C).
    13. Xie, Haonan & Ahmad, Tanveer & Zhang, Dongdong & Goh, Hui Hwang & Wu, Thomas, 2024. "Community-based virtual power plants’ technology and circular economy models in the energy sector: A Techno-economy study," Renewable and Sustainable Energy Reviews, Elsevier, vol. 192(C).
    14. Sergio Cantillo-Luna & Ricardo Moreno-Chuquen & David Celeita & George J. Anders, 2024. "A Stochastic Decision-Making Tool Suite for Distributed Energy Resources Integration in Energy Markets," Energies, MDPI, vol. 17(10), pages 1-28, May.
    15. Li, Jinchao & Sun, Zihao & Niu, Xiaoxuan & Li, Shiwei, 2024. "Economic optimization scheduling of virtual power plants considering an incentive based tiered carbon price," Energy, Elsevier, vol. 305(C).
    16. Yuanyuan, Zhang & Huiru, Zhao & Bingkang, Li, 2023. "Distributionally robust comprehensive declaration strategy of virtual power plant participating in the power market considering flexible ramping product and uncertainties," Applied Energy, Elsevier, vol. 343(C).
    17. Li, Qiang & Zhou, Yongcheng & Wei, Fanchao & Li, Shuangxiu & Wang, Zhonghao & Li, Jiajia & Zhou, Guowen & Liu, Jinfu & Yan, Peigang & Yu, Daren, 2024. "Multi-time scale scheduling for virtual power plants: Integrating the flexibility of power generation and multi-user loads while considering the capacity degradation of energy storage systems," Applied Energy, Elsevier, vol. 362(C).
    18. Wu, Haochi & Qiu, Dawei & Zhang, Liyu & Sun, Mingyang, 2024. "Adaptive multi-agent reinforcement learning for flexible resource management in a virtual power plant with dynamic participating multi-energy buildings," Applied Energy, Elsevier, vol. 374(C).
    19. Martin Bichler & Hans Ulrich Buhl & Johannes Knörr & Felipe Maldonado & Paul Schott & Stefan Waldherr & Martin Weibelzahl, 2022. "Electricity Markets in a Time of Change: A Call to Arms for Business Research," Schmalenbach Journal of Business Research, Springer, vol. 74(1), pages 77-102, March.
    20. Weigang Jin & Peihua Wang & Jiaxin Yuan, 2024. "Key Role and Optimization Dispatch Research of Technical Virtual Power Plants in the New Energy Era," Energies, MDPI, vol. 17(22), pages 1-24, November.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:306:y:2024:i:c:s0360544224021625. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.