IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v350y2023ics0306261923010681.html
   My bibliography  Save this article

Finding Nash equilibrium based on reinforcement learning for bidding strategy and distributed algorithm for ISO in imperfect electricity market

Author

Listed:
  • Yu, Liying
  • Wang, Peng
  • Chen, Zhe
  • Li, Dewen
  • Li, Ning
  • Cherkaoui, Rachid

Abstract

Finding Nash equilibrium in an imperfect market with active and strategic resources is key to the successful operation of the electricity market. A hierarchical Nash distributed reinforcement learning (HNDRL) framework is proposed to realize the interaction between the market participants and independent system operator (ISO). In the bidding stage, multi-agent Nash policy reinforcement learning is utilized to update the bidding strategies for market participants, i.e., generation company (GenCo) and distributed company (DisCo), which maximizes their own profit with imperfect information. The bidding action is updated by the probability iteration to deal with the continuous bidding strategy. And in the market clearing stage, the distributed dynamic-average consensus optimization algorithm is proposed at the ISO level to obtain the locational marginal price (LMP) and transaction quantities. With the participation of the demand-side in the electricity market, the proposed HNDRL framework can reach a Nash equilibrium with the optimal bidding strategy and handle global resource constraints in a distributed way. In addition, theoretical analysis is provided to ensure that the participants’ strategy exponentially converges to the Nash equilibrium for both convex and non-convex problems. Finally, the IEEE 30-bus system is employed to illustrate the efficiency. Compared with the existing method, the simulation results show that the proposed HNDRL framework has a faster convergence rate with the range of 2.667–3.333 times and can obtain higher profit with the range of 1.4286%–7.1429% at the Nash equilibrium point.

Suggested Citation

  • Yu, Liying & Wang, Peng & Chen, Zhe & Li, Dewen & Li, Ning & Cherkaoui, Rachid, 2023. "Finding Nash equilibrium based on reinforcement learning for bidding strategy and distributed algorithm for ISO in imperfect electricity market," Applied Energy, Elsevier, vol. 350(C).
    • Handle: RePEc:eee:appene:v:350:y:2023:i:c:s0306261923010681
    DOI: 10.1016/j.apenergy.2023.121704
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261923010681
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2023.121704?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Tsaousoglou, Georgios & Ellinas, Petros & Varvarigos, Emmanouel, 2023. "Operating peer-to-peer electricity markets under uncertainty via learning-based, distributed optimal control," Applied Energy, Elsevier, vol. 343(C).
    2. Daron Acemoglu & Ali Kakhbod & Asuman Ozdaglar, 2017. "Competition in Electricity Markets with Renewable Energy Sources," The Energy Journal, , vol. 38(1_suppl), pages 137-156, June.
    3. Federico, Giulio & Rahman, David, 2003. "Bidding in an Electricity Pay-as-Bid Auction," Journal of Regulatory Economics, Springer, vol. 24(2), pages 175-211, September.
    4. Hong, Qiuyi & Meng, Fanlin & Liu, Jian & Bo, Rui, 2023. "A bilevel game-theoretic decision-making framework for strategic retailers in both local and wholesale electricity markets," Applied Energy, Elsevier, vol. 330(PA).
    5. Wang, Jianhui & Zhou, Zhi & Botterud, Audun, 2011. "An evolutionary game approach to analyzing bidding strategies in electricity markets with elastic demand," Energy, Elsevier, vol. 36(5), pages 3459-3467.
    6. Tang, Hong & Wang, Shengwei, 2023. "Game-theoretic optimization of demand-side flexibility engagement considering the perspectives of different stakeholders and multiple flexibility services," Applied Energy, Elsevier, vol. 332(C).
    7. Lu, Xiaohui & Yang, Yang & Wang, Peifang & Fan, Yiming & Yu, Fangzhong & Zafetti, Nicholas, 2021. "A new converged Emperor Penguin Optimizer for biding strategy in a day-ahead deregulated market clearing price: A case study in China," Energy, Elsevier, vol. 227(C).
    8. Jong-Shi Pang & Masao Fukushima, 2005. "Quasi-variational inequalities, generalized Nash equilibria, and multi-leader-follower games," Computational Management Science, Springer, vol. 2(1), pages 21-56, January.
    9. Shen, Jian-jian & Cheng, Chun-tian & Jia, Ze-bin & Zhang, Yang & Lv, Quan & Cai, Hua-xiang & Wang, Bang-can & Xie, Meng-fei, 2022. "Impacts, challenges and suggestions of the electricity market for hydro-dominated power systems in China," Renewable Energy, Elsevier, vol. 187(C), pages 743-759.
    10. Huang, Yujing & Wang, Yudong & Liu, Nian, 2022. "A two-stage energy management for heat-electricity integrated energy system considering dynamic pricing of Stackelberg game and operation strategy optimization," Energy, Elsevier, vol. 244(PA).
    11. Moretti, Luca & Martelli, Emanuele & Manzolini, Giampaolo, 2020. "An efficient robust optimization model for the unit commitment and dispatch of multi-energy systems and microgrids," Applied Energy, Elsevier, vol. 261(C).
    12. Esmaeili Aliabadi, Danial & Kaya, Murat & Sahin, Guvenc, 2017. "Competition, risk and learning in electricity markets: An agent-based simulation study," Applied Energy, Elsevier, vol. 195(C), pages 1000-1011.
    13. Al-Agtash, Salem Y., 2010. "Supply curve bidding of electricity in constrained power networks," Energy, Elsevier, vol. 35(7), pages 2886-2892.
    14. Jin, Xiaolong & Wu, Qiuwei & Jia, Hongjie, 2020. "Local flexibility markets: Literature review on concepts, models and clearing methods," Applied Energy, Elsevier, vol. 261(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jiang, Yuzheng & Dong, Jun & Huang, Hexiang, 2024. "Optimal bidding strategy for the price-maker virtual power plant in the day-ahead market based on multi-agent twin delayed deep deterministic policy gradient algorithm," Energy, Elsevier, vol. 306(C).
    2. Adriano C. Lisboa & Fellipe F. G. Santos & Douglas A. G. Vieira & Rodney R. Saldanha & Felipe A. C. Pereira, 2025. "An Enhanced Gradient Algorithm for Computing Generalized Nash Equilibrium Applied to Electricity Market Games," Energies, MDPI, vol. 18(3), pages 1-14, February.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Li, Gong & Shi, Jing & Qu, Xiuli, 2011. "Modeling methods for GenCo bidding strategy optimization in the liberalized electricity spot market–A state-of-the-art review," Energy, Elsevier, vol. 36(8), pages 4686-4700.
    2. Huiru Zhao & Yuwei Wang & Sen Guo & Mingrui Zhao & Chao Zhang, 2016. "Application of a Gradient Descent Continuous Actor-Critic Algorithm for Double-Side Day-Ahead Electricity Market Modeling," Energies, MDPI, vol. 9(9), pages 1-20, September.
    3. Ghaninejad, Mousa, 2020. "عرضه، تقاضا، و پیشنهاد قیمت در بازار برق ایران [Supply, Demand, and Bidding in Iran’s Electricity Market]," MPRA Paper 105340, University Library of Munich, Germany.
    4. Polimeni, Simone & Moretti, Luca & Martelli, Emanuele & Leva, Sonia & Manzolini, Giampaolo, 2023. "A novel stochastic model for flexible unit commitment of off-grid microgrids," Applied Energy, Elsevier, vol. 331(C).
    5. Karim L. Anaya & Michael G. Pollitt, 2021. "How to Procure Flexibility Services within the Electricity Distribution System: Lessons from an International Review of Innovation Projects," Energies, MDPI, vol. 14(15), pages 1-26, July.
    6. Moritz Bohland & Sebastian Schwenen, 2020. "Technology Policy and Market Structure: Evidence from the Power Sector," Discussion Papers of DIW Berlin 1856, DIW Berlin, German Institute for Economic Research.
    7. Zhihan Shi & Weisong Han & Guangming Zhang & Zhiqing Bai & Mingxiang Zhu & Xiaodong Lv, 2022. "Research on Low-Carbon Energy Sharing through the Alliance of Integrated Energy Systems with Multiple Uncertainties," Energies, MDPI, vol. 15(24), pages 1-20, December.
    8. Benedikt Finnah, 2022. "Optimal bidding functions for renewable energies in sequential electricity markets," OR Spectrum: Quantitative Approaches in Management, Springer;Gesellschaft für Operations Research e.V., vol. 44(1), pages 1-27, March.
    9. Pinto, T. & Morais, H. & Oliveira, P. & Vale, Z. & Praça, I. & Ramos, C., 2011. "A new approach for multi-agent coalition formation and management in the scope of electricity markets," Energy, Elsevier, vol. 36(8), pages 5004-5015.
    10. Giorgia Oggioni & Yves Smeers & Elisabetta Allevi & Siegfried Schaible, 2012. "A Generalized Nash Equilibrium Model of Market Coupling in the European Power System," Networks and Spatial Economics, Springer, vol. 12(4), pages 503-560, December.
    11. Jiang, Sufan & Gao, Shan & Pan, Guangsheng & Zhao, Xin & Liu, Yu & Guo, Yasen & Wang, Sicheng, 2020. "A novel robust security constrained unit commitment model considering HVDC regulation," Applied Energy, Elsevier, vol. 278(C).
    12. Wang, Tonghe & Hua, Haochen & Shi, Tianying & Wang, Rui & Sun, Yizhong & Naidoo, Pathmanathan, 2024. "A bi-level dispatch optimization of multi-microgrid considering green electricity consumption willingness under renewable portfolio standard policy," Applied Energy, Elsevier, vol. 356(C).
    13. Canan Karatekin & Hakan elik, 2020. "The Effects of Renewable Energy Sources on the Structure of the Turkish Electricity Market," International Journal of Energy Economics and Policy, Econjournals, vol. 10(2), pages 64-70.
    14. Wang, Yi & Qiu, Dawei & Sun, Mingyang & Strbac, Goran & Gao, Zhiwei, 2023. "Secure energy management of multi-energy microgrid: A physical-informed safe reinforcement learning approach," Applied Energy, Elsevier, vol. 335(C).
    15. Ciarcià, Carla & Daniele, Patrizia, 2016. "New existence theorems for quasi-variational inequalities and applications to financial models," European Journal of Operational Research, Elsevier, vol. 251(1), pages 288-299.
    16. Majid Motamedi & Shahram Moeeni & Salman Gharakhani & Iman Keyfarokhi, 2014. "The Behavior of Iranian Restructured Electricity Market in Supply Function Equilibrium Framework," International Journal of Academic Research in Business and Social Sciences, Human Resource Management Academic Research Society, International Journal of Academic Research in Business and Social Sciences, vol. 4(1), pages 178-191, January.
    17. Lu Wei & Yiyin Li & Boyu Xie & Ke Xu & Gaojun Meng, 2024. "Two-Stage Optimal Scheduling Based on the Meteorological Prediction of a Wind–Solar-Energy Storage System with Demand Response," Energies, MDPI, vol. 17(6), pages 1-13, March.
    18. Lan, Penghang & Chen, She & Li, Qihang & Li, Kelin & Wang, Feng & Zhao, Yaoxun, 2024. "Intelligent hydrogen-ammonia combined energy storage system with deep reinforcement learning," Renewable Energy, Elsevier, vol. 237(PB).
    19. Contreras, Javier & Krawczyk, Jacek & Zuccollo, James, 2008. "The invisible polluter: Can regulators save consumer surplus?," MPRA Paper 9890, University Library of Munich, Germany.
    20. Llobet, Gerard & Fabra, Natalia, 2019. "Auctions with Unknown Capacities: Understanding Competition among Renewables," CEPR Discussion Papers 14060, C.E.P.R. Discussion Papers.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:350:y:2023:i:c:s0306261923010681. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.