IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v350y2023ics0306261923010681.html
   My bibliography  Save this article

Finding Nash equilibrium based on reinforcement learning for bidding strategy and distributed algorithm for ISO in imperfect electricity market

Author

Listed:
  • Yu, Liying
  • Wang, Peng
  • Chen, Zhe
  • Li, Dewen
  • Li, Ning
  • Cherkaoui, Rachid

Abstract

Finding Nash equilibrium in an imperfect market with active and strategic resources is key to the successful operation of the electricity market. A hierarchical Nash distributed reinforcement learning (HNDRL) framework is proposed to realize the interaction between the market participants and independent system operator (ISO). In the bidding stage, multi-agent Nash policy reinforcement learning is utilized to update the bidding strategies for market participants, i.e., generation company (GenCo) and distributed company (DisCo), which maximizes their own profit with imperfect information. The bidding action is updated by the probability iteration to deal with the continuous bidding strategy. And in the market clearing stage, the distributed dynamic-average consensus optimization algorithm is proposed at the ISO level to obtain the locational marginal price (LMP) and transaction quantities. With the participation of the demand-side in the electricity market, the proposed HNDRL framework can reach a Nash equilibrium with the optimal bidding strategy and handle global resource constraints in a distributed way. In addition, theoretical analysis is provided to ensure that the participants’ strategy exponentially converges to the Nash equilibrium for both convex and non-convex problems. Finally, the IEEE 30-bus system is employed to illustrate the efficiency. Compared with the existing method, the simulation results show that the proposed HNDRL framework has a faster convergence rate with the range of 2.667–3.333 times and can obtain higher profit with the range of 1.4286%–7.1429% at the Nash equilibrium point.

Suggested Citation

  • Yu, Liying & Wang, Peng & Chen, Zhe & Li, Dewen & Li, Ning & Cherkaoui, Rachid, 2023. "Finding Nash equilibrium based on reinforcement learning for bidding strategy and distributed algorithm for ISO in imperfect electricity market," Applied Energy, Elsevier, vol. 350(C).
    • Handle: RePEc:eee:appene:v:350:y:2023:i:c:s0306261923010681
    DOI: 10.1016/j.apenergy.2023.121704
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261923010681
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2023.121704?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Shen, Jian-jian & Cheng, Chun-tian & Jia, Ze-bin & Zhang, Yang & Lv, Quan & Cai, Hua-xiang & Wang, Bang-can & Xie, Meng-fei, 2022. "Impacts, challenges and suggestions of the electricity market for hydro-dominated power systems in China," Renewable Energy, Elsevier, vol. 187(C), pages 743-759.
    2. Tsaousoglou, Georgios & Ellinas, Petros & Varvarigos, Emmanouel, 2023. "Operating peer-to-peer electricity markets under uncertainty via learning-based, distributed optimal control," Applied Energy, Elsevier, vol. 343(C).
    3. Daron Acemoglu & Ali Kakhbod & Asuman Ozdaglar, 2017. "Competition in Electricity Markets with Renewable Energy Sources," The Energy Journal, , vol. 38(1_suppl), pages 137-156, June.
    4. Federico, Giulio & Rahman, David, 2003. "Bidding in an Electricity Pay-as-Bid Auction," Journal of Regulatory Economics, Springer, vol. 24(2), pages 175-211, September.
    5. Tang, Hong & Wang, Shengwei, 2023. "Game-theoretic optimization of demand-side flexibility engagement considering the perspectives of different stakeholders and multiple flexibility services," Applied Energy, Elsevier, vol. 332(C).
    6. Lu, Xiaohui & Yang, Yang & Wang, Peifang & Fan, Yiming & Yu, Fangzhong & Zafetti, Nicholas, 2021. "A new converged Emperor Penguin Optimizer for biding strategy in a day-ahead deregulated market clearing price: A case study in China," Energy, Elsevier, vol. 227(C).
    7. Huang, Yujing & Wang, Yudong & Liu, Nian, 2022. "A two-stage energy management for heat-electricity integrated energy system considering dynamic pricing of Stackelberg game and operation strategy optimization," Energy, Elsevier, vol. 244(PA).
    8. Moretti, Luca & Martelli, Emanuele & Manzolini, Giampaolo, 2020. "An efficient robust optimization model for the unit commitment and dispatch of multi-energy systems and microgrids," Applied Energy, Elsevier, vol. 261(C).
    9. Hong, Qiuyi & Meng, Fanlin & Liu, Jian & Bo, Rui, 2023. "A bilevel game-theoretic decision-making framework for strategic retailers in both local and wholesale electricity markets," Applied Energy, Elsevier, vol. 330(PA).
    10. Esmaeili Aliabadi, Danial & Kaya, Murat & Sahin, Guvenc, 2017. "Competition, risk and learning in electricity markets: An agent-based simulation study," Applied Energy, Elsevier, vol. 195(C), pages 1000-1011.
    11. Wang, Jianhui & Zhou, Zhi & Botterud, Audun, 2011. "An evolutionary game approach to analyzing bidding strategies in electricity markets with elastic demand," Energy, Elsevier, vol. 36(5), pages 3459-3467.
    12. Al-Agtash, Salem Y., 2010. "Supply curve bidding of electricity in constrained power networks," Energy, Elsevier, vol. 35(7), pages 2886-2892.
    13. Jong-Shi Pang & Masao Fukushima, 2005. "Quasi-variational inequalities, generalized Nash equilibria, and multi-leader-follower games," Computational Management Science, Springer, vol. 2(1), pages 21-56, January.
    14. Jin, Xiaolong & Wu, Qiuwei & Jia, Hongjie, 2020. "Local flexibility markets: Literature review on concepts, models and clearing methods," Applied Energy, Elsevier, vol. 261(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jiang, Yuzheng & Dong, Jun & Huang, Hexiang, 2024. "Optimal bidding strategy for the price-maker virtual power plant in the day-ahead market based on multi-agent twin delayed deep deterministic policy gradient algorithm," Energy, Elsevier, vol. 306(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ghaninejad, Mousa, 2020. "عرضه، تقاضا، و پیشنهاد قیمت در بازار برق ایران [Supply, Demand, and Bidding in Iran’s Electricity Market]," MPRA Paper 105340, University Library of Munich, Germany.
    2. Li, Gong & Shi, Jing & Qu, Xiuli, 2011. "Modeling methods for GenCo bidding strategy optimization in the liberalized electricity spot market–A state-of-the-art review," Energy, Elsevier, vol. 36(8), pages 4686-4700.
    3. Huiru Zhao & Yuwei Wang & Sen Guo & Mingrui Zhao & Chao Zhang, 2016. "Application of a Gradient Descent Continuous Actor-Critic Algorithm for Double-Side Day-Ahead Electricity Market Modeling," Energies, MDPI, vol. 9(9), pages 1-20, September.
    4. Polimeni, Simone & Moretti, Luca & Martelli, Emanuele & Leva, Sonia & Manzolini, Giampaolo, 2023. "A novel stochastic model for flexible unit commitment of off-grid microgrids," Applied Energy, Elsevier, vol. 331(C).
    5. Karim L. Anaya & Michael G. Pollitt, 2021. "How to Procure Flexibility Services within the Electricity Distribution System: Lessons from an International Review of Innovation Projects," Energies, MDPI, vol. 14(15), pages 1-26, July.
    6. Axel Dreves & Christian Kanzow, 2011. "Nonsmooth optimization reformulations characterizing all solutions of jointly convex generalized Nash equilibrium problems," Computational Optimization and Applications, Springer, vol. 50(1), pages 23-48, September.
    7. Armghan, Hammad & Xu, Yinliang & Bai, Xiang & Ali, Naghmash & Chang, Xinyue & Xue, Yixun, 2024. "A tri-level control framework for carbon-aware multi-energy microgrid cluster considering shared hydrogen energy storage," Applied Energy, Elsevier, vol. 373(C).
    8. Flam, Sjur & Ruszczynski, A., 2006. "Computing Normalized Equilibria in Convex-Concave Games," Working Papers 2006:9, Lund University, Department of Economics.
    9. Ali, Liaqat & Azim, M. Imran & Peters, Jan & Pashajavid, Ehsan, 2024. "Expediting battery investment returns for residential customers utilising spot price-aware local energy exchanges," Energy, Elsevier, vol. 306(C).
    10. repec:diw:diwwpp:dp1856 is not listed on IDEAS
    11. Johannes Dahlke & Kristina Bogner & Matthias Mueller & Thomas Berger & Andreas Pyka & Bernd Ebersberger, 2020. "Is the Juice Worth the Squeeze? Machine Learning (ML) In and For Agent-Based Modelling (ABM)," Papers 2003.11985, arXiv.org.
    12. Zhihan Shi & Weisong Han & Guangming Zhang & Zhiqing Bai & Mingxiang Zhu & Xiaodong Lv, 2022. "Research on Low-Carbon Energy Sharing through the Alliance of Integrated Energy Systems with Multiple Uncertainties," Energies, MDPI, vol. 15(24), pages 1-20, December.
    13. Benedikt Finnah, 2022. "Optimal bidding functions for renewable energies in sequential electricity markets," OR Spectrum: Quantitative Approaches in Management, Springer;Gesellschaft für Operations Research e.V., vol. 44(1), pages 1-27, March.
    14. Pinto, T. & Morais, H. & Oliveira, P. & Vale, Z. & Praça, I. & Ramos, C., 2011. "A new approach for multi-agent coalition formation and management in the scope of electricity markets," Energy, Elsevier, vol. 36(8), pages 5004-5015.
    15. Giorgia Oggioni & Yves Smeers & Elisabetta Allevi & Siegfried Schaible, 2012. "A Generalized Nash Equilibrium Model of Market Coupling in the European Power System," Networks and Spatial Economics, Springer, vol. 12(4), pages 503-560, December.
    16. Jiang, Sufan & Gao, Shan & Pan, Guangsheng & Zhao, Xin & Liu, Yu & Guo, Yasen & Wang, Sicheng, 2020. "A novel robust security constrained unit commitment model considering HVDC regulation," Applied Energy, Elsevier, vol. 278(C).
    17. Doumen, Sjoerd C. & Nguyen, Phuong & Kok, Koen, 2022. "Challenges for large-scale Local Electricity Market implementation reviewed from the stakeholder perspective," Renewable and Sustainable Energy Reviews, Elsevier, vol. 165(C).
    18. Wang, Tonghe & Hua, Haochen & Shi, Tianying & Wang, Rui & Sun, Yizhong & Naidoo, Pathmanathan, 2024. "A bi-level dispatch optimization of multi-microgrid considering green electricity consumption willingness under renewable portfolio standard policy," Applied Energy, Elsevier, vol. 356(C).
    19. Bin Ye & Jingjing Jiang & Lixin Miao & Ji Li & Yang Peng, 2015. "Innovative Carbon Allowance Allocation Policy for the Shenzhen Emission Trading Scheme in China," Sustainability, MDPI, vol. 8(1), pages 1-23, December.
    20. Canan Karatekin & Hakan elik, 2020. "The Effects of Renewable Energy Sources on the Structure of the Turkish Electricity Market," International Journal of Energy Economics and Policy, Econjournals, vol. 10(2), pages 64-70.
    21. Wang, Yi & Qiu, Dawei & Sun, Mingyang & Strbac, Goran & Gao, Zhiwei, 2023. "Secure energy management of multi-energy microgrid: A physical-informed safe reinforcement learning approach," Applied Energy, Elsevier, vol. 335(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:350:y:2023:i:c:s0306261923010681. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.