IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v336y2023ics0306261923001770.html
   My bibliography  Save this article

Reinforcement Learning-Based Bi-Level strategic bidding model of Gas-fired unit in integrated electricity and natural gas markets preventing market manipulation

Author

Listed:
  • Ren, Kezheng
  • Liu, Jun
  • Liu, Xinglei
  • Nie, Yongxin

Abstract

Due to its efficient operation and environment-friendly characteristic, gas-fired unit (GFU) plays a more and more important role in electric power systems and natural gas systems. To investigate the performance of GFU's participation in integrated electricity and natural gas markets, a bi-level strategic bidding model considering price and quantity factors is proposed. With the increasing participation of consumers in electricity markets, demand response (DR) management is implemented in the electricity market clearing process and user comfort level (UCL) is considered in the market clearing model. Since GFU participates in both the electricity market and the natural gas market, a local marginal price penalty (LMPP) variable is defined in this paper to prevent potential market manipulation (MM) of GFU. Then a modified reinforcement learning (RL)-based method is proposed to solve the model, combining deep deterministic policy gradient (DDPG) algorithm with autocorrelated noise. Test results on an integrated electricity-gas system show that the proposed method can reflect the strategic behaviors of GFU effectively. The proposed method has better performance than traditional DDPG algorithm with Gaussian noise and the Deep Q-Network (DQN) algorithm. And electricity markets with LMPP can save about 3.03% in generation cost by preventing MM of GFU.

Suggested Citation

  • Ren, Kezheng & Liu, Jun & Liu, Xinglei & Nie, Yongxin, 2023. "Reinforcement Learning-Based Bi-Level strategic bidding model of Gas-fired unit in integrated electricity and natural gas markets preventing market manipulation," Applied Energy, Elsevier, vol. 336(C).
    • Handle: RePEc:eee:appene:v:336:y:2023:i:c:s0306261923001770
    DOI: 10.1016/j.apenergy.2023.120813
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261923001770
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2023.120813?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Birge, John R. & Hortaçsu, Ali & Mercadal, Ignacia & Pavlin, J. Michael, 2018. "Limits to arbitrage in electricity markets: A case study of MISO," Energy Economics, Elsevier, vol. 75(C), pages 518-533.
    2. David Pozo & Enzo Sauma & Javier Contreras, 2017. "Basic theoretical foundations and insights on bilevel models and their applications to power systems," Annals of Operations Research, Springer, vol. 254(1), pages 303-334, July.
    3. Hopkins, Caroline A., 2020. "Convergence bids and market manipulation in the California electricity market," Energy Economics, Elsevier, vol. 89(C).
    4. Li, Jiawen & Yu, Tao & Zhang, Xiaoshun & Li, Fusheng & Lin, Dan & Zhu, Hanxin, 2021. "Efficient experience replay based deep deterministic policy gradient for AGC dispatch in integrated energy system," Applied Energy, Elsevier, vol. 285(C).
    5. Harrold, Daniel J.B. & Cao, Jun & Fan, Zhong, 2022. "Renewable energy integration and microgrid energy trading using multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 318(C).
    6. Kirchem, Dana & Lynch, Muireann Á. & Bertsch, Valentin & Casey, Eoin, 2020. "Modelling demand response with process models and energy systems models: Potential applications for wastewater treatment within the energy-water nexus," Applied Energy, Elsevier, vol. 260(C).
    7. Coelho, António & Iria, José & Soares, Filipe, 2021. "Network-secure bidding optimization of aggregators of multi-energy systems in electricity, gas, and carbon markets," Applied Energy, Elsevier, vol. 301(C).
    8. Tian, Ruijie & Zhang, Qi & Wang, Ge & Li, Hailong & Chen, Siyuan & Li, Yan & Tian, Yulu, 2017. "Study on the promotion of natural gas-fired electricity with energy market reform in China using a dynamic game-theoretic model," Applied Energy, Elsevier, vol. 185(P2), pages 1832-1839.
    9. Liu, Xinglei & Liu, Jun & Ren, Kezheng & Liu, Xiaoming & Liu, Jiacheng, 2022. "An integrated fuzzy multi-energy transaction evaluation approach for energy internet markets considering judgement credibility and variable rough precision," Energy, Elsevier, vol. 261(PB).
    10. DE WOLF, Daniel & SMEERS, Yves, 2000. "The gas transmission problem solved by an extension of the simplex algorithm," LIDAM Reprints CORE 1489, Université catholique de Louvain, Center for Operations Research and Econometrics (CORE).
    11. Wang, Jianxiao & Zhong, Haiwang & Ma, Ziming & Xia, Qing & Kang, Chongqing, 2017. "Review and prospect of integrated demand response in the multi-energy system," Applied Energy, Elsevier, vol. 202(C), pages 772-782.
    12. Daniel De Wolf & Yves Smeers, 2000. "The Gas Transmission Problem Solved by an Extension of the Simplex Algorithm," Management Science, INFORMS, vol. 46(11), pages 1454-1465, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Zeng, Huibin & Shao, Bilin & Dai, Hongbin & Tian, Ning & Zhao, Wei, 2023. "Incentive-based demand response strategies for natural gas considering carbon emissions and load volatility," Applied Energy, Elsevier, vol. 348(C).
    2. Wang, Yi & Yang, Zhifang & Yu, Juan & Liu, Junyong, 2023. "An optimization-based partial marginal pricing method to reduce excessive consumer payment in electricity markets," Applied Energy, Elsevier, vol. 352(C).
    3. Ren, Kezheng & Liu, Jun & Wu, Zeyang & Liu, Xinglei & Nie, Yongxin & Xu, Haitao, 2024. "A data-driven DRL-based home energy management system optimization framework considering uncertain household parameters," Applied Energy, Elsevier, vol. 355(C).
    4. Guo, Xiaopeng & Zhang, Xinyue & Zhang, Xingping, 2024. "Incentive-oriented power‑carbon emissions trading-tradable green certificate integrated market mechanisms using multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 357(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Liu, Tianhao & Tian, Jun & Zhu, Hongyu & Goh, Hui Hwang & Liu, Hui & Wu, Thomas & Zhang, Dongdong, 2023. "Key technologies and developments of multi-energy system: Three-layer framework, modelling and optimisation," Energy, Elsevier, vol. 277(C).
    2. Bao, Minglei & Hui, Hengyu & Ding, Yi & Sun, Xiaocong & Zheng, Chenghang & Gao, Xiang, 2023. "An efficient framework for exploiting operational flexibility of load energy hubs in risk management of integrated electricity-gas systems," Applied Energy, Elsevier, vol. 338(C).
    3. Beyza, Jesus & Ruiz-Paredes, Hector F. & Garcia-Paricio, Eduardo & Yusta, Jose M., 2020. "Assessing the criticality of interdependent power and gas systems using complex networks and load flow techniques," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 540(C).
    4. Daniel de Wolf, 2017. "Mathematical Properties of Formulations of the Gas Transmission Problem," Post-Print halshs-02396747, HAL.
    5. Mengying Xue & Tianhu Deng & Zuo‐Jun Max Shen, 2019. "Optimizing natural gas pipeline transmission with nonuniform elevation: A new initialization approach," Naval Research Logistics (NRL), John Wiley & Sons, vol. 66(7), pages 547-564, October.
    6. Conrado Borraz-Sánchez & Dag Haugland, 2013. "Optimization methods for pipeline transportation of natural gas with variable specific gravity and compressibility," TOP: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 21(3), pages 524-541, October.
    7. Zhou, Li & Liao, Zuwei & Wang, Jingdai & Jiang, Binbo & Yang, Yongrong & Du, Wenli, 2015. "Energy configuration and operation optimization of refinery fuel gas networks," Applied Energy, Elsevier, vol. 139(C), pages 365-375.
    8. Massol, O., 2011. "A cost function for the natural gas transmission industry: further considerations," Working Papers 1464, Department of Economics, City University London.
    9. Hong, Sung-Pil & Kim, Taegyoon & Lee, Subin, 2019. "A precision pump schedule optimization for the water supply networks with small buffers," Omega, Elsevier, vol. 82(C), pages 24-37.
    10. Dieckhoener, Caroline, 2010. "Simulating security of supply effects of the Nabucco and South Stream projects for the European natural gas market," EWI Working Papers 2010-7, Energiewirtschaftliches Institut an der Universitaet zu Koeln (EWI), revised 21 Jan 2012.
    11. Song, Chenhui & Xiao, Jun & Zu, Guoqiang & Hao, Ziyuan & Zhang, Xinsong, 2021. "Security region of natural gas pipeline network system: Concept, method and application," Energy, Elsevier, vol. 217(C).
    12. Daniel de Wolf & Yves Smeers, 2021. "Generalized derivatives of the optimal value of a linear program with respect to matrix coefficients," Post-Print halshs-02396708, HAL.
    13. Shabanpour-Haghighi, Amin & Seifi, Ali Reza, 2015. "Multi-objective operation management of a multi-carrier energy system," Energy, Elsevier, vol. 88(C), pages 430-442.
    14. Jin, Xiaolong & Mu, Yunfei & Jia, Hongjie & Wu, Jianzhong & Xu, Xiandong & Yu, Xiaodan, 2016. "Optimal day-ahead scheduling of integrated urban energy systems," Applied Energy, Elsevier, vol. 180(C), pages 1-13.
    15. Tian, Xingtao & Lin, Xiaojie & Zhong, Wei & Zhou, Yi, 2023. "Analytical sensitivity analysis of radial natural gas networks," Energy, Elsevier, vol. 263(PC).
    16. R. Navarro & H. Rojas & Izabelly S. De Oliveira & J. E. Luyo & Y. P. Molina, 2022. "Optimization Model for the Integration of the Electric System and Gas Network: Peruvian Case," Energies, MDPI, vol. 15(10), pages 1-32, May.
    17. Frédéric Babonneau & Yurii Nesterov & Jean-Philippe Vial, 2012. "Design and Operations of Gas Transmission Networks," Operations Research, INFORMS, vol. 60(1), pages 34-47, February.
    18. Lu, Qing & Lü, Shuaikang & Leng, Yajun, 2019. "A Nash-Stackelberg game approach in regional energy market considering users’ integrated demand response," Energy, Elsevier, vol. 175(C), pages 456-470.
    19. Johannes Thürauf, 2022. "Deciding the feasibility of a booking in the European gas market is coNP-hard," Annals of Operations Research, Springer, vol. 318(1), pages 591-618, November.
    20. Yao, Wenliang & Wang, Chengfu & Yang, Ming & Wang, Kang & Dong, Xiaoming & Zhang, Zhenwei, 2023. "A tri-layer decision-making framework for IES considering the interaction of integrated demand response and multi-energy market clearing," Applied Energy, Elsevier, vol. 342(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:336:y:2023:i:c:s0306261923001770. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.