IDEAS home Printed from https://ideas.repec.org/a/eee/energy/v239y2022ipas0360544221021745.html
   My bibliography  Save this article

Bi-level stochastic real-time pricing model in multi-energy generation system: A reinforcement learning approach

Author

Listed:
  • Zhang, Li
  • Gao, Yan
  • Zhu, Hongbo
  • Tao, Li

Abstract

With the penetration of intermittent renewable energy sources, greater uncertainty has been brought to the power generation system, creating increased challenges to real-time pricing (RTP). Different from the existing studies, this paper aims to design an RTP strategy for the smart grid which integrates multi-energy generation on the supply side. Without loss of generality, small-scale distributed energy generation and power storage devices for users are also considered. Taking the interests of both supply and demand sides into consideration, a bilevel stochastic model for real-time demand response in the framework of Markov decision process (MDP) is formulated. The model well captures the interactive characters of both sides. Regarding the difficulty of collecting exact information from users in a centralized way in practice, a novel distributed online multi-agent reinforcement learning algorithm is proposed to solve the MDP model without acquisition of the transition probabilities. Through the information interaction between the upper and lower levels, the real-time electricity prices are decided adaptively, meanwhile, the optimal strategy of power supply and consumption is obtained. Simulation results demonstrate that the proposed pricing method and algorithm have a good performance in cutting peak and filling the valley and guarantee the benefits of both supply and demand.

Suggested Citation

  • Zhang, Li & Gao, Yan & Zhu, Hongbo & Tao, Li, 2022. "Bi-level stochastic real-time pricing model in multi-energy generation system: A reinforcement learning approach," Energy, Elsevier, vol. 239(PA).
    • Handle: RePEc:eee:energy:v:239:y:2022:i:pa:s0360544221021745
    DOI: 10.1016/j.energy.2021.121926
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0360544221021745
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.energy.2021.121926?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Zareen, N. & Mustafa, M.W. & Sultana, U. & Nadia, R. & Khattak, M.A., 2015. "Optimal real time cost-benefit based demand response with intermittent resources," Energy, Elsevier, vol. 90(P2), pages 1695-1706.
    2. Lu, Renzhi & Hong, Seung Ho & Zhang, Xiongfeng, 2018. "A Dynamic pricing demand response algorithm for smart grid: Reinforcement learning approach," Applied Energy, Elsevier, vol. 220(C), pages 220-230.
    3. Srinivasan, Dipti & Rajgarhia, Sanjana & Radhakrishnan, Bharat Menon & Sharma, Anurag & Khincha, H.P., 2017. "Game-Theory based dynamic pricing strategies for demand side management in smart grids," Energy, Elsevier, vol. 126(C), pages 132-143.
    4. Ma, Tengfei & Pei, Wei & Xiao, Hao & Kong, Li & Mu, Yunfei & Pu, Tianjiao, 2020. "The energy management strategies based on dynamic energy pricing for community integrated energy system considering the interactions between suppliers and users," Energy, Elsevier, vol. 211(C).
    5. Chang, Zihan & Zhang, Yang & Chen, Wenbo, 2019. "Electricity price prediction based on hybrid model of adam optimized LSTM neural network and wavelet transform," Energy, Elsevier, vol. 187(C).
    6. Chen, Zheng & Hu, Hengjie & Wu, Yitao & Zhang, Yuanjian & Li, Guang & Liu, Yonggang, 2020. "Stochastic model predictive control for energy management of power-split plug-in hybrid electric vehicles based on reinforcement learning," Energy, Elsevier, vol. 211(C).
    7. Wang, Fei & Ge, Xinxin & Yang, Peng & Li, Kangping & Mi, Zengqiang & Siano, Pierluigi & Duić, Neven, 2020. "Day-ahead optimal bidding and scheduling strategies for DER aggregator considering responsive uncertainty under real-time pricing," Energy, Elsevier, vol. 213(C).
    8. Hossain, Md Alamgir & Pota, Hemanshu Roy & Squartini, Stefano & Abdou, Ahmed Fathi, 2019. "Modified PSO algorithm for real-time energy management in grid-connected microgrids," Renewable Energy, Elsevier, vol. 136(C), pages 746-757.
    9. Yang, Ningkang & Han, Lijin & Xiang, Changle & Liu, Hui & Li, Xunmin, 2021. "An indirect reinforcement learning based real-time energy management strategy via high-order Markov Chain model for a hybrid electric vehicle," Energy, Elsevier, vol. 236(C).
    10. Kuznetsova, Elizaveta & Li, Yan-Fu & Ruiz, Carlos & Zio, Enrico & Ault, Graham & Bell, Keith, 2013. "Reinforcement learning for microgrid energy management," Energy, Elsevier, vol. 59(C), pages 133-146.
    11. Zhao, Xueyuan & Gao, Weijun & Qian, Fanyue & Ge, Jian, 2021. "Electricity cost comparison of dynamic pricing model based on load forecasting in home energy management system," Energy, Elsevier, vol. 229(C).
    12. Favuzza, S. & Galioto, G. & Ippolito, M.G. & Massaro, F. & Milazzo, F. & Pecoraro, G. & Riva Sanseverino, E. & Telaretti, E., 2015. "Real-time pricing for aggregates energy resources in the Italian energy market," Energy, Elsevier, vol. 87(C), pages 251-258.
    13. Kaygusuz, Asim, 2019. "Closed loop elastic demand control by dynamic energy pricing in smart grids," Energy, Elsevier, vol. 176(C), pages 596-603.
    14. Javadi, Mohammad Sadegh & Gough, Matthew & Lotfi, Mohamed & Esmaeel Nezhad, Ali & Santos, Sérgio F. & Catalão, João P.S., 2020. "Optimal self-scheduling of home energy management system in the presence of photovoltaic power generation and batteries," Energy, Elsevier, vol. 210(C).
    15. Nyong-Bassey, Bassey Etim & Giaouris, Damian & Patsios, Charalampos & Papadopoulou, Simira & Papadopoulos, Athanasios I. & Walker, Sara & Voutetakis, Spyros & Seferlis, Panos & Gadoue, Shady, 2020. "Reinforcement learning based adaptive power pinch analysis for energy management of stand-alone hybrid energy storage systems considering uncertainty," Energy, Elsevier, vol. 193(C).
    16. Kong, Xiangyu & Kong, Deqian & Yao, Jingtao & Bai, Linquan & Xiao, Jie, 2020. "Online pricing of demand response based on long short-term memory and reinforcement learning," Applied Energy, Elsevier, vol. 271(C).
    17. Wang, Yongli & Huang, Yujing & Wang, Yudong & Zeng, Ming & Yu, Haiyang & Li, Fang & Zhang, Fuli, 2018. "Optimal scheduling of the RIES considering time-based demand response programs with energy price," Energy, Elsevier, vol. 164(C), pages 773-793.
    18. Li, Yuanyuan & Li, Junxiang & He, Jianjia & Zhang, Shuyuan, 2021. "The real-time pricing optimization model of smart grid based on the utility function of the logistic function," Energy, Elsevier, vol. 224(C).
    19. Motalleb, Mahdi & Annaswamy, Anuradha & Ghorbani, Reza, 2018. "A real-time demand response market through a repeated incomplete-information game," Energy, Elsevier, vol. 143(C), pages 424-438.
    20. Scott Fay & Jinhong Xie, 2010. "The Economics of Buyer Uncertainty: Advance Selling vs. Probabilistic Selling," Marketing Science, INFORMS, vol. 29(6), pages 1040-1057, 11-12.
    21. Ghasemi, Ahmad & Jamshidi Monfared, Houman & Loni, Abdolah & Marzband, Mousa, 2021. "CVaR-based retail electricity pricing in day-ahead scheduling of microgrids," Energy, Elsevier, vol. 227(C).
    22. Dehnavi, Ehsan & Abdi, Hamdi, 2016. "Optimal pricing in time of use demand response by integrating with dynamic economic dispatch problem," Energy, Elsevier, vol. 109(C), pages 1086-1094.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Sabarathinam Srinivasan & Suresh Kumarasamy & Zacharias E. Andreadakis & Pedro G. Lind, 2023. "Artificial Intelligence and Mathematical Models of Power Grids Driven by Renewable Energy Sources: A Survey," Energies, MDPI, vol. 16(14), pages 1-56, July.
    2. Li, Ningning & Gao, Yan, 2023. "Real-time pricing based on convex hull method for smart grid with multiple generating units," Energy, Elsevier, vol. 285(C).
    3. Qiuyi Hong & Fanlin Meng & Jian Liu, 2023. "Customised Multi-Energy Pricing: Model and Solutions," Energies, MDPI, vol. 16(4), pages 1-31, February.
    4. Fan, Lurong & Wang, Binyu & Song, Xiaoling, 2023. "An authority-enterprise equilibrium differentiated subsidy mechanism for promoting coalbed methane extraction in multiple coal seams," Energy, Elsevier, vol. 263(PA).
    5. Zhou, Yanting & Ma, Zhongjing & Zhang, Jinhui & Zou, Suli, 2022. "Data-driven stochastic energy management of multi energy system using deep reinforcement learning," Energy, Elsevier, vol. 261(PA).
    6. Wang, Yudong & Hu, Junjie, 2023. "Two-stage energy management method of integrated energy system considering pre-transaction behavior of energy service provider and users," Energy, Elsevier, vol. 271(C).
    7. Huang, Qian & Xu, Jiuping, 2023. "Carbon tax revenue recycling for biomass/coal co-firing using Stackelberg game: A case study of Jiangsu province, China," Energy, Elsevier, vol. 272(C).
    8. Feng, Wenxiu & Ruiz, Carlos, 2023. "Risk management of energy communities with hydrogen production and storage technologies," Applied Energy, Elsevier, vol. 348(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zhu, Ziqing & Hu, Ze & Chan, Ka Wing & Bu, Siqi & Zhou, Bin & Xia, Shiwei, 2023. "Reinforcement learning in deregulated energy market: A comprehensive review," Applied Energy, Elsevier, vol. 329(C).
    2. Cai, Qiran & Xu, Qingyang & Qing, Jing & Shi, Gang & Liang, Qiao-Mei, 2022. "Promoting wind and photovoltaics renewable energy integration through demand response: Dynamic pricing mechanism design and economic analysis for smart residential communities," Energy, Elsevier, vol. 261(PB).
    3. Lee, Hyun-Suk, 2024. "Automated tariff design for energy supply–demand matching based on Bayesian optimization: Technical framework and policy implications," Energy Policy, Elsevier, vol. 188(C).
    4. Sabarathinam Srinivasan & Suresh Kumarasamy & Zacharias E. Andreadakis & Pedro G. Lind, 2023. "Artificial Intelligence and Mathematical Models of Power Grids Driven by Renewable Energy Sources: A Survey," Energies, MDPI, vol. 16(14), pages 1-56, July.
    5. Zhang, Xiongfeng & Lu, Renzhi & Jiang, Junhui & Hong, Seung Ho & Song, Won Seok, 2021. "Testbed implementation of reinforcement learning-based demand response energy management system," Applied Energy, Elsevier, vol. 297(C).
    6. Xu, Fangyuan & Zhu, Weidong & Wang, Yi Fei & Lai, Chun Sing & Yuan, Haoliang & Zhao, Yujia & Guo, Siming & Fu, Zhengxin, 2022. "A new deregulated demand response scheme for load over-shifting city in regulated power market," Applied Energy, Elsevier, vol. 311(C).
    7. Lu, Renzhi & Bai, Ruichang & Ding, Yuemin & Wei, Min & Jiang, Junhui & Sun, Mingyang & Xiao, Feng & Zhang, Hai-Tao, 2021. "A hybrid deep learning-based online energy management scheme for industrial microgrid," Applied Energy, Elsevier, vol. 304(C).
    8. Pinciroli, Luca & Baraldi, Piero & Compare, Michele & Zio, Enrico, 2023. "Optimal operation and maintenance of energy storage systems in grid-connected microgrids by deep reinforcement learning," Applied Energy, Elsevier, vol. 352(C).
    9. Mansouri, S.A. & Ahmarinejad, A. & Nematbakhsh, E. & Javadi, M.S. & Esmaeel Nezhad, A. & Catalão, J.P.S., 2022. "A sustainable framework for multi-microgrids energy management in automated distribution network by considering smart homes and high penetration of renewable energy resources," Energy, Elsevier, vol. 245(C).
    10. Huang, Pei & Sun, Yongjun, 2019. "A collaborative demand control of nearly zero energy buildings in response to dynamic pricing for performance improvements at cluster level," Energy, Elsevier, vol. 174(C), pages 911-921.
    11. Zhao, Liyuan & Yang, Ting & Li, Wei & Zomaya, Albert Y., 2022. "Deep reinforcement learning-based joint load scheduling for household multi-energy system," Applied Energy, Elsevier, vol. 324(C).
    12. Zheng, Lingwei & Wu, Hao & Guo, Siqi & Sun, Xinyu, 2023. "Real-time dispatch of an integrated energy system based on multi-stage reinforcement learning with an improved action-choosing strategy," Energy, Elsevier, vol. 277(C).
    13. Ma, Siyu & Liu, Hui & Wang, Ni & Huang, Lidong & Goh, Hui Hwang, 2023. "Incentive-based demand response under incomplete information based on the deep deterministic policy gradient," Applied Energy, Elsevier, vol. 351(C).
    14. Sarker, Eity & Seyedmahmoudian, Mehdi & Jamei, Elmira & Horan, Ben & Stojcevski, Alex, 2020. "Optimal management of home loads with renewable energy integration and demand response strategy," Energy, Elsevier, vol. 210(C).
    15. Charwand, Mansour & Gitizadeh, Mohsen, 2018. "Optimal TOU tariff design using robust intuitionistic fuzzy divergence based thresholding," Energy, Elsevier, vol. 147(C), pages 655-662.
    16. Grace Muriithi & Sunetra Chowdhury, 2021. "Optimal Energy Management of a Grid-Tied Solar PV-Battery Microgrid: A Reinforcement Learning Approach," Energies, MDPI, vol. 14(9), pages 1-24, May.
    17. Marouane Adnane & Ahmed Khoumsi & João Pedro F. Trovão, 2023. "Efficient Management of Energy Consumption of Electric Vehicles Using Machine Learning—A Systematic and Comprehensive Survey," Energies, MDPI, vol. 16(13), pages 1-39, June.
    18. Xie, Jiahan & Ajagekar, Akshay & You, Fengqi, 2023. "Multi-Agent attention-based deep reinforcement learning for demand response in grid-responsive buildings," Applied Energy, Elsevier, vol. 342(C).
    19. Yamashita, Daniela Yassuda & Vechiu, Ionel & Gaubert, Jean-Paul, 2020. "A review of hierarchical control for building microgrids," Renewable and Sustainable Energy Reviews, Elsevier, vol. 118(C).
    20. Eduardo J. Salazar & Mauro Jurado & Mauricio E. Samper, 2023. "Reinforcement Learning-Based Pricing and Incentive Strategy for Demand Response in Smart Grids," Energies, MDPI, vol. 16(3), pages 1-33, February.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:energy:v:239:y:2022:i:pa:s0360544221021745. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.journals.elsevier.com/energy .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.