IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v12y2019i15p2891-d252180.html
   My bibliography  Save this article

A Q-Cube Framework of Reinforcement Learning Algorithm for Continuous Double Auction among Microgrids

Author

Listed:
  • Ning Wang

    (School of Electronics and Information Engineering, Tongji University, Shanghai 201804, China)

  • Weisheng Xu

    (School of Electronics and Information Engineering, Tongji University, Shanghai 201804, China)

  • Weihui Shao

    (Education Technology and Computing Center, Tongji University, Shanghai 200092, China)

  • Zhiyu Xu

    (School of Electronics and Information Engineering, Tongji University, Shanghai 201804, China)

Abstract

Decision-making of microgrids in the condition of a dynamic uncertain bidding environment has always been a significant subject of interest in the context of energy markets. The emerging application of reinforcement learning algorithms in energy markets provides solutions to this problem. In this paper, we investigate the potential of applying a Q-learning algorithm into a continuous double auction mechanism. By choosing a global supply and demand relationship as states and considering both bidding price and quantity as actions, a new Q-learning architecture is proposed to better reflect personalized bidding preferences and response to real-time market conditions. The application of battery energy storage system performs an alternative form of demand response by exerting potential capacity. A Q-cube framework is designed to describe the Q-value distribution iteration. Results from a case study on 14 microgrids in Guizhou Province, China indicate that the proposed Q-cube framework is capable of making rational bidding decisions and raising the microgrids’ profits.

Suggested Citation

  • Ning Wang & Weisheng Xu & Weihui Shao & Zhiyu Xu, 2019. "A Q-Cube Framework of Reinforcement Learning Algorithm for Continuous Double Auction among Microgrids," Energies, MDPI, vol. 12(15), pages 1-26, July.
    • Handle: RePEc:gam:jeners:v:12:y:2019:i:15:p:2891-:d:252180
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/12/15/2891/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1996-1073/12/15/2891/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Bui, Van-Hai & Hussain, Akhtar & Im, Yong-Hoon & Kim, Hak-Man, 2019. "An internal trading strategy for optimal energy management of combined cooling, heat and power in building microgrids," Applied Energy, Elsevier, vol. 239(C), pages 536-548.
    2. James Nicolaisen & Valentin Petrov & Leigh Tesfatsion, 2000. "Market Power and Efficiency in a Computational Electricity Market with Discriminatory Double-Auction Pricing," Computational Economics 0004005, University Library of Munich, Germany.
    3. Ning Wang & Weisheng Xu & Zhiyu Xu & Weihui Shao, 2018. "Peer-to-Peer Energy Trading among Microgrids with Multidimensional Willingness," Energies, MDPI, vol. 11(12), pages 1-22, November.
    4. Salehizadeh, Mohammad Reza & Soltaniyan, Salman, 2016. "Application of fuzzy Q-learning for electricity market modeling by considering renewable power penetration," Renewable and Sustainable Energy Reviews, Elsevier, vol. 56(C), pages 1172-1181.
    5. Zhang, Chenghua & Wu, Jianzhong & Zhou, Yue & Cheng, Meng & Long, Chao, 2018. "Peer-to-Peer energy trading in a Microgrid," Applied Energy, Elsevier, vol. 220(C), pages 1-12.
    6. Jian Wang & Qianggang Wang & Niancheng Zhou & Yuan Chi, 2017. "A Novel Electricity Transaction Mode of Microgrids Based on Blockchain and Continuous Double Auction," Energies, MDPI, vol. 10(12), pages 1-22, November.
    7. Rocchetta, R. & Bellani, L. & Compare, M. & Zio, E. & Patelli, E., 2019. "A reinforcement learning framework for optimal operation and maintenance of power grids," Applied Energy, Elsevier, vol. 241(C), pages 291-301.
    8. Long, Chao & Wu, Jianzhong & Zhou, Yue & Jenkins, Nick, 2018. "Peer-to-peer energy sharing through a two-stage aggregated battery control in a community Microgrid," Applied Energy, Elsevier, vol. 226(C), pages 261-276.
    9. Brida V. Mbuwir & Frederik Ruelens & Fred Spiessens & Geert Deconinck, 2017. "Battery Energy Management in a Microgrid Using Batch Reinforcement Learning," Energies, MDPI, vol. 10(11), pages 1-19, November.
    10. Derek W. Bunn and Fernando Oliveira, 2001. "An Application of Agent-based Simulation to the New Electricity Trading Arrangements of England and Wales," Computing in Economics and Finance 2001 93, Society for Computational Economics.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Harri Aaltonen & Seppo Sierla & Rakshith Subramanya & Valeriy Vyatkin, 2021. "A Simulation Environment for Training a Reinforcement Learning Agent Trading a Battery Storage," Energies, MDPI, vol. 14(17), pages 1-20, September.
    2. Wang, Longze & Liu, Jinxin & Yuan, Rongfang & Wu, Jing & Zhang, Delong & Zhang, Yan & Li, Meicheng, 2020. "Adaptive bidding strategy for real-time energy management in multi-energy market enhanced by blockchain," Applied Energy, Elsevier, vol. 279(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rodrigues, Stefane Dias & Garcia, Vinicius Jacques, 2023. "Transactive energy in microgrid communities: A systematic review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 171(C).
    2. Arnob Das & Susmita Datta Peu & Md. Abdul Mannan Akanda & Abu Reza Md. Towfiqul Islam, 2023. "Peer-to-Peer Energy Trading Pricing Mechanisms: Towards a Comprehensive Analysis of Energy and Network Service Pricing (NSP) Mechanisms to Get Sustainable Enviro-Economical Energy Sector," Energies, MDPI, vol. 16(5), pages 1-27, February.
    3. Park, Sung-Won & Zhang, Zhong & Li, Furong & Son, Sung-Yong, 2021. "Peer-to-peer trading-based efficient flexibility securing mechanism to support distribution system stability," Applied Energy, Elsevier, vol. 285(C).
    4. Esmat, Ayman & de Vos, Martijn & Ghiassi-Farrokhfal, Yashar & Palensky, Peter & Epema, Dick, 2021. "A novel decentralized platform for peer-to-peer energy trading market with blockchain technology," Applied Energy, Elsevier, vol. 282(PA).
    5. Capper, Timothy & Gorbatcheva, Anna & Mustafa, Mustafa A. & Bahloul, Mohamed & Schwidtal, Jan Marc & Chitchyan, Ruzanna & Andoni, Merlinda & Robu, Valentin & Montakhabi, Mehdi & Scott, Ian J. & Franci, 2022. "Peer-to-peer, community self-consumption, and transactive energy: A systematic literature review of local energy market models," Renewable and Sustainable Energy Reviews, Elsevier, vol. 162(C).
    6. Kuruseelan S & Vaithilingam C, 2019. "Peer-to-Peer Energy Trading of a Community Connected with an AC and DC Microgrid," Energies, MDPI, vol. 12(19), pages 1-15, September.
    7. Paladin, Andrea & Das, Ridoy & Wang, Yue & Ali, Zunaib & Kotter, Richard & Putrus, Ghanim & Turri, Roberto, 2021. "Micro market based optimisation framework for decentralised management of distributed flexibility assets," Renewable Energy, Elsevier, vol. 163(C), pages 1595-1611.
    8. Schwidtal, J.M. & Piccini, P. & Troncia, M. & Chitchyan, R. & Montakhabi, M. & Francis, C. & Gorbatcheva, A. & Capper, T. & Mustafa, M.A. & Andoni, M. & Robu, V. & Bahloul, M. & Scott, I.J. & Mbavarir, 2023. "Emerging business models in local energy markets: A systematic review of peer-to-peer, community self-consumption, and transactive energy models," Renewable and Sustainable Energy Reviews, Elsevier, vol. 179(C).
    9. Adamu Sani Yahaya & Nadeem Javaid & Fahad A. Alzahrani & Amjad Rehman & Ibrar Ullah & Affaf Shahid & Muhammad Shafiq, 2020. "Blockchain Based Sustainable Local Energy Trading Considering Home Energy Management and Demurrage Mechanism," Sustainability, MDPI, vol. 12(8), pages 1-28, April.
    10. Jin, Xiaolong & Wu, Qiuwei & Jia, Hongjie, 2020. "Local flexibility markets: Literature review on concepts, models and clearing methods," Applied Energy, Elsevier, vol. 261(C).
    11. Xiaoyu Lyu & Zhiyu Xu & Ning Wang & Min Fu & Weisheng Xu, 2019. "A Two-Layer Interactive Mechanism for Peer-to-Peer Energy Trading Among Virtual Power Plants," Energies, MDPI, vol. 12(19), pages 1-28, September.
    12. Bidan Zhang & Yang Du & Xiaoyang Chen & Eng Gee Lim & Lin Jiang & Ke Yan, 2022. "Potential Benefits for Residential Building with Photovoltaic Battery System Participation in Peer-to-Peer Energy Trading," Energies, MDPI, vol. 15(11), pages 1-21, May.
    13. Azim, M. Imran & Tushar, Wayes & Saha, Tapan K., 2021. "Cooperative negawatt P2P energy trading for low-voltage distribution networks," Applied Energy, Elsevier, vol. 299(C).
    14. Kirchhoff, Hannes & Strunz, Kai, 2019. "Key drivers for successful development of peer-to-peer microgrids for swarm electrification," Applied Energy, Elsevier, vol. 244(C), pages 46-62.
    15. Lyu, Cheng & Jia, Youwei & Xu, Zhao, 2021. "Fully decentralized peer-to-peer energy sharing framework for smart buildings with local battery system and aggregated electric vehicles," Applied Energy, Elsevier, vol. 299(C).
    16. Sensfuß, Frank & Ragwitz, Mario & Genoese, Massimo & Möst, Dominik, 2007. "Agent-based simulation of electricity markets: a literature review," Working Papers "Sustainability and Innovation" S5/2007, Fraunhofer Institute for Systems and Innovation Research (ISI).
    17. Esmaeili Aliabadi, Danial & Kaya, Murat & Sahin, Guvenc, 2017. "Competition, risk and learning in electricity markets: An agent-based simulation study," Applied Energy, Elsevier, vol. 195(C), pages 1000-1011.
    18. Miguel Carpintero-Rentería & David Santos-Martín & Josep M. Guerrero, 2019. "Microgrids Literature Review through a Layers Structure," Energies, MDPI, vol. 12(22), pages 1-22, November.
    19. Hutty, Timothy D. & Brown, Solomon, 2024. "P2P trading of heat and power via a continuous double auction," Applied Energy, Elsevier, vol. 369(C).
    20. Wang, Zibo & Yu, Xiaodan & Mu, Yunfei & Jia, Hongjie, 2020. "A distributed Peer-to-Peer energy transaction method for diversified prosumers in Urban Community Microgrid System," Applied Energy, Elsevier, vol. 260(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:12:y:2019:i:15:p:2891-:d:252180. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.