IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2004.05940.html
   My bibliography  Save this paper

A Deep Reinforcement Learning Framework for Continuous Intraday Market Bidding

Author

Listed:
  • Ioannis Boukas
  • Damien Ernst
  • Thibaut Th'eate
  • Adrien Bolland
  • Alexandre Huynen
  • Martin Buchwald
  • Christelle Wynants
  • Bertrand Corn'elusse

Abstract

The large integration of variable energy resources is expected to shift a large part of the energy exchanges closer to real-time, where more accurate forecasts are available. In this context, the short-term electricity markets and in particular the intraday market are considered a suitable trading floor for these exchanges to occur. A key component for the successful renewable energy sources integration is the usage of energy storage. In this paper, we propose a novel modelling framework for the strategic participation of energy storage in the European continuous intraday market where exchanges occur through a centralized order book. The goal of the storage device operator is the maximization of the profits received over the entire trading horizon, while taking into account the operational constraints of the unit. The sequential decision-making problem of trading in the intraday market is modelled as a Markov Decision Process. An asynchronous distributed version of the fitted Q iteration algorithm is chosen for solving this problem due to its sample efficiency. The large and variable number of the existing orders in the order book motivates the use of high-level actions and an alternative state representation. Historical data are used for the generation of a large number of artificial trajectories in order to address exploration issues during the learning process. The resulting policy is back-tested and compared against a benchmark strategy that is the current industrial standard. Results indicate that the agent converges to a policy that achieves in average higher total revenues than the benchmark strategy.

Suggested Citation

  • Ioannis Boukas & Damien Ernst & Thibaut Th'eate & Adrien Bolland & Alexandre Huynen & Martin Buchwald & Christelle Wynants & Bertrand Corn'elusse, 2020. "A Deep Reinforcement Learning Framework for Continuous Intraday Market Bidding," Papers 2004.05940, arXiv.org.
    • Handle: RePEc:arx:papers:2004.05940
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2004.05940
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Arthur Henriot, 2014. "Market Design with Centralized Wind Power Management: Handling Low-predictability in Intraday Markets," The Energy Journal, International Association for Energy Economics, vol. 0(Number 1).
    2. repec:diw:diwwpp:dp1544 is not listed on IDEAS
    3. repec:diw:diwwpp:dp1162 is not listed on IDEAS
    4. Fatih Karanfil and Yuanjing Li, 2017. "The Role of Continuous Intraday Electricity Markets: The Integration of Large-Share Wind Power Generation in Denmark," The Energy Journal, International Association for Energy Economics, vol. 0(Number 2).
    5. Fatih Karanfil and Yuanjing Li, 2017. "The Role of Continuous Intraday Electricity Markets: The Integration of Large-Share Wind Power Generation in Denmark," The Energy Journal, International Association for Energy Economics, vol. 0(Number 2).
    6. Fleten, Stein-Erik & Kristoffersen, Trine Krogh, 2007. "Stochastic programming for optimizing bidding strategies of a Nordic hydropower producer," European Journal of Operational Research, Elsevier, vol. 181(2), pages 916-928, September.
    7. Le, Hong Lam & Ilea, Valentin & Bovo, Cristian, 2019. "Integrated European intra-day electricity market: Rules, modeling and analysis," Applied Energy, Elsevier, vol. 238(C), pages 258-273.
    8. Scharff, Richard & Amelin, Mikael, 2016. "Trading behaviour on the continuous intraday market Elbas," Energy Policy, Elsevier, vol. 88(C), pages 544-557.
    9. Karsten Neuhoff & Nolan Ritter & Aymen Salah-Abou-El-Enien & Philippe Vassilopoulos, 2016. "Intraday Markets for Power: Discretizing the Continuous Trading?," Discussion Papers of DIW Berlin 1544, DIW Berlin, German Institute for Economic Research.
    10. René Aïd & P. Gruet & H. Pham, 2016. "An optimal trading problem in intraday electricity markets," Post-Print hal-01609481, HAL.
    11. Boomsma, Trine Krogh & Juul, Nina & Fleten, Stein-Erik, 2014. "Bidding in sequential electricity markets: The Nordic case," European Journal of Operational Research, Elsevier, vol. 238(3), pages 797-809.
    12. Jochen Gönsch & Michael Hassler, 2016. "Sell or store? An ADP approach to marketing renewable energy," OR Spectrum: Quantitative Approaches in Management, Springer;Gesellschaft für Operations Research e.V., vol. 38(3), pages 633-660, July.
    13. Pandžić, Hrvoje & Morales, Juan M. & Conejo, Antonio J. & Kuzle, Igor, 2013. "Offering model for a virtual power plant based on stochastic programming," Applied Energy, Elsevier, vol. 105(C), pages 282-292.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Priyanka Shinde & Ioannis Boukas & David Radu & Miguel Manuel de Villena & Mikael Amelin, 2021. "Analyzing Trade in Continuous Intra-Day Electricity Market: An Agent-Based Modeling Approach," Energies, MDPI, vol. 14(13), pages 1-31, June.
    2. Supriya Bajpai, 2021. "Application of deep reinforcement learning for Indian stock trading automation," Papers 2106.16088, arXiv.org.
    3. Adrian Millea, 2021. "Deep Reinforcement Learning for Trading—A Critical Survey," Data, MDPI, vol. 6(11), pages 1-25, November.
    4. Karush Suri & Xiao Qi Shi & Konstantinos Plataniotis & Yuri Lawryshyn, 2021. "TradeR: Practical Deep Hierarchical Reinforcement Learning for Trade Execution," Papers 2104.00620, arXiv.org.
    5. Benedikt Finnah, 2022. "Optimal bidding functions for renewable energies in sequential electricity markets," OR Spectrum: Quantitative Approaches in Management, Springer;Gesellschaft für Operations Research e.V., vol. 44(1), pages 1-27, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rainer Baule & Michael Naumann, 2022. "Flexible Short-Term Electricity Certificates—An Analysis of Trading Strategies on the Continuous Intraday Market," Energies, MDPI, vol. 15(17), pages 1-28, August.
    2. Hu, Xiao & Jaraitė, Jūratė & Kažukauskas, Andrius, 2021. "The effects of wind power on electricity markets: A case study of the Swedish intraday market," Energy Economics, Elsevier, vol. 96(C).
    3. Spodniak, Petr & Ollikka, Kimmo & Honkapuro, Samuli, 2019. "The Relevance of Wholesale Electricity Market Places: The Nordic Case," Working Papers 126, VATT Institute for Economic Research.
    4. Spodniak, Petr & Ollikka, Kimmo & Honkapuro, Samuli, 2021. "The impact of wind power and electricity demand on the relevance of different short-term electricity markets: The Nordic case," Applied Energy, Elsevier, vol. 283(C).
    5. Benedikt Finnah, 2022. "Optimal bidding functions for renewable energies in sequential electricity markets," OR Spectrum: Quantitative Approaches in Management, Springer;Gesellschaft für Operations Research e.V., vol. 44(1), pages 1-27, March.
    6. Thomas Kuppelwieser & David Wozabal, 2023. "Intraday power trading: toward an arms race in weather forecasting?," OR Spectrum: Quantitative Approaches in Management, Springer;Gesellschaft für Operations Research e.V., vol. 45(1), pages 57-83, March.
    7. Micha{l} Narajewski & Florian Ziel, 2020. "Ensemble Forecasting for Intraday Electricity Prices: Simulating Trajectories," Papers 2005.01365, arXiv.org, revised Aug 2020.
    8. Russo, Marianna & Kraft, Emil & Bertsch, Valentin & Keles, Dogan, 2022. "Short-term risk management of electricity retailers under rising shares of decentralized solar generation," Energy Economics, Elsevier, vol. 109(C).
    9. Kraft, Emil & Russo, Marianna & Keles, Dogan & Bertsch, Valentin, 2023. "Stochastic optimization of trading strategies in sequential electricity markets," European Journal of Operational Research, Elsevier, vol. 308(1), pages 400-421.
    10. Olivier F'eron & Peter Tankov & Laura Tinsi, 2020. "Price formation and optimal trading in intraday electricity markets," Papers 2009.04786, arXiv.org, revised Jun 2021.
    11. Hohl, Cody & Lo Prete, Chiara & Radhakrishnan, Ashish & Webster, Mort, 2023. "Intraday markets, wind integration and uplift payments in a regional U.S. power system," Energy Policy, Elsevier, vol. 175(C).
    12. Narajewski, Michał & Ziel, Florian, 2020. "Ensemble forecasting for intraday electricity prices: Simulating trajectories," Applied Energy, Elsevier, vol. 279(C).
    13. Rintamäki, Tuomas & Siddiqui, Afzal S. & Salo, Ahti, 2020. "Strategic offering of a flexible producer in day-ahead and intraday power markets," European Journal of Operational Research, Elsevier, vol. 284(3), pages 1136-1153.
    14. Knaut, Andreas & Paschmann, Martin, 2017. "Decoding Restricted Participation in Sequential Electricity Markets," EWI Working Papers 2017-5, Energiewirtschaftliches Institut an der Universitaet zu Koeln (EWI), revised 31 Aug 2017.
    15. Juraj Čurpek, 2019. "Time Evolution of Hurst Exponent: Czech Wholesale Electricity Market Study," European Financial and Accounting Journal, Prague University of Economics and Business, vol. 2019(3), pages 25-44.
    16. Ocker, Fabian & Jaenisch, Vincent, 2020. "The way towards European electricity intraday auctions – Status quo and future developments," Energy Policy, Elsevier, vol. 145(C).
    17. Laureen Deman & Quentin Boucher & Sonia Djebali & Guillaume Guerard & C?dric Clastres, 2023. "Bidding strategy of storage hydropower plants in reserve markets," ECONOMICS AND POLICY OF ENERGY AND THE ENVIRONMENT, FrancoAngeli Editore, vol. 2023(2), pages 77-101.
    18. Richstein, Jörn C. & Lorenz, Casimir & Neuhoff, Karsten, 2020. "An auction story: How simple bids struggle with uncertainty," Energy Economics, Elsevier, vol. 89(C).
    19. Ningyuan Chen & Steven Kou & Chun Wang, 2018. "A Partitioning Algorithm for Markov Decision Processes with Applications to Market Microstructure," Management Science, INFORMS, vol. 64(2), pages 784-803, February.
    20. Russo, Marianna & Bertsch, Valentin, 2020. "A looming revolution: Implications of self-generation for the risk exposure of retailers," Energy Economics, Elsevier, vol. 92(C).

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2004.05940. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.