IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v14y2021i24p8365-d700404.html
   My bibliography  Save this article

Co-Optimizing Battery Storage for Energy Arbitrage and Frequency Regulation in Real-Time Markets Using Deep Reinforcement Learning

Author

Listed:
  • Yushen Miao

    (James Watt School of Engineering, University of Glasgow, Glasgow G12 8QQ, UK)

  • Tianyi Chen

    (James Watt School of Engineering, University of Glasgow, Glasgow G12 8QQ, UK)

  • Shengrong Bu

    (Department of Engineering, Brock University, St. Catharines, ON L2S 3A1, Canada)

  • Hao Liang

    (Department of Electrical and Computer Engineering, University of Alberta, Edmonton, AB T6G 2R3, Canada)

  • Zhu Han

    (Department of Electrical and Computer Engineering, University of Houston, Houston, TX 77204, USA)

Abstract

Battery energy storage systems (BESSs) play a critical role in eliminating uncertainties associated with renewable energy generation, to maintain stability and improve flexibility of power networks. In this paper, a BESS is used to provide energy arbitrage (EA) and frequency regulation (FR) services simultaneously to maximize its total revenue within the physical constraints. The EA and FR actions are taken at different timescales. The multitimescale problem is formulated as two nested Markov decision process (MDP) submodels. The problem is a complex decision-making problem with enormous high-dimensional data and uncertainty (e.g., the price of the electricity). Therefore, a novel co-optimization scheme is proposed to handle the multitimescale problem, and also coordinate EA and FR services. A triplet deep deterministic policy gradient with exploration noise decay (TDD–ND) approach is used to obtain the optimal policy at each timescale. Simulations are conducted with real-time electricity prices and regulation signals data from the American PJM regulation market. The simulation results show that the proposed approach performs better than other studied policies in literature.

Suggested Citation

  • Yushen Miao & Tianyi Chen & Shengrong Bu & Hao Liang & Zhu Han, 2021. "Co-Optimizing Battery Storage for Energy Arbitrage and Frequency Regulation in Real-Time Markets Using Deep Reinforcement Learning," Energies, MDPI, vol. 14(24), pages 1-17, December.
    • Handle: RePEc:gam:jeners:v:14:y:2021:i:24:p:8365-:d:700404
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/14/24/8365/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1996-1073/14/24/8365/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Walawalkar, Rahul & Apt, Jay & Mancini, Rick, 2007. "Economics of electric energy storage for energy arbitrage and regulation in New York," Energy Policy, Elsevier, vol. 35(4), pages 2558-2568, April.
    2. Sioshansi, Ramteen & Denholm, Paul & Jenkin, Thomas & Weiss, Jurgen, 2009. "Estimating the value of electricity storage in PJM: Arbitrage and some welfare effects," Energy Economics, Elsevier, vol. 31(2), pages 269-277, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Alba Leduchowicz-Municio & Miguel Edgar Morales Udaeta & André Luiz Veiga Gimenes & Tuo Ji & Victor Baiochi Riboldi, 2022. "Socio-Environmental Evaluation of MV Commercial Time-Shift Application Based on Battery Energy Storage Systems," Energies, MDPI, vol. 15(14), pages 1-21, July.
    2. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Pedro Crespo Del Granado & Stein Wallace & Zhan Pang, 2016. "The impact of wind uncertainty on the strategic valuation of distributed electricity storage," Computational Management Science, Springer, vol. 13(1), pages 5-27, January.
    2. McConnell, Dylan & Forcey, Tim & Sandiford, Mike, 2015. "Estimating the value of electricity storage in an energy-only wholesale market," Applied Energy, Elsevier, vol. 159(C), pages 422-432.
    3. Dusonchet, L. & Favuzza, S. & Massaro, F. & Telaretti, E. & Zizzo, G., 2019. "Technological and legislative status point of stationary energy storages in the EU," Renewable and Sustainable Energy Reviews, Elsevier, vol. 101(C), pages 158-167.
    4. Wonchang Hur & Yongma Moon & Kwangsup Shin & Wooje Kim & Suchul Nam & Kijun Park, 2015. "Economic Value of Li-ion Energy Storage System in Frequency Regulation Application from Utility Firm’s Perspective in Korea," Energies, MDPI, vol. 8(6), pages 1-18, May.
    5. Sioshansi, Ramteen & Denholm, Paul & Jenkin, Thomas, 2011. "A comparative analysis of the value of pure and hybrid electricity storage," Energy Economics, Elsevier, vol. 33(1), pages 56-66, January.
    6. He, Xian & Delarue, Erik & D'haeseleer, William & Glachant, Jean-Michel, 2011. "A novel business model for aggregating the values of electricity storage," Energy Policy, Elsevier, vol. 39(3), pages 1575-1585, March.
    7. Chyong, Chi Kong & Newbery, David, 2022. "A unit commitment and economic dispatch model of the GB electricity market – Formulation and application to hydro pumped storage," Energy Policy, Elsevier, vol. 170(C).
    8. Telaretti, E. & Dusonchet, L., 2017. "Stationary battery technologies in the U.S.: Development Trends and prospects," Renewable and Sustainable Energy Reviews, Elsevier, vol. 75(C), pages 380-392.
    9. Brändle, Gregor & Schönfisch, Max & Schulte, Simon, 2020. "Estimating Long-Term Global Supply Costs for Low-Carbon Hydrogen," EWI Working Papers 2020-4, Energiewirtschaftliches Institut an der Universitaet zu Koeln (EWI), revised 10 Aug 2021.
    10. Telaretti, E. & Graditi, G. & Ippolito, M.G. & Zizzo, G., 2016. "Economic feasibility of stationary electrochemical storages for electric bill management applications: The Italian scenario," Energy Policy, Elsevier, vol. 94(C), pages 126-137.
    11. Dufo-López, Rodolfo, 2015. "Optimisation of size and control of grid-connected storage under real time electricity pricing conditions," Applied Energy, Elsevier, vol. 140(C), pages 395-408.
    12. Yangfang (Helen) Zhou & Alan Scheller-Wolf & Nicola Secomandi & Stephen Smith, 2016. "Electricity Trading and Negative Prices: Storage vs. Disposal," Management Science, INFORMS, vol. 62(3), pages 880-898, March.
    13. Arcos-Vargas, Ángel & Canca, David & Núñez, Fernando, 2020. "Impact of battery technological progress on electricity arbitrage: An application to the Iberian market," Applied Energy, Elsevier, vol. 260(C).
    14. Núñez, Fernando & Canca, David & Arcos-Vargas, Ángel, 2022. "An assessment of European electricity arbitrage using storage systems," Energy, Elsevier, vol. 242(C).
    15. Carson, Richard T. & Novan, Kevin, 2013. "The private and social economics of bulk electricity storage," Journal of Environmental Economics and Management, Elsevier, vol. 66(3), pages 404-423.
    16. Metz, Dennis & Saraiva, João Tomé, 2018. "Simultaneous co-integration of multiple electrical storage applications in a consumer setting," Energy, Elsevier, vol. 143(C), pages 202-211.
    17. Yu, Nanpeng & Foggo, Brandon, 2017. "Stochastic valuation of energy storage in wholesale power markets," Energy Economics, Elsevier, vol. 64(C), pages 177-185.
    18. Ikechi Emmanuel, Michael & Denholm, Paul, 2022. "A market feedback framework for improved estimates of the arbitrage value of energy storage using price-taker models," Applied Energy, Elsevier, vol. 310(C).
    19. Berrada, Asmae & Loudiyi, Khalid & Zorkani, Izeddine, 2016. "Valuation of energy storage in energy and regulation markets," Energy, Elsevier, vol. 115(P1), pages 1109-1118.
    20. Cho, Joohyun & Kleit, Andrew N., 2015. "Energy storage systems in energy and ancillary markets: A backwards induction approach," Applied Energy, Elsevier, vol. 147(C), pages 176-183.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:14:y:2021:i:24:p:8365-:d:700404. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.