IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v317y2022ics0306261922005025.html
   My bibliography  Save this article

Multi-agent deep deterministic policy gradient algorithm for peer-to-peer energy trading considering distribution network constraints

Author

Listed:
  • Samende, Cephas
  • Cao, Jun
  • Fan, Zhong

Abstract

In this paper, we investigate an energy cost minimization problem for prosumers participating in peer-to-peer energy trading. Due to (i) uncertainties caused by renewable energy generation and consumption, (ii) difficulties in developing an accurate and efficient energy trading model, and (iii) the need to satisfy distribution network constraints, it is challenging for prosumers to obtain optimal energy trading decisions that minimize their individual energy costs. To address the challenge, we first formulate the above problem as a Markov decision process and propose a multi-agent deep deterministic policy gradient algorithm to learn optimal energy trading decisions. To satisfy the distribution network constraints, we propose distribution network tariffs which we incorporate in the algorithm as incentives to incentivize energy trading decisions that help to satisfy the constraints and penalize the decisions that violate them. The proposed algorithm is model-free and allows the agents to learn the optimal energy trading decisions without having prior information about other agents in the network. Simulation results based on real-world datasets show the effectiveness and robustness of the proposed algorithm.

Suggested Citation

  • Samende, Cephas & Cao, Jun & Fan, Zhong, 2022. "Multi-agent deep deterministic policy gradient algorithm for peer-to-peer energy trading considering distribution network constraints," Applied Energy, Elsevier, vol. 317(C).
    • Handle: RePEc:eee:appene:v:317:y:2022:i:c:s0306261922005025
    DOI: 10.1016/j.apenergy.2022.119123
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261922005025
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2022.119123?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Jin-Gyeom Kim & Bowon Lee, 2020. "Automatic P2P Energy Trading Model Based on Reinforcement Learning Using Long Short-Term Delayed Reward," Energies, MDPI, vol. 13(20), pages 1-27, October.
    2. Olivella-Rosell, Pol & Bullich-Massagué, Eduard & Aragüés-Peñalba, Mònica & Sumper, Andreas & Ottesen, Stig Ødegaard & Vidal-Clos, Josep-Andreu & Villafáfila-Robles, Roberto, 2018. "Optimization problem for meeting distribution system operator requests in local flexibility markets with distributed energy resources," Applied Energy, Elsevier, vol. 210(C), pages 881-895.
    3. Qiu, Dawei & Ye, Yujian & Papadaskalopoulos, Dimitrios & Strbac, Goran, 2021. "Scalable coordinated management of peer-to-peer energy trading: A multi-cluster deep reinforcement learning approach," Applied Energy, Elsevier, vol. 292(C).
    4. Han, Sekyung & Han, Soohee & Aki, Hirohisa, 2014. "A practical battery wear model for electric vehicle charging applications," Applied Energy, Elsevier, vol. 113(C), pages 1100-1108.
    5. Thomas Morstyn & Niall Farrell & Sarah J. Darby & Malcolm D. McCulloch, 2018. "Using peer-to-peer energy-trading platforms to incentivize prosumers to form federated power plants," Nature Energy, Nature, vol. 3(2), pages 94-101, February.
    6. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    7. Long, Chao & Wu, Jianzhong & Zhou, Yue & Jenkins, Nick, 2018. "Peer-to-peer energy sharing through a two-stage aggregated battery control in a community Microgrid," Applied Energy, Elsevier, vol. 226(C), pages 261-276.
    8. Juhar Abdella & Khaled Shuaib, 2018. "Peer to Peer Distributed Energy Trading in Smart Grids: A Survey," Energies, MDPI, vol. 11(6), pages 1-22, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Meng, Yuan & Qiu, Jing & Zhang, Cuo & Lei, Gang & Zhu, Jianguo, 2024. "A Holistic P2P market for active and reactive energy trading in VPPs considering both financial benefits and network constraints," Applied Energy, Elsevier, vol. 356(C).
    2. Cephas Samende & Zhong Fan & Jun Cao & Renzo Fabián & Gregory N. Baltas & Pedro Rodriguez, 2023. "Battery and Hydrogen Energy Storage Control in a Smart Energy Network with Flexible Energy Demand Using Deep Reinforcement Learning," Energies, MDPI, vol. 16(19), pages 1-20, September.
    3. Pikkanate Angaphiwatchawal & Surachai Chaitusaney, 2024. "Optimizing Peer-to-Peer Energy Transactions: Determining the Allowable Maximum Trading Power for Participants," Energies, MDPI, vol. 17(6), pages 1-23, March.
    4. Chang, Weiguang & Dong, Wei & Yang, Qiang, 2023. "Day-ahead bidding strategy of cloud energy storage serving multiple heterogeneous microgrids in the electricity market," Applied Energy, Elsevier, vol. 336(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Meritxell Domènech Monfort & César De Jesús & Natapon Wanapinit & Niklas Hartmann, 2022. "A Review of Peer-to-Peer Energy Trading with Standard Terminology Proposal and a Techno-Economic Characterisation Matrix," Energies, MDPI, vol. 15(23), pages 1-29, November.
    2. Azim, M. Imran & Tushar, Wayes & Saha, Tapan K. & Yuen, Chau & Smith, David, 2022. "Peer-to-peer kilowatt and negawatt trading: A review of challenges and recent advances in distribution networks," Renewable and Sustainable Energy Reviews, Elsevier, vol. 169(C).
    3. Qiu, Dawei & Ye, Yujian & Papadaskalopoulos, Dimitrios & Strbac, Goran, 2021. "Scalable coordinated management of peer-to-peer energy trading: A multi-cluster deep reinforcement learning approach," Applied Energy, Elsevier, vol. 292(C).
    4. Tushar, Wayes & Yuen, Chau & Saha, Tapan K. & Morstyn, Thomas & Chapman, Archie C. & Alam, M. Jan E. & Hanif, Sarmad & Poor, H. Vincent, 2021. "Peer-to-peer energy systems for connected communities: A review of recent advances and emerging challenges," Applied Energy, Elsevier, vol. 282(PA).
    5. Zhou, Yuekuan & Lund, Peter D., 2023. "Peer-to-peer energy sharing and trading of renewable energy in smart communities ─ trading pricing models, decision-making and agent-based collaboration," Renewable Energy, Elsevier, vol. 207(C), pages 177-193.
    6. Jin, Xiaolong & Wu, Qiuwei & Jia, Hongjie, 2020. "Local flexibility markets: Literature review on concepts, models and clearing methods," Applied Energy, Elsevier, vol. 261(C).
    7. Wang, Zibo & Yu, Xiaodan & Mu, Yunfei & Jia, Hongjie, 2020. "A distributed Peer-to-Peer energy transaction method for diversified prosumers in Urban Community Microgrid System," Applied Energy, Elsevier, vol. 260(C).
    8. Liu, Jia & Yang, Hongxing & Zhou, Yuekuan, 2021. "Peer-to-peer trading optimizations on net-zero energy communities with energy storage of hydrogen and battery vehicles," Applied Energy, Elsevier, vol. 302(C).
    9. Francesca Andreolli & Chiara D'Alpaos & Peter Kort, 2023. "Does P2P Trading Favor Investments in PV-Battery Systems?," Working Papers 2023.02, Fondazione Eni Enrico Mattei.
    10. Wang, Zibo & Yu, Xiaodan & Mu, Yunfei & Jia, Hongjie & Jiang, Qian & Wang, Xiaoyu, 2021. "Peer-to-Peer energy trading strategy for energy balance service provider (EBSP) considering market elasticity in community microgrid," Applied Energy, Elsevier, vol. 303(C).
    11. Cephas Samende & Zhong Fan & Jun Cao & Renzo Fabián & Gregory N. Baltas & Pedro Rodriguez, 2023. "Battery and Hydrogen Energy Storage Control in a Smart Energy Network with Flexible Energy Demand Using Deep Reinforcement Learning," Energies, MDPI, vol. 16(19), pages 1-20, September.
    12. Lüth, Alexandra & Zepter, Jan Martin & Crespo del Granado, Pedro & Egging, Ruud, 2018. "Local electricity market designs for peer-to-peer trading: The role of battery flexibility," Applied Energy, Elsevier, vol. 229(C), pages 1233-1243.
    13. Javier Parra-Domínguez & Esteban Sánchez & Ángel Ordóñez, 2023. "The Prosumer: A Systematic Review of the New Paradigm in Energy and Sustainable Development," Sustainability, MDPI, vol. 15(13), pages 1-44, July.
    14. Kobashi, Takuro & Choi, Younghun & Hirano, Yujiro & Yamagata, Yoshiki & Say, Kelvin, 2022. "Rapid rise of decarbonization potentials of photovoltaics plus electric vehicles in residential houses over commercial districts," Applied Energy, Elsevier, vol. 306(PB).
    15. Liangyi Pu & Song Wang & Xiaodong Huang & Xing Liu & Yawei Shi & Huiwei Wang, 2022. "Peer-to-Peer Trading for Energy-Saving Based on Reinforcement Learning," Energies, MDPI, vol. 15(24), pages 1-16, December.
    16. Barone, G. & Buonomano, A. & Forzano, C. & Palombo, A. & Russo, G., 2023. "The role of energy communities in electricity grid balancing: A flexible tool for smart grid power distribution optimization," Renewable and Sustainable Energy Reviews, Elsevier, vol. 187(C).
    17. Neves, Diana & Scott, Ian & Silva, Carlos A., 2020. "Peer-to-peer energy trading potential: An assessment for the residential sector under different technology and tariff availabilities," Energy, Elsevier, vol. 205(C).
    18. Zheng, Boshen & Wei, Wei & Chen, Yue & Wu, Qiuwei & Mei, Shengwei, 2022. "A peer-to-peer energy trading market embedded with residential shared energy storage units," Applied Energy, Elsevier, vol. 308(C).
    19. Arnob Das & Susmita Datta Peu & Md. Abdul Mannan Akanda & Abu Reza Md. Towfiqul Islam, 2023. "Peer-to-Peer Energy Trading Pricing Mechanisms: Towards a Comprehensive Analysis of Energy and Network Service Pricing (NSP) Mechanisms to Get Sustainable Enviro-Economical Energy Sector," Energies, MDPI, vol. 16(5), pages 1-27, February.
    20. Johnson, R.C. & Mayfield, M., 2020. "The economic and environmental implications of post feed-in tariff PV on constrained low voltage networks," Applied Energy, Elsevier, vol. 279(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:317:y:2022:i:c:s0306261922005025. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.