IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v351y2023ics0306261923011418.html
   My bibliography  Save this article

Two-timescale autonomous energy management strategy based on multi-agent deep reinforcement learning approach for residential multicarrier energy system

Author

Listed:
  • Zhang, Bin
  • Hu, Weihao
  • Ghias, Amer M.Y.M.
  • Xu, Xiao
  • Chen, Zhe

Abstract

In the residential multicarrier energy system (RMES), autonomous energy management (AEM) is beneficial for prosumers' costs that actively controls generation, energy conversion, and storage in real-time. Conventional model-based AEM methods rely on forecast models of distributed energy resources and proper system parameters, which are difficult for practical application. This article proposes a novel model-free two-timescale real-time AEM strategy for the RMES. The energy management problem in the RMES is time-decomposed into two timescales (hourly-ahead external energy trading and 15-min ahead internal energy conversion), formulated as Markov Games. Then, defining a multi-agent system not only enables separate energy management diagrams but also accelerates learning. Intelligent agents account for optimizing energy trading and energy conversion to minimize daily costs under the training of a deep deterministic policy gradient algorithm. In order to learn a collaborative strategy, all agents are trained centrally while being executed based on local information in a decentralized manner for fast response. A deterministic study indicates that the proposed AEM strategy can effectively and flexibly schedule the operation of components with respect to different price signals and load profiles. In the stochastic study that considers the error thresholds of solar panels generation and loads, the difference between the energy cost of test scenarios with a trained strategy and the no-regret learning method is equal to 0.32%, which is appropriate. In addition, the proposed method not only achieves a reduction in power imbalance but also exhibits lower energy costs when compared to various benchmark methods.

Suggested Citation

  • Zhang, Bin & Hu, Weihao & Ghias, Amer M.Y.M. & Xu, Xiao & Chen, Zhe, 2023. "Two-timescale autonomous energy management strategy based on multi-agent deep reinforcement learning approach for residential multicarrier energy system," Applied Energy, Elsevier, vol. 351(C).
  • Handle: RePEc:eee:appene:v:351:y:2023:i:c:s0306261923011418
    DOI: 10.1016/j.apenergy.2023.121777
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261923011418
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.apenergy.2023.121777?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Gunther Glenk & Stefan Reichelstein, 2019. "Publisher Correction: Economics of converting renewable power to hydrogen," Nature Energy, Nature, vol. 4(4), pages 347-347, April.
    2. Gunther Glenk & Stefan Reichelstein, 2019. "Economics of converting renewable power to hydrogen," Nature Energy, Nature, vol. 4(3), pages 216-222, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Zhou, Yuan & Wang, Jiangjiang & Wei, Changqi & Li, Yuxin, 2024. "A novel two-stage multi-objective dispatch model for a distributed hybrid CCHP system considering source-load fluctuations mitigation," Energy, Elsevier, vol. 300(C).
    2. Wu, Qian & Song, Qiankun & He, Xing & Chen, Guo & Huang, Tingwen, 2024. "Distributed peer-to-peer energy trading framework with manufacturing assembly process and uncertain renewable energy plants in multi-industrial micro-grids," Energy, Elsevier, vol. 302(C).
    3. Zhang, Rongquan & Bu, Siqi & Li, Gangqiang, 2024. "Multi-market P2P trading of cooling–heating-power-hydrogen integrated energy systems: An equilibrium-heuristic online prediction optimization approach," Applied Energy, Elsevier, vol. 367(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tao Zhang & Qitong Ye & Zengyu Han & Qingyi Liu & Yipu Liu & Dongshuang Wu & Hong Jin Fan, 2024. "Biaxial strain induced OH engineer for accelerating alkaline hydrogen evolution," Nature Communications, Nature, vol. 15(1), pages 1-11, December.
    2. Jong-Hyun Kim & Yong-Gil Lee, 2021. "Factors of Collaboration Affecting the Performance of Alternative Energy Patents in South Korea from 2010 to 2017," Sustainability, MDPI, vol. 13(18), pages 1-25, September.
    3. Yong Zuo & Sebastiano Bellani & Michele Ferri & Gabriele Saleh & Dipak V. Shinde & Marilena Isabella Zappia & Rosaria Brescia & Mirko Prato & Luca Trizio & Ivan Infante & Francesco Bonaccorso & Libera, 2023. "High-performance alkaline water electrolyzers based on Ru-perturbed Cu nanoplatelets cathode," Nature Communications, Nature, vol. 14(1), pages 1-15, December.
    4. Vassilis M. Charitopoulos & Mathilde Fajardy & Chi Kong Chyong & David M. Reiner, 2022. "The case of 100% electrification of domestic heat in Great Britain," Working Papers EPRG2206, Energy Policy Research Group, Cambridge Judge Business School, University of Cambridge.
    5. Tubagus Aryandi Gunawan & Alessandro Singlitico & Paul Blount & James Burchill & James G. Carton & Rory F. D. Monaghan, 2020. "At What Cost Can Renewable Hydrogen Offset Fossil Fuel Use in Ireland’s Gas Network?," Energies, MDPI, vol. 13(7), pages 1-23, April.
    6. Matthias Maldet & Daniel Schwabeneder & Georg Lettner & Christoph Loschan & Carlo Corinaldesi & Hans Auer, 2022. "Beyond Traditional Energy Sector Coupling: Conserving and Efficient Use of Local Resources," Sustainability, MDPI, vol. 14(12), pages 1-36, June.
    7. Heming Liu & Ruikuan Xie & Yuting Luo & Zhicheng Cui & Qiangmin Yu & Zhiqiang Gao & Zhiyuan Zhang & Fengning Yang & Xin Kang & Shiyu Ge & Shaohai Li & Xuefeng Gao & Guoliang Chai & Le Liu & Bilu Liu, 2022. "Dual interfacial engineering of a Chevrel phase electrode material for stable hydrogen evolution at 2500 mA cm−2," Nature Communications, Nature, vol. 13(1), pages 1-10, December.
    8. Abadie, Luis Mª & Chamorro, José M., 2023. "Investment in wind-based hydrogen production under economic and physical uncertainties," Applied Energy, Elsevier, vol. 337(C).
    9. Haugen, Molly J. & Paoli, Leonardo & Cullen, Jonathan & Cebon, David & Boies, Adam M., 2021. "A fork in the road: Which energy pathway offers the greatest energy efficiency and CO2 reduction potential for low-carbon vehicles?," Applied Energy, Elsevier, vol. 283(C).
    10. Jafri, Yawer & Wetterlund, Elisabeth & Mesfun, Sennai & Rådberg, Henrik & Mossberg, Johanna & Hulteberg, Christian & Furusjö, Erik, 2020. "Combining expansion in pulp capacity with production of sustainable biofuels – Techno-economic and greenhouse gas emissions assessment of drop-in fuels from black liquor part-streams," Applied Energy, Elsevier, vol. 279(C).
    11. Stöckl, Fabian & Schill, Wolf-Peter & Zerrahn, Alexander, 2021. "Optimal supply chains and power sector benefits of green hydrogen," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 11.
    12. Shaojie Song & Haiyang Lin & Peter Sherman & Xi Yang & Chris P. Nielsen & Xinyu Chen & Michael B. McElroy, 2021. "Production of hydrogen from offshore wind in China and cost-competitive supply to Japan," Nature Communications, Nature, vol. 12(1), pages 1-8, December.
    13. Shirizadeh, Behrang & Quirion, Philippe, 2023. "Long-term optimization of the hydrogen-electricity nexus in France: Green, blue, or pink hydrogen?," Energy Policy, Elsevier, vol. 181(C).
    14. Lim, Dongjun & Lee, Boreum & Lee, Hyunjun & Byun, Manhee & Lim, Hankwon, 2022. "Projected cost analysis of hybrid methanol production from tri-reforming of methane integrated with various water electrolysis systems: Technical and economic assessment," Renewable and Sustainable Energy Reviews, Elsevier, vol. 155(C).
    15. Klöckner, Kai & Letmathe, Peter, 2020. "Is the coherence of coal phase-out and electrolytic hydrogen production the golden path to effective decarbonisation?," Applied Energy, Elsevier, vol. 279(C).
    16. Martin Thema & Tobias Weidlich & Manuel Hörl & Annett Bellack & Friedemann Mörs & Florian Hackl & Matthias Kohlmayer & Jasmin Gleich & Carsten Stabenau & Thomas Trabold & Michael Neubert & Felix Ortlo, 2019. "Biological CO 2 -Methanation: An Approach to Standardization," Energies, MDPI, vol. 12(9), pages 1-32, May.
    17. Gong, Mei & Ottermo, Fredric, 2022. "High-temperature thermal storage in combined heat and power plants," Energy, Elsevier, vol. 252(C).
    18. Ringkjøb, Hans-Kristian & Haugan, Peter M. & Nybø, Astrid, 2020. "Transitioning remote Arctic settlements to renewable energy systems – A modelling study of Longyearbyen, Svalbard," Applied Energy, Elsevier, vol. 258(C).
    19. Chauvy, Remi & Dubois, Lionel & Lybaert, Paul & Thomas, Diane & De Weireld, Guy, 2020. "Production of synthetic natural gas from industrial carbon dioxide," Applied Energy, Elsevier, vol. 260(C).
    20. Xun, Qian & Murgovski, Nikolce & Liu, Yujing, 2022. "Chance-constrained robust co-design optimization for fuel cell hybrid electric trucks," Applied Energy, Elsevier, vol. 320(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:351:y:2023:i:c:s0306261923011418. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.