IDEAS home Printed from https://ideas.repec.org/a/eee/appene/v335y2023ics0306261923000715.html
   My bibliography  Save this article

Design and planning of flexible mobile Micro-Grids using Deep Reinforcement Learning

Author

Listed:
  • Caputo, Cesare
  • Cardin, Michel-Alexandre
  • Ge, Pudong
  • Teng, Fei
  • Korre, Anna
  • Antonio del Rio Chanona, Ehecatl

Abstract

Ongoing risks from climate change have significantly impacted the livelihood of global nomadic communities and are likely to lead to increased migratory movements in coming years. As a result, mobility considerations are becoming increasingly important in energy systems planning, particularly to achieve energy access in developing countries. Advanced “Plug and Play” control strategies have been recently developed with such a decentralized framework in mind, allowing easier interconnection of nomadic communities, both to each other and to the main grid. Considering the above, the design and planning strategy of a mobile multi-energy supply system for a nomadic community is investigated in this work. Motivated by the scale and dimensionality of the associated uncertainties, impacting all major design and decision variables over the 30-year planning horizon, Deep Reinforcement Learning (DRL) Flexibility Analysis is implemented for the design and planning problem. DRL based solutions are benchmarked against several rigid baseline design options to compare expected performance under uncertainty. The results on a case study for ger communities in Mongolia suggest that mobile nomadic energy systems can be both technically and economically feasible, particularly when considering flexibility, although the degree of spatial dispersion among households is an important limiting factor. Additionally, the DRL based policies lead to the development of dynamic evolution and adaptability strategies, which can be used by the targeted communities under a very wide range of potential scenarios. Key economic, sustainability and resilience indicators such as Cost, Equivalent Emissions and Total Unmet Load are measured, suggesting potential improvements compared to available baselines of up to 25%, 67% and 76%, respectively. Finally, the decomposition of values of flexibility and plug and play operation is presented using a variation of real options theory, with important implications for both nomadic communities and policymakers focused on enabling their energy access.

Suggested Citation

  • Caputo, Cesare & Cardin, Michel-Alexandre & Ge, Pudong & Teng, Fei & Korre, Anna & Antonio del Rio Chanona, Ehecatl, 2023. "Design and planning of flexible mobile Micro-Grids using Deep Reinforcement Learning," Applied Energy, Elsevier, vol. 335(C).
    • Handle: RePEc:eee:appene:v:335:y:2023:i:c:s0306261923000715
    DOI: 10.1016/j.apenergy.2023.120707
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0306261923000715
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.apenergy.2023.120707?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Bat-Erdene Bayandelger & Yuzuru Ueda & Amarbayar Adiyabat, 2020. "Experimental Investigation and Energy Performance Simulation of Mongolian Ger with ETS Heater and Solar PV in Ulaanbaatar City," Energies, MDPI, vol. 13(21), pages 1-13, November.
    2. Abdin, Adam F. & Caunhye, Aakil & Zio, Enrico & Cardin, Michel-Alexandre, 2022. "Optimizing generation expansion planning with operational uncertainty: A multistage adaptive robust approach," Applied Energy, Elsevier, vol. 306(PA).
    3. Perera, A.T.D. & Kamalaruban, Parameswaran, 2021. "Applications of reinforcement learning in energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 137(C).
    4. Pecenak, Zachary K. & Stadler, Michael & Fahy, Kelsey, 2019. "Efficient multi-year economic energy planning in microgrids," Applied Energy, Elsevier, vol. 255(C).
    5. Kaoru Kakinuma & Aki Yanagawa & Takehiro Sasaki & Mukund Palat Rao & Shinjiro Kanae, 2019. "Socio-ecological Interactions in a Changing Climate: A Review of the Mongolian Pastoral System," Sustainability, MDPI, vol. 11(21), pages 1-17, October.
    6. Vázquez-Canteli, José R. & Nagy, Zoltán, 2019. "Reinforcement learning for demand response: A review of algorithms and modeling techniques," Applied Energy, Elsevier, vol. 235(C), pages 1072-1089.
    7. Martínez-Ceseña, E.A. & Mutale, J., 2011. "Application of an advanced real options approach for renewable energy generation projects planning," Renewable and Sustainable Energy Reviews, Elsevier, vol. 15(4), pages 2087-2094, May.
    8. Fernandes, Bartolomeu & Cunha, Jorge & Ferreira, Paula, 2011. "The use of real options approach in energy sector investments," Renewable and Sustainable Energy Reviews, Elsevier, vol. 15(9), pages 4491-4497.
    9. Pfenninger, Stefan & Staffell, Iain, 2016. "Long-term patterns of European PV output using 30 years of validated hourly reanalysis and satellite data," Energy, Elsevier, vol. 114(C), pages 1251-1265.
    10. Xiujuan Yang & Jiying Sun & Julin Gao & Shuaishuai Qiao & Baolin Zhang & Haizhu Bao & Xinwei Feng & Songyu Wang, 2021. "Effects of Climate Change on Cultivation Patterns and Climate Suitability of Spring Maize in Inner Mongolia," Sustainability, MDPI, vol. 13(14), pages 1-21, July.
    11. Roman Hoffmann & Anna Dimitrova & Raya Muttarak & Jesus Crespo Cuaresma & Jonas Peisker, 2020. "A meta-analysis of country-level studies on environmental change and migration," Nature Climate Change, Nature, vol. 10(10), pages 904-912, October.
    12. Michel-Alexandre Cardin & Qihui Xie & Tsan Sheng Ng & Shuming Wang & Junfei Hu, 2017. "An approach for analyzing and managing flexibility in engineering systems design based on decision rules and multistage stochastic programming," IISE Transactions, Taylor & Francis Journals, vol. 49(1), pages 1-12, January.
    13. Otsuki, Takashi, 2017. "Costs and benefits of large-scale deployment of wind turbines and solar PV in Mongolia for international power exports," Renewable Energy, Elsevier, vol. 108(C), pages 321-335.
    14. Subbiah, Adritha & Mansoor, Sahar & Misra, Rachita & Jaffer, Huda & Tiwary, Raunak, 2016. "Addressing developmental needs through energy access in informal settlements," LSE Research Online Documents on Economics 83626, London School of Economics and Political Science, LSE Library.
    15. Lammers, Katrin & Bertheau, Paul & Blechinger, Philipp, 2020. "Exploring requirements for sustainable energy supply planning with regard to climate resilience of Southeast Asian islands," Energy Policy, Elsevier, vol. 146(C).
    16. Martha N. Acosta & Choidorj Adiyabazar & Francisco Gonzalez-Longatt & Manuel A. Andrade & José Rueda Torres & Ernesto Vazquez & Jesús Manuel Riquelme Santos, 2020. "Optimal Under-Frequency Load Shedding Setting at Altai-Uliastai Regional Power System, Mongolia," Energies, MDPI, vol. 13(20), pages 1-18, October.
    17. Gamarra, Carlos & Guerrero, Josep M., 2015. "Computational optimization techniques applied to microgrids planning: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 48(C), pages 413-424.
    18. Peter Boait & Rupert Gammon & Varun Advani & Neal Wade & David Greenwood & Peter Davison, 2017. "ESCoBox: A Set of Tools for Mini-Grid Sustainability in the Developing World," Sustainability, MDPI, vol. 9(5), pages 1-15, May.
    19. Jonas Degrave & Federico Felici & Jonas Buchli & Michael Neunert & Brendan Tracey & Francesco Carpanese & Timo Ewalds & Roland Hafner & Abbas Abdolmaleki & Diego de las Casas & Craig Donner & Leslie F, 2022. "Magnetic control of tokamak plasmas through deep reinforcement learning," Nature, Nature, vol. 602(7897), pages 414-419, February.
    20. Mandelli, Stefano & Barbieri, Jacopo & Mereu, Riccardo & Colombo, Emanuela, 2016. "Off-grid systems for rural electrification in developing countries: Definitions, classification and a comprehensive literature review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 58(C), pages 1621-1646.
    21. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    22. Staffell, Iain & Pfenninger, Stefan, 2016. "Using bias-corrected reanalysis to simulate current and future wind power output," Energy, Elsevier, vol. 114(C), pages 1224-1239.
    23. Wang, Yi & Rousis, Anastasios Oulis & Strbac, Goran, 2020. "On microgrids and resilience: A comprehensive review on modeling and operational strategies," Renewable and Sustainable Energy Reviews, Elsevier, vol. 134(C).
    24. Postali, Fernando A.S. & Picchetti, Paulo, 2006. "Geometric Brownian Motion and structural breaks in oil prices: A quantitative analysis," Energy Economics, Elsevier, vol. 28(4), pages 506-522, July.
    25. Andriotis, C.P. & Papakonstantinou, K.G., 2019. "Managing engineering systems with large state and action spaces through deep reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 191(C).
    26. Aakil M. Caunhye & Michel-Alexandre Cardin & Muhammad Rahmat, 2022. "Flexibility and real options analysis in power system generation expansion planning under uncertainty," IISE Transactions, Taylor & Francis Journals, vol. 54(9), pages 832-844, June.
    27. Randy L. Maddalena & Melissa M. Lunden & Daniel L. Wilson & Cristina Ceballos & Thomas W. Kirchstetter & Jonathan L. Slack & Larry L. Dale, 2014. "Quantifying Space Heating Stove Emissions Related to Different Use Patterns in Mongolia," Energy and Environment Research, Canadian Center of Science and Education, vol. 4(3), pages 147-147, December.
    28. Ehsan, Ali & Yang, Qiang, 2018. "Optimal integration and planning of renewable distributed generation in the power distribution networks: A review of analytical techniques," Applied Energy, Elsevier, vol. 210(C), pages 44-59.
    29. Borchers, Allison M. & Duke, Joshua M. & Parsons, George R., 2007. "Does willingness to pay for green energy differ by source?," Energy Policy, Elsevier, vol. 35(6), pages 3327-3334, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Bigestans, Davis & Cardin, Michel-Alexandre & Kazantzis, Nikolaos, 2023. "Economic performance evaluation of flexible centralised and decentralised blue hydrogen production systems design under uncertainty," Applied Energy, Elsevier, vol. 352(C).
    2. Davi-Arderius, Daniel & Jamasb, Tooraj & Rosellon, Juan, 2024. "Network Operation and Constraints and the Path to Net Zero," Working Papers 8-2024, Copenhagen Business School, Department of Economics.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Petkov, Ivalin & Gabrielli, Paolo, 2020. "Power-to-hydrogen as seasonal energy storage: an uncertainty analysis for optimal design of low-carbon multi-energy systems," Applied Energy, Elsevier, vol. 274(C).
    2. Qiu, Dawei & Wang, Yi & Hua, Weiqi & Strbac, Goran, 2023. "Reinforcement learning for electric vehicle applications in power systems:A critical review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 173(C).
    3. Kim, Sunwoo & Choi, Yechan & Park, Joungho & Adams, Derrick & Heo, Seongmin & Lee, Jay H., 2024. "Multi-period, multi-timescale stochastic optimization model for simultaneous capacity investment and energy management decisions for hybrid Micro-Grids with green hydrogen production under uncertainty," Renewable and Sustainable Energy Reviews, Elsevier, vol. 190(PA).
    4. Harrold, Daniel J.B. & Cao, Jun & Fan, Zhong, 2022. "Data-driven battery operation for energy arbitrage using rainbow deep reinforcement learning," Energy, Elsevier, vol. 238(PC).
    5. Petrelli, Marina & Fioriti, Davide & Berizzi, Alberto & Bovo, Cristian & Poli, Davide, 2021. "A novel multi-objective method with online Pareto pruning for multi-year optimization of rural microgrids," Applied Energy, Elsevier, vol. 299(C).
    6. Biemann, Marco & Scheller, Fabian & Liu, Xiufeng & Huang, Lizhen, 2021. "Experimental evaluation of model-free reinforcement learning algorithms for continuous HVAC control," Applied Energy, Elsevier, vol. 298(C).
    7. Abada, Ibrahim & Othmani, Mehdi & Tatry, Léa, 2021. "An innovative approach for the optimal sizing of mini-grids in rural areas integrating the demand, the supply, and the grid," Renewable and Sustainable Energy Reviews, Elsevier, vol. 146(C).
    8. Petkov, Ivalin & Mavromatidis, Georgios & Knoeri, Christof & Allan, James & Hoffmann, Volker H., 2022. "MANGOret: An optimization framework for the long-term investment planning of building multi-energy system and envelope retrofits," Applied Energy, Elsevier, vol. 314(C).
    9. Maeder, Mattia & Weiss, Olga & Boulouchos, Konstantinos, 2021. "Assessing the need for flexibility technologies in decarbonized power systems: A new model applied to Central Europe," Applied Energy, Elsevier, vol. 282(PA).
    10. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    11. Harrold, Daniel J.B. & Cao, Jun & Fan, Zhong, 2022. "Renewable energy integration and microgrid energy trading using multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 318(C).
    12. Marko Hočevar & Lovrenc Novak & Primož Drešar & Gašper Rak, 2022. "The Status Quo and Future of Hydropower in Slovenia," Energies, MDPI, vol. 15(19), pages 1-13, September.
    13. Lukas Kriechbaum & Philipp Gradl & Romeo Reichenhauser & Thomas Kienberger, 2020. "Modelling Grid Constraints in a Multi-Energy Municipal Energy System Using Cumulative Exergy Consumption Minimisation," Energies, MDPI, vol. 13(15), pages 1-23, July.
    14. Behrang Shirizadeh, Quentin Perrier, and Philippe Quirion, 2022. "How Sensitive are Optimal Fully Renewable Power Systems to Technology Cost Uncertainty?," The Energy Journal, International Association for Energy Economics, vol. 0(Number 1).
    15. Omoyele, Olalekan & Hoffmann, Maximilian & Koivisto, Matti & Larrañeta, Miguel & Weinand, Jann Michael & Linßen, Jochen & Stolten, Detlef, 2024. "Increasing the resolution of solar and wind time series for energy system modeling: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 189(PB).
    16. Liu, Hailiang & Andresen, Gorm Bruun & Greiner, Martin, 2018. "Cost-optimal design of a simplified highly renewable Chinese electricity network," Energy, Elsevier, vol. 147(C), pages 534-546.
    17. Géremi Gilson Dranka & Paula Ferreira, 2020. "Electric Vehicles and Biofuels Synergies in the Brazilian Energy System," Energies, MDPI, vol. 13(17), pages 1-22, August.
    18. Shirizadeh, Behrang & Quirion, Philippe, 2022. "The importance of renewable gas in achieving carbon-neutrality: Insights from an energy system optimization model," Energy, Elsevier, vol. 255(C).
    19. Sun, Hongchang & Niu, Yanlei & Li, Chengdong & Zhou, Changgeng & Zhai, Wenwen & Chen, Zhe & Wu, Hao & Niu, Lanqiang, 2022. "Energy consumption optimization of building air conditioning system via combining the parallel temporal convolutional neural network and adaptive opposition-learning chimp algorithm," Energy, Elsevier, vol. 259(C).
    20. Langer, Lissy & Volling, Thomas, 2022. "A reinforcement learning approach to home energy management for modulating heat pumps and photovoltaic systems," Applied Energy, Elsevier, vol. 327(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:appene:v:335:y:2023:i:c:s0306261923000715. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/405891/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.