IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v14y2021i20p6743-d658112.html
   My bibliography  Save this article

Deep Reinforcement Learning Based on Proximal Policy Optimization for the Maintenance of a Wind Farm with Multiple Crews

Author

Listed:
  • Luca Pinciroli

    (Energy Department, Politecnico di Milano, 20133 Milan, Italy)

  • Piero Baraldi

    (Energy Department, Politecnico di Milano, 20133 Milan, Italy)

  • Guido Ballabio

    (Aramis S.r.l., 20121 Milan, Italy)

  • Michele Compare

    (Aramis S.r.l., 20121 Milan, Italy)

  • Enrico Zio

    (Energy Department, Politecnico di Milano, 20133 Milan, Italy
    MINES ParisTech, Centre de Recherche sur les Risques et les Crises (CRC), PSL Research University, 06904 Sophia Antipolis, France)

Abstract

The life cycle of wind turbines depends on the operation and maintenance policies adopted. With the critical components of wind turbines being equipped with condition monitoring and Prognostics and Health Management (PHM) capabilities, it is feasible to significantly optimize operation and maintenance (O&M) by combining the (uncertain) information provided by PHM with the other factors influencing O&M activities, including the limited availability of maintenance crews, the variability of energy demand and corresponding production requests, and the long-time horizons of energy systems operation. In this work, we consider the operation and maintenance optimization of wind turbines in wind farms woth multiple crews. A new formulation of the problem as a sequential decision problem over a long-time horizon is proposed and solved by deep reinforcement learning based on proximal policy optimization. The proposed method is applied to a wind farm of 50 turbines, considering the availability of multiple maintenance crews. The optimal O&M policy found outperforms other state-of-the-art strategies, regardless of the number of available maintenance crews.

Suggested Citation

  • Luca Pinciroli & Piero Baraldi & Guido Ballabio & Michele Compare & Enrico Zio, 2021. "Deep Reinforcement Learning Based on Proximal Policy Optimization for the Maintenance of a Wind Farm with Multiple Crews," Energies, MDPI, vol. 14(20), pages 1-17, October.
    • Handle: RePEc:gam:jeners:v:14:y:2021:i:20:p:6743-:d:658112
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/14/20/6743/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1996-1073/14/20/6743/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Hu, Yang & Baraldi, Piero & Di Maio, Francesco & Zio, Enrico, 2015. "A particle filtering and kernel smoothing-based approach for new design component prognostics," Reliability Engineering and System Safety, Elsevier, vol. 134(C), pages 19-31.
    2. Sameer Al-Dahidi & Piero Baraldi & Enrico Zio & Lorenzo Montelatici, 2021. "Bootstrapped Ensemble of Artificial Neural Networks Technique for Quantifying Uncertainty in Prediction of Wind Energy Production," Sustainability, MDPI, vol. 13(11), pages 1-19, June.
    3. Shamshad, A. & Bawadi, M.A. & Wan Hussin, W.M.A. & Majid, T.A. & Sanusi, S.A.M., 2005. "First and second order Markov chain models for synthetic generation of wind speed time series," Energy, Elsevier, vol. 30(5), pages 693-708.
    4. Zhen Zhang & Cheng Ma & Rong Zhu, 2018. "Thermal and Energy Management Based on Bimodal Airflow-Temperature Sensing and Reinforcement Learning," Energies, MDPI, vol. 11(10), pages 1-14, September.
    5. Li, Y.F. & Valla, S. & Zio, E., 2015. "Reliability assessment of generic geared wind turbines by GTST-MLD model and Monte Carlo simulation," Renewable Energy, Elsevier, vol. 83(C), pages 222-233.
    6. Leite, Gustavo de Novaes Pires & Araújo, Alex Maurício & Rosas, Pedro André Carvalho, 2018. "Prognostic techniques applied to maintenance of wind turbines: a concise and specific review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 81(P2), pages 1917-1925.
    7. Shafiee, Mahmood & Sørensen, John Dalsgaard, 2019. "Maintenance optimization and inspection planning of wind energy assets: Models, methods and strategies," Reliability Engineering and System Safety, Elsevier, vol. 192(C).
    8. Rocchetta, R. & Bellani, L. & Compare, M. & Zio, E. & Patelli, E., 2019. "A reinforcement learning framework for optimal operation and maintenance of power grids," Applied Energy, Elsevier, vol. 241(C), pages 291-301.
    9. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    10. Perera, A.T.D. & Kamalaruban, Parameswaran, 2021. "Applications of reinforcement learning in energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 137(C).
    11. Zio, Enrico & Compare, Michele, 2013. "Evaluating maintenance policies by quantitative modeling and analysis," Reliability Engineering and System Safety, Elsevier, vol. 109(C), pages 53-65.
    12. Yuri Merizalde & Luis Hernández-Callejo & Oscar Duque-Perez & Víctor Alonso-Gómez, 2019. "Maintenance Models Applied to Wind Turbines. A Comprehensive Overview," Energies, MDPI, vol. 12(2), pages 1-41, January.
    13. Samet Ozturk & Vasilis Fthenakis & Stefan Faulstich, 2018. "Failure Modes, Effects and Criticality Analysis for Wind Turbines Considering Climatic Regions and Comparing Geared and Direct Drive Wind Turbines," Energies, MDPI, vol. 11(9), pages 1-18, September.
    14. Prabhakar V. Varde & Michael G. Pecht, 2018. "Prognostics and Health Management," Springer Series in Reliability Engineering, in: Risk-Based Engineering, chapter 0, pages 447-507, Springer.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Paweł Dymora & Mirosław Mazurek & Bartosz Sudek, 2021. "Comparative Analysis of Selected Open-Source Solutions for Traffic Balancing in Server Infrastructures Providing WWW Service," Energies, MDPI, vol. 14(22), pages 1-23, November.
    2. Pinciroli, Luca & Baraldi, Piero & Zio, Enrico, 2023. "Maintenance optimization in industry 4.0," Reliability Engineering and System Safety, Elsevier, vol. 234(C).
    3. Anwar, Ghazanfar Ali & Zhang, Xiaoge, 2024. "Deep reinforcement learning for intelligent risk optimization of buildings under hazard," Reliability Engineering and System Safety, Elsevier, vol. 247(C).
    4. Piero Baraldi & Roozbeh Razavi-Far & Enrico Zio, 2023. "Guest Editorial: Special Issue of ESREL2020 PSAM15," Energies, MDPI, vol. 16(4), pages 1-2, February.
    5. Daniele Candelaresi & Linda Moretti & Alessandra Perna & Giuseppe Spazzafumo, 2021. "Heat Recovery from a PtSNG Plant Coupled with Wind Energy," Energies, MDPI, vol. 14(22), pages 1-21, November.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Pinciroli, Luca & Baraldi, Piero & Zio, Enrico, 2023. "Maintenance optimization in industry 4.0," Reliability Engineering and System Safety, Elsevier, vol. 234(C).
    2. Pinciroli, Luca & Baraldi, Piero & Ballabio, Guido & Compare, Michele & Zio, Enrico, 2022. "Optimization of the Operation and Maintenance of renewable energy systems by Deep Reinforcement Learning," Renewable Energy, Elsevier, vol. 183(C), pages 752-763.
    3. Omar Al-Ani & Sanjoy Das, 2022. "Reinforcement Learning: Theory and Applications in HEMS," Energies, MDPI, vol. 15(17), pages 1-37, September.
    4. Izquierdo, J. & Márquez, A. Crespo & Uribetxebarria, J. & Erguido, A., 2020. "On the importance of assessing the operational context impact on maintenance management for life cycle cost of wind energy projects," Renewable Energy, Elsevier, vol. 153(C), pages 1100-1110.
    5. Caputo, Antonio C. & Federici, Alessandro & Pelagagge, Pacifico M. & Salini, Paolo, 2023. "Offshore wind power system economic evaluation framework under aleatory and epistemic uncertainty," Applied Energy, Elsevier, vol. 350(C).
    6. Li, Yanxue & Wang, Zixuan & Xu, Wenya & Gao, Weijun & Xu, Yang & Xiao, Fu, 2023. "Modeling and energy dynamic control for a ZEH via hybrid model-based deep reinforcement learning," Energy, Elsevier, vol. 277(C).
    7. Harrold, Daniel J.B. & Cao, Jun & Fan, Zhong, 2022. "Data-driven battery operation for energy arbitrage using rainbow deep reinforcement learning," Energy, Elsevier, vol. 238(PC).
    8. Harrold, Daniel J.B. & Cao, Jun & Fan, Zhong, 2022. "Renewable energy integration and microgrid energy trading using multi-agent deep reinforcement learning," Applied Energy, Elsevier, vol. 318(C).
    9. Mancuso, A. & Compare, M. & Salo, A. & Zio, E., 2021. "Optimal Prognostics and Health Management-driven inspection and maintenance strategies for industrial systems," Reliability Engineering and System Safety, Elsevier, vol. 210(C).
    10. Guan, Xiaoshu & Xiang, Zhengliang & Bao, Yuequan & Li, Hui, 2022. "Structural dominant failure modes searching method based on deep reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 219(C).
    11. Compare, Michele & Antonello, Federico & Pinciroli, Luca & Zio, Enrico, 2022. "A general model for life-cycle cost analysis of Condition-Based Maintenance enabled by PHM capabilities," Reliability Engineering and System Safety, Elsevier, vol. 224(C).
    12. Theissler, Andreas & Pérez-Velázquez, Judith & Kettelgerdes, Marcel & Elger, Gordon, 2021. "Predictive maintenance enabled by machine learning: Use cases and challenges in the automotive industry," Reliability Engineering and System Safety, Elsevier, vol. 215(C).
    13. Hao, Zhaojun & Di Maio, Francesco & Zio, Enrico, 2023. "A sequential decision problem formulation and deep reinforcement learning solution of the optimization of O&M of cyber-physical energy systems (CPESs) for reliable and safe power production and supply," Reliability Engineering and System Safety, Elsevier, vol. 235(C).
    14. Adedipe, Tosin & Shafiee, Mahmood & Zio, Enrico, 2020. "Bayesian Network Modelling for the Wind Energy Industry: An Overview," Reliability Engineering and System Safety, Elsevier, vol. 202(C).
    15. Dimitrios Vamvakas & Panagiotis Michailidis & Christos Korkas & Elias Kosmatopoulos, 2023. "Review and Evaluation of Reinforcement Learning Frameworks on Smart Grid Applications," Energies, MDPI, vol. 16(14), pages 1-38, July.
    16. Li, Mingxin & Jiang, Xiaoli & Carroll, James & Negenborn, Rudy R., 2023. "A closed-loop maintenance strategy for offshore wind farms: Incorporating dynamic wind farm states and uncertainty-awareness in decision-making," Renewable and Sustainable Energy Reviews, Elsevier, vol. 184(C).
    17. Kirstin Beyer & Robert Beckmann & Stefan Geißendörfer & Karsten von Maydell & Carsten Agert, 2021. "Adaptive Online-Learning Volt-Var Control for Smart Inverters Using Deep Reinforcement Learning," Energies, MDPI, vol. 14(7), pages 1-11, April.
    18. Biemann, Marco & Scheller, Fabian & Liu, Xiufeng & Huang, Lizhen, 2021. "Experimental evaluation of model-free reinforcement learning algorithms for continuous HVAC control," Applied Energy, Elsevier, vol. 298(C).
    19. Du, Yan & Zandi, Helia & Kotevska, Olivera & Kurte, Kuldeep & Munk, Jeffery & Amasyali, Kadir & Mckee, Evan & Li, Fangxing, 2021. "Intelligent multi-zone residential HVAC control strategy based on deep reinforcement learning," Applied Energy, Elsevier, vol. 281(C).
    20. Perera, A.T.D. & Kamalaruban, Parameswaran, 2021. "Applications of reinforcement learning in energy systems," Renewable and Sustainable Energy Reviews, Elsevier, vol. 137(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:14:y:2021:i:20:p:6743-:d:658112. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.