IDEAS home Printed from https://ideas.repec.org/a/eee/reensy/v252y2024ics0951832024005386.html
   My bibliography  Save this article

A reinforcement learning agent for maintenance of deteriorating systems with increasingly imperfect repairs

Author

Listed:
  • Pliego Marugán, Alberto
  • Pinar-Pérez, Jesús M.
  • García Márquez, Fausto Pedro

Abstract

Efficient maintenance has always been essential for the successful application of engineering systems. However, the challenges to be overcome in the implementation of Industry 4.0 necessitate new paradigms of maintenance optimization. Machine learning techniques are becoming increasingly used in engineering and maintenance, with reinforcement learning being one of the most promising. In this paper, we propose a gamma degradation process together with a novel maintenance model in which repairs are increasingly imperfect, i.e., the beneficial effect of system repairs decreases as more repairs are performed, reflecting the degradational behavior of real-world systems. To generate maintenance policies for this system, we developed a reinforcement-learning-based agent using a Double Deep Q-Network architecture. This agent presents two important advantages: it works without a predefined preventive threshold, and it can operate in a continuous degradation state space. Our agent learns to behave in different scenarios, showing great flexibility. In addition, we performed an analysis of how changes in the main parameters of the environment affect the maintenance policy proposed by the agent. The proposed approach is demonstrated to be appropriate and to significatively improve long-run cost as compared with other common maintenance strategies.

Suggested Citation

  • Pliego Marugán, Alberto & Pinar-Pérez, Jesús M. & García Márquez, Fausto Pedro, 2024. "A reinforcement learning agent for maintenance of deteriorating systems with increasingly imperfect repairs," Reliability Engineering and System Safety, Elsevier, vol. 252(C).
  • Handle: RePEc:eee:reensy:v:252:y:2024:i:c:s0951832024005386
    DOI: 10.1016/j.ress.2024.110466
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0951832024005386
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.ress.2024.110466?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Wang, Jun & Zhu, Xiaoyan, 2021. "Joint optimization of condition-based maintenance and inventory control for a k-out-of-n:F system of multi-state degrading components," European Journal of Operational Research, Elsevier, vol. 290(2), pages 514-529.
    2. Zhang, Mimi & Gaudoin, Olivier & Xie, Min, 2015. "Degradation-based maintenance decision using stochastic filtering for systems under imperfect maintenance," European Journal of Operational Research, Elsevier, vol. 245(2), pages 531-541.
    3. Wang, Hongzhou, 2002. "A survey of maintenance policies of deteriorating systems," European Journal of Operational Research, Elsevier, vol. 139(3), pages 469-489, June.
    4. Xiao Wang & Hongwei Wang & Chao Qi, 2016. "Multi-agent reinforcement learning based maintenance policy for a resource constrained flow line system," Journal of Intelligent Manufacturing, Springer, vol. 27(2), pages 325-333, April.
    5. Do, Phuc & Voisin, Alexandre & Levrat, Eric & Iung, Benoit, 2015. "A proactive condition-based maintenance strategy with both perfect and imperfect maintenance actions," Reliability Engineering and System Safety, Elsevier, vol. 133(C), pages 22-32.
    6. Zhao, Yunfei & Smidts, Carol, 2022. "Reinforcement learning for adaptive maintenance policy optimization under imperfect knowledge of the system degradation model and partial observability of system states," Reliability Engineering and System Safety, Elsevier, vol. 224(C).
    7. Shahraki, Ameneh Forouzandeh & Yadav, Om Prakash & Vogiatzis, Chrysafis, 2020. "Selective maintenance optimization for multi-state systems considering stochastically dependent components and stochastic imperfect maintenance actions," Reliability Engineering and System Safety, Elsevier, vol. 196(C).
    8. Alberto Pliego Marugán & Fausto Pedro García Márquez & Jesús María Pinar Pérez, 2016. "Optimal Maintenance Management of Offshore Wind Farms," Energies, MDPI, vol. 9(1), pages 1-20, January.
    9. Zhang, Nailong & Si, Wujun, 2020. "Deep reinforcement learning for condition-based maintenance planning of multi-component systems under dependent competing risks," Reliability Engineering and System Safety, Elsevier, vol. 203(C).
    10. Rocchetta, R. & Bellani, L. & Compare, M. & Zio, E. & Patelli, E., 2019. "A reinforcement learning framework for optimal operation and maintenance of power grids," Applied Energy, Elsevier, vol. 241(C), pages 291-301.
    11. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    12. van Noortwijk, J.M., 2009. "A survey of the application of gamma processes in maintenance," Reliability Engineering and System Safety, Elsevier, vol. 94(1), pages 2-21.
    13. Cheng, Wanqing & Zhao, Xiujie, 2023. "Maintenance optimization for dependent two-component degrading systems subject to imperfect repair," Reliability Engineering and System Safety, Elsevier, vol. 240(C).
    14. Riccardo Manzini & Alberto Regattieri & Hoang Pham & Emilio Ferrari, 2010. "Maintenance for Industrial Systems," Springer Series in Reliability Engineering, Springer, number 978-1-84882-575-8, April.
    15. Andriotis, C.P. & Papakonstantinou, K.G., 2021. "Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints," Reliability Engineering and System Safety, Elsevier, vol. 212(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lee, Juseong & Mitici, Mihaela, 2023. "Deep reinforcement learning for predictive aircraft maintenance using probabilistic Remaining-Useful-Life prognostics," Reliability Engineering and System Safety, Elsevier, vol. 230(C).
    2. Mohammadi, Reza & He, Qing, 2022. "A deep reinforcement learning approach for rail renewal and maintenance planning," Reliability Engineering and System Safety, Elsevier, vol. 225(C).
    3. Nguyen, Van-Thai & Do, Phuc & Vosin, Alexandre & Iung, Benoit, 2022. "Artificial-intelligence-based maintenance decision-making and optimization for multi-state component systems," Reliability Engineering and System Safety, Elsevier, vol. 228(C).
    4. Alaswad, Suzan & Xiang, Yisha, 2017. "A review on condition-based maintenance optimization models for stochastically deteriorating system," Reliability Engineering and System Safety, Elsevier, vol. 157(C), pages 54-63.
    5. Pinciroli, Luca & Baraldi, Piero & Zio, Enrico, 2023. "Maintenance optimization in industry 4.0," Reliability Engineering and System Safety, Elsevier, vol. 234(C).
    6. Ye, Zhenggeng & Cai, Zhiqiang & Yang, Hui & Si, Shubin & Zhou, Fuli, 2023. "Joint optimization of maintenance and quality inspection for manufacturing networks based on deep reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 236(C).
    7. Mosayebi Omshi, E. & Grall, A. & Shemehsavar, S., 2020. "A dynamic auto-adaptive predictive maintenance policy for degradation with unknown parameters," European Journal of Operational Research, Elsevier, vol. 282(1), pages 81-92.
    8. Najafi, Seyedvahid & Lee, Chi-Guhn, 2023. "A deep reinforcement learning approach for repair-based maintenance of multi-unit systems using proportional hazards model," Reliability Engineering and System Safety, Elsevier, vol. 234(C).
    9. Pedersen, Tom Ivar & Liu, Xingheng & Vatn, Jørn, 2023. "Maintenance optimization of a system subject to two-stage degradation, hard failure, and imperfect repair," Reliability Engineering and System Safety, Elsevier, vol. 237(C).
    10. Zhang, Qin & Liu, Yu & Xiang, Yisha & Xiahou, Tangfan, 2024. "Reinforcement learning in reliability and maintenance optimization: A tutorial," Reliability Engineering and System Safety, Elsevier, vol. 251(C).
    11. de Jonge, Bram & Scarf, Philip A., 2020. "A review on maintenance optimization," European Journal of Operational Research, Elsevier, vol. 285(3), pages 805-824.
    12. Zheng, Meimei & Su, Zhiyun & Wang, Dong & Pan, Ershun, 2024. "Joint maintenance and spare part ordering from multiple suppliers for multicomponent systems using a deep reinforcement learning algorithm," Reliability Engineering and System Safety, Elsevier, vol. 241(C).
    13. Tseremoglou, Iordanis & Santos, Bruno F., 2024. "Condition-Based Maintenance scheduling of an aircraft fleet under partial observability: A Deep Reinforcement Learning approach," Reliability Engineering and System Safety, Elsevier, vol. 241(C).
    14. Mikhail, Mina & Ouali, Mohamed-Salah & Yacout, Soumaya, 2024. "A data-driven methodology with a nonparametric reliability method for optimal condition-based maintenance strategies," Reliability Engineering and System Safety, Elsevier, vol. 241(C).
    15. Liu, Biyu & Pang, Jie & Yang, Haidong & Zhao, Yilin, 2024. "Optimal condition-based maintenance policy for leased equipment considering hybrid preventive maintenance and periodic inspection," Reliability Engineering and System Safety, Elsevier, vol. 242(C).
    16. Zhang, Wenyu & Zhang, Xiaohong & He, Shuguang & Zhao, Xing & He, Zhen, 2024. "Optimal condition-based maintenance policy for multi-component repairable systems with economic dependence in a finite-horizon," Reliability Engineering and System Safety, Elsevier, vol. 241(C).
    17. Kamariotis, Antonios & Tatsis, Konstantinos & Chatzi, Eleni & Goebel, Kai & Straub, Daniel, 2024. "A metric for assessing and optimizing data-driven prognostic algorithms for predictive maintenance," Reliability Engineering and System Safety, Elsevier, vol. 242(C).
    18. Liu, Hengchang & Li, Bo & Yao, Fengming & Hu, Gexi & Xie, Lei, 2024. "Maintenance optimization of multi-unit balanced systems using deep reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 244(C).
    19. Lee, Dongkyu & Song, Junho, 2023. "Risk-informed operation and maintenance of complex lifeline systems using parallelized multi-agent deep Q-network," Reliability Engineering and System Safety, Elsevier, vol. 239(C).
    20. Ji Hwan Cha & Maxim Finkelstein & Gregory Levitin, 2022. "Replacement Policy for Heterogeneous Items Subject to Gamma Degradation Processes," Methodology and Computing in Applied Probability, Springer, vol. 24(3), pages 1323-1340, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:reensy:v:252:y:2024:i:c:s0951832024005386. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/reliability-engineering-and-system-safety .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.