IDEAS home Printed from https://ideas.repec.org/a/eee/reensy/v234y2023ics0951832023000947.html
   My bibliography  Save this article

A deep reinforcement learning approach for repair-based maintenance of multi-unit systems using proportional hazards model

Author

Listed:
  • Najafi, Seyedvahid
  • Lee, Chi-Guhn

Abstract

Condition-based maintenance (CBM) optimization may turn intractable when a complex system with multiple units becomes an asset of interest. This paper aims to find a CBM policy for a multi-unit series system subject to stochastic degradation, where a new inspection is scheduled based on age and condition monitoring data upon each inspection. The novelty of this study lies in proposing a modified deep reinforcement learning (DRL) algorithm for the semi-Markov decision processes (SMDP) to find an opportunistic CBM policy for a multi-unit system with economic dependency over an infinite horizon, where a range of repair actions are allowed under an aperiodic inspection scheme. We also suggested a novel environment simulator that considers the simultaneous impact of age and covariates using the proportional hazards (PH) model and the system's reliability characteristics. DRL acts as not only a learning algorithm obviating the full specification of the model but also an approximate scheme producing a solution in a limited computation. The proposed algorithm is applied to a multi-unit hydroelectric power system with the damage self-healing property to demonstrate the higher performance of the DRL algorithm in cost reduction than alternative policies and explain how enhancing system reliability reduces costs during the learning process.

Suggested Citation

  • Najafi, Seyedvahid & Lee, Chi-Guhn, 2023. "A deep reinforcement learning approach for repair-based maintenance of multi-unit systems using proportional hazards model," Reliability Engineering and System Safety, Elsevier, vol. 234(C).
    • Handle: RePEc:eee:reensy:v:234:y:2023:i:c:s0951832023000947
    DOI: 10.1016/j.ress.2023.109179
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0951832023000947
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ress.2023.109179?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Ronald Martinod & Olivier Bistorin & Leonel Castañeda & Nidhal Rezg, 2018. "Maintenance policy optimisation for multi-component systems considering degradation of components and imperfect maintenance actions," Post-Print hal-01922272, HAL.
    2. Kijima, Masaaki & Morimura, Hidenori & Suzuki, Yasusuke, 1988. "Periodical replacement problem without assuming minimal repair," European Journal of Operational Research, Elsevier, vol. 37(2), pages 194-203, November.
    3. Nooshin Salari & Viliam Makis, 2020. "Application of Markov renewal theory and semi‐Markov decision processes in maintenance modeling and optimization of multi‐unit systems," Naval Research Logistics (NRL), John Wiley & Sons, vol. 67(7), pages 548-558, October.
    4. Pinciroli, Luca & Baraldi, Piero & Ballabio, Guido & Compare, Michele & Zio, Enrico, 2022. "Optimization of the Operation and Maintenance of renewable energy systems by Deep Reinforcement Learning," Renewable Energy, Elsevier, vol. 183(C), pages 752-763.
    5. Najafi, Seyedvahid & Zheng, Rui & Lee, Chi-Guhn, 2021. "An optimal opportunistic maintenance policy for a two-unit series system with general repair using proportional hazards models," Reliability Engineering and System Safety, Elsevier, vol. 215(C).
    6. Liu, Yu & Chen, Yiming & Jiang, Tao, 2020. "Dynamic selective maintenance optimization for multi-state systems over a finite horizon: A deep reinforcement learning approach," European Journal of Operational Research, Elsevier, vol. 283(1), pages 166-181.
    7. Wu, Fan & Niknam, Seyed A. & Kobza, John E., 2015. "A cost effective degradation-based maintenance strategy under imperfect repair," Reliability Engineering and System Safety, Elsevier, vol. 144(C), pages 234-243.
    8. Lee, Juseong & Mitici, Mihaela, 2023. "Deep reinforcement learning for predictive aircraft maintenance using probabilistic Remaining-Useful-Life prognostics," Reliability Engineering and System Safety, Elsevier, vol. 230(C).
    9. Zhang, Nailong & Si, Wujun, 2020. "Deep reinforcement learning for condition-based maintenance planning of multi-component systems under dependent competing risks," Reliability Engineering and System Safety, Elsevier, vol. 203(C).
    10. Rocchetta, R. & Bellani, L. & Compare, M. & Zio, E. & Patelli, E., 2019. "A reinforcement learning framework for optimal operation and maintenance of power grids," Applied Energy, Elsevier, vol. 241(C), pages 291-301.
    11. Volodymyr Mnih & Koray Kavukcuoglu & David Silver & Andrei A. Rusu & Joel Veness & Marc G. Bellemare & Alex Graves & Martin Riedmiller & Andreas K. Fidjeland & Georg Ostrovski & Stig Petersen & Charle, 2015. "Human-level control through deep reinforcement learning," Nature, Nature, vol. 518(7540), pages 529-533, February.
    12. Andriotis, C.P. & Papakonstantinou, K.G., 2019. "Managing engineering systems with large state and action spaces through deep reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 191(C).
    13. Mohammadi, Reza & He, Qing, 2022. "A deep reinforcement learning approach for rail renewal and maintenance planning," Reliability Engineering and System Safety, Elsevier, vol. 225(C).
    14. Nguyen, Khanh T. P. & Do, Phuc & Huynh, Khac Tuan & Bérenguer, Christophe & Grall, Antoine, 2019. "Joint optimization of monitoring quality and replacement decisions in condition-based maintenance," Reliability Engineering and System Safety, Elsevier, vol. 189(C), pages 177-195.
    15. Wijnmalen, Diederik J. D. & Hontelez, Jan A. M., 1997. "Coordinated condition-based repair strategies for components of a multi-component maintenance system with discounts," European Journal of Operational Research, Elsevier, vol. 98(1), pages 52-63, April.
    16. Leila Jafari & Farnoosh Naderkhani & Viliam Makis, 2018. "Joint optimization of maintenance policy and inspection interval for a multi-unit series system using proportional hazards model," Journal of the Operational Research Society, Taylor & Francis Journals, vol. 69(1), pages 36-48, January.
    17. Alaa H. Elwany & Nagi Z. Gebraeel & Lisa M. Maillart, 2011. "Structured Replacement Policies for Components with Complex Degradation Processes and Dedicated Sensors," Operations Research, INFORMS, vol. 59(3), pages 684-695, June.
    18. Fan, Lin & Su, Huai & Wang, Wei & Zio, Enrico & Zhang, Li & Yang, Zhaoming & Peng, Shiliang & Yu, Weichao & Zuo, Lili & Zhang, Jinjun, 2022. "A systematic method for the optimization of gas supply reliability in natural gas pipeline network based on Bayesian networks and deep reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 225(C).
    19. Flage, Roger & Coit, David W. & Luxhøj, James T. & Aven, Terje, 2012. "Safety constraints applied to an adaptive Bayesian condition-based maintenance optimization model," Reliability Engineering and System Safety, Elsevier, vol. 102(C), pages 16-26.
    20. Kampitsis, Dimitris & Panagiotidou, Sofia, 2022. "A Bayesian condition-based maintenance and monitoring policy with variable sampling intervals," Reliability Engineering and System Safety, Elsevier, vol. 218(PA).
    21. de Pater, Ingeborg & Mitici, Mihaela, 2021. "Predictive maintenance for multi-component systems of repairables with Remaining-Useful-Life prognostics and a limited stock of spare components," Reliability Engineering and System Safety, Elsevier, vol. 214(C).
    22. Andriotis, C.P. & Papakonstantinou, K.G., 2021. "Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints," Reliability Engineering and System Safety, Elsevier, vol. 212(C).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Anwar, Ghazanfar Ali & Zhang, Xiaoge, 2024. "Deep reinforcement learning for intelligent risk optimization of buildings under hazard," Reliability Engineering and System Safety, Elsevier, vol. 247(C).
    2. Yang, Sen & Zhang, Yi & Lu, Xinzheng & Guo, Wei & Miao, Huiquan, 2024. "Multi-agent deep reinforcement learning based decision support model for resilient community post-hazard recovery," Reliability Engineering and System Safety, Elsevier, vol. 242(C).
    3. Ferreira Neto, Waldomiro Alves & Virgínio Cavalcante, Cristiano Alexandre & Do, Phuc, 2024. "Deep reinforcement learning for maintenance optimization of a scrap-based steel production line," Reliability Engineering and System Safety, Elsevier, vol. 249(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tseremoglou, Iordanis & Santos, Bruno F., 2024. "Condition-Based Maintenance scheduling of an aircraft fleet under partial observability: A Deep Reinforcement Learning approach," Reliability Engineering and System Safety, Elsevier, vol. 241(C).
    2. Mohammadi, Reza & He, Qing, 2022. "A deep reinforcement learning approach for rail renewal and maintenance planning," Reliability Engineering and System Safety, Elsevier, vol. 225(C).
    3. Morato, P.G. & Andriotis, C.P. & Papakonstantinou, K.G. & Rigo, P., 2023. "Inference and dynamic decision-making for deteriorating systems with probabilistic dependencies through Bayesian networks and deep reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 235(C).
    4. Zhou, Yifan & Li, Bangcheng & Lin, Tian Ran, 2022. "Maintenance optimisation of multicomponent systems using hierarchical coordinated reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 217(C).
    5. Lee, Dongkyu & Song, Junho, 2023. "Risk-informed operation and maintenance of complex lifeline systems using parallelized multi-agent deep Q-network," Reliability Engineering and System Safety, Elsevier, vol. 239(C).
    6. Azar, Kamyar & Hajiakhondi-Meybodi, Zohreh & Naderkhani, Farnoosh, 2022. "Semi-supervised clustering-based method for fault diagnosis and prognosis: A case study," Reliability Engineering and System Safety, Elsevier, vol. 222(C).
    7. Lee, Juseong & Mitici, Mihaela, 2023. "Deep reinforcement learning for predictive aircraft maintenance using probabilistic Remaining-Useful-Life prognostics," Reliability Engineering and System Safety, Elsevier, vol. 230(C).
    8. Lee, Jun S. & Yeo, In-Ho & Bae, Younghoon, 2024. "A stochastic track maintenance scheduling model based on deep reinforcement learning approaches," Reliability Engineering and System Safety, Elsevier, vol. 241(C).
    9. Pinciroli, Luca & Baraldi, Piero & Zio, Enrico, 2023. "Maintenance optimization in industry 4.0," Reliability Engineering and System Safety, Elsevier, vol. 234(C).
    10. Liu, Hengchang & Li, Bo & Yao, Fengming & Hu, Gexi & Xie, Lei, 2024. "Maintenance optimization of multi-unit balanced systems using deep reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 244(C).
    11. Ye, Zhenggeng & Cai, Zhiqiang & Yang, Hui & Si, Shubin & Zhou, Fuli, 2023. "Joint optimization of maintenance and quality inspection for manufacturing networks based on deep reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 236(C).
    12. Nguyen, Van-Thai & Do, Phuc & Vosin, Alexandre & Iung, Benoit, 2022. "Artificial-intelligence-based maintenance decision-making and optimization for multi-state component systems," Reliability Engineering and System Safety, Elsevier, vol. 228(C).
    13. Liu, Yu & Chen, Yiming & Jiang, Tao, 2020. "Dynamic selective maintenance optimization for multi-state systems over a finite horizon: A deep reinforcement learning approach," European Journal of Operational Research, Elsevier, vol. 283(1), pages 166-181.
    14. Anwar, Ghazanfar Ali & Zhang, Xiaoge, 2024. "Deep reinforcement learning for intelligent risk optimization of buildings under hazard," Reliability Engineering and System Safety, Elsevier, vol. 247(C).
    15. Alaswad, Suzan & Xiang, Yisha, 2017. "A review on condition-based maintenance optimization models for stochastically deteriorating system," Reliability Engineering and System Safety, Elsevier, vol. 157(C), pages 54-63.
    16. Hao, Zhaojun & Di Maio, Francesco & Zio, Enrico, 2023. "A sequential decision problem formulation and deep reinforcement learning solution of the optimization of O&M of cyber-physical energy systems (CPESs) for reliable and safe power production and supply," Reliability Engineering and System Safety, Elsevier, vol. 235(C).
    17. Najafi, Seyedvahid & Zheng, Rui & Lee, Chi-Guhn, 2021. "An optimal opportunistic maintenance policy for a two-unit series system with general repair using proportional hazards models," Reliability Engineering and System Safety, Elsevier, vol. 215(C).
    18. Deep, Akash & Zhou, Shiyu & Veeramani, Dharmaraj & Chen, Yong, 2023. "Partially observable Markov decision process-based optimal maintenance planning with time-dependent observations," European Journal of Operational Research, Elsevier, vol. 311(2), pages 533-544.
    19. Ferreira Neto, Waldomiro Alves & Virgínio Cavalcante, Cristiano Alexandre & Do, Phuc, 2024. "Deep reinforcement learning for maintenance optimization of a scrap-based steel production line," Reliability Engineering and System Safety, Elsevier, vol. 249(C).
    20. Cheng, Jianda & Cheng, Minghui & Liu, Yan & Wu, Jun & Li, Wei & Frangopol, Dan M., 2024. "Knowledge transfer for adaptive maintenance policy optimization in engineering fleets based on meta-reinforcement learning," Reliability Engineering and System Safety, Elsevier, vol. 247(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:reensy:v:234:y:2023:i:c:s0951832023000947. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/reliability-engineering-and-system-safety .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.