IDEAS home Printed from https://ideas.repec.org/a/eee/reensy/v226y2022ics0951832022003167.html
   My bibliography  Save this article

A predictive Markov decision process for optimizing inspection and maintenance strategies of partially observable multi-state systems

Author

Listed:
  • Guo, Chunhui
  • Liang, Zhenglin

Abstract

Optimizing both inspection and maintenance strategies for multi-state systems is challenging, especially when the inspected conditions contain uncertainties. One classic approach for addressing this issue is the Partially Observable Markov Decision Process (POMDP). However, the POMDP often considers the system is periodically inspected, resulting in a waste of inspection resources (cost and manpower) in the early stage of the system. To predictively optimize the inspection strategies, we formulate a new model-Predictive Markov Decision Process (PMDP). It extends the POMDP by embedding the Forward algorithm for inspection timing prediction and the Baum–Welch algorithm for model parameters estimation. Therefore, it could harvest the inspection information for predicting the successive inspection timing in an online updating scheme based on the new observation. In this manner, maintenance actions can take place at the predicted inspection timing that reduces unnecessary inspections. The PMDP manifests the power of predictive maintenance. As illustrated by the case study, the PMDP outperforms the POMDP under routine inspection by saving 26.3% of the cost on average.

Suggested Citation

  • Guo, Chunhui & Liang, Zhenglin, 2022. "A predictive Markov decision process for optimizing inspection and maintenance strategies of partially observable multi-state systems," Reliability Engineering and System Safety, Elsevier, vol. 226(C).
    • Handle: RePEc:eee:reensy:v:226:y:2022:i:c:s0951832022003167
    DOI: 10.1016/j.ress.2022.108683
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0951832022003167
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ress.2022.108683?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Chen, Zhen & Li, Yaping & Xia, Tangbin & Pan, Ershun, 2019. "Hidden Markov model with auto-correlated observations for remaining useful life prediction and optimal maintenance policy," Reliability Engineering and System Safety, Elsevier, vol. 184(C), pages 123-136.
    2. Sheu, Shey-Huei & Chang, Chin-Chih & Chen, Yen-Luan & George Zhang, Zhe, 2015. "Optimal preventive maintenance and repair policies for multi-state systems," Reliability Engineering and System Safety, Elsevier, vol. 140(C), pages 78-87.
    3. Richard D. Smallwood & Edward J. Sondik, 1973. "The Optimal Control of Partially Observable Markov Processes over a Finite Horizon," Operations Research, INFORMS, vol. 21(5), pages 1071-1088, October.
    4. Li, Ruopu & Arzaghi, Ehsan & Abbassi, Rouzbeh & Chen, Diyi & Li, Chunhao & Li, Huanhuan & Xu, Beibei, 2020. "Dynamic maintenance planning of a hydro-turbine in operational life cycle," Reliability Engineering and System Safety, Elsevier, vol. 204(C).
    5. Nguyen, Khanh T. P. & Do, Phuc & Huynh, Khac Tuan & Bérenguer, Christophe & Grall, Antoine, 2019. "Joint optimization of monitoring quality and replacement decisions in condition-based maintenance," Reliability Engineering and System Safety, Elsevier, vol. 189(C), pages 177-195.
    6. Barker, C.T. & Newby, M.J., 2009. "Optimal non-periodic inspection for a multivariate degradation model," Reliability Engineering and System Safety, Elsevier, vol. 94(1), pages 33-43.
    7. Rebello, Sinda & Yu, Hongyang & Ma, Lin, 2018. "An integrated approach for system functional reliability assessment using Dynamic Bayesian Network and Hidden Markov Model," Reliability Engineering and System Safety, Elsevier, vol. 180(C), pages 124-135.
    8. Song, Chaolin & Zhang, Chi & Shafieezadeh, Abdollah & Xiao, Rucheng, 2022. "Value of information analysis in non-stationary stochastic decision environments: A reliability-assisted POMDP approach," Reliability Engineering and System Safety, Elsevier, vol. 217(C).
    9. Soleimani, Morteza & Campean, Felician & Neagu, Daniel, 2021. "Integration of Hidden Markov Modelling and Bayesian Network for fault detection and prediction of complex engineered systems," Reliability Engineering and System Safety, Elsevier, vol. 215(C).
    10. Do, Phuc & Voisin, Alexandre & Levrat, Eric & Iung, Benoit, 2015. "A proactive condition-based maintenance strategy with both perfect and imperfect maintenance actions," Reliability Engineering and System Safety, Elsevier, vol. 133(C), pages 22-32.
    11. Compare, Michele & Baraldi, Piero & Marelli, Paolo & Zio, Enrico, 2020. "Partially observable Markov decision processes for optimal operations of gas transmission networks," Reliability Engineering and System Safety, Elsevier, vol. 199(C).
    12. Nguyen, Kim-Anh & Do, Phuc & Grall, Antoine, 2015. "Multi-level predictive maintenance for multi-component systems," Reliability Engineering and System Safety, Elsevier, vol. 144(C), pages 83-94.
    13. Eslam Mohammed Abdelkader & Tarek Zayed & Mohamed Marzouk, 2019. "Modelling the Deterioration of Bridge Decks Based on Semi-Markov Decision Process," International Journal of Strategic Decision Sciences (IJSDS), IGI Global, vol. 10(1), pages 23-45, January.
    14. Koutras, V.P. & Malefaki, S. & Platis, A.N., 2017. "Optimization of the dependability and performance measures of a generic model for multi-state deteriorating systems under maintenance," Reliability Engineering and System Safety, Elsevier, vol. 166(C), pages 73-86.
    15. Liu, Xingchen & Sun, Qiuzhuang & Ye, Zhi-Sheng & Yildirim, Murat, 2021. "Optimal multi-type inspection policy for systems with imperfect online monitoring," Reliability Engineering and System Safety, Elsevier, vol. 207(C).
    16. Michael Jong Kim, 2016. "Robust Control of Partially Observable Failing Systems," Operations Research, INFORMS, vol. 64(4), pages 999-1014, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Finkelstein, Maxim & Cha, Ji Hwan & Bedford, Tim, 2023. "Optimal preventive maintenance strategy for populations of systems that generate outputs," Reliability Engineering and System Safety, Elsevier, vol. 237(C).
    2. Finkelstein, Maxim & Cha, Ji Hwan & Langston, Amy, 2023. "Improving classical optimal age-replacement policies for degrading items," Reliability Engineering and System Safety, Elsevier, vol. 236(C).
    3. Karabağ, Oktay & Bulut, Önder & Toy, Ayhan Özgür & Fadıloğlu, Mehmet Murat, 2024. "An efficient procedure for optimal maintenance intervention in partially observable multi-component systems," Reliability Engineering and System Safety, Elsevier, vol. 244(C).
    4. Wang, Siqi & Zhao, Xian & Wu, Congshan & Wang, Xiaoyue, 2023. "Joint optimization of multi-stage component reassignment and preventive maintenance for balanced systems considering imperfect maintenance," Reliability Engineering and System Safety, Elsevier, vol. 237(C).
    5. Gámiz, M.L. & Navas-Gómez, F. & Raya-Miranda, R. & Segovia-García, M.C., 2023. "Dynamic reliability and sensitivity analysis based on HMM models with Markovian signal process," Reliability Engineering and System Safety, Elsevier, vol. 239(C).
    6. Cha, Ji Hwan & Finkelstein, Maxim, 2024. "Preventive maintenance for the constrained multi-attempt minimal repair," Reliability Engineering and System Safety, Elsevier, vol. 243(C).
    7. Arcieri, Giacomo & Hoelzl, Cyprien & Schwery, Oliver & Straub, Daniel & Papakonstantinou, Konstantinos G. & Chatzi, Eleni, 2023. "Bridging POMDPs and Bayesian decision making for robust maintenance planning under model uncertainty: An application to railway systems," Reliability Engineering and System Safety, Elsevier, vol. 239(C).
    8. Sun, Qin & Li, Hongxu & Zhong, Yuanfu & Ren, Kezhou & Zhang, Yingchao, 2024. "Deep reinforcement learning-based resilience enhancement strategy of unmanned weapon system-of-systems under inevitable interferences," Reliability Engineering and System Safety, Elsevier, vol. 242(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. de Jonge, Bram & Scarf, Philip A., 2020. "A review on maintenance optimization," European Journal of Operational Research, Elsevier, vol. 285(3), pages 805-824.
    2. Alaswad, Suzan & Xiang, Yisha, 2017. "A review on condition-based maintenance optimization models for stochastically deteriorating system," Reliability Engineering and System Safety, Elsevier, vol. 157(C), pages 54-63.
    3. Tseremoglou, Iordanis & Santos, Bruno F., 2024. "Condition-Based Maintenance scheduling of an aircraft fleet under partial observability: A Deep Reinforcement Learning approach," Reliability Engineering and System Safety, Elsevier, vol. 241(C).
    4. Zhao, Yunfei & Smidts, Carol, 2022. "Reinforcement learning for adaptive maintenance policy optimization under imperfect knowledge of the system degradation model and partial observability of system states," Reliability Engineering and System Safety, Elsevier, vol. 224(C).
    5. Gámiz, M.L. & Navas-Gómez, F. & Raya-Miranda, R. & Segovia-García, M.C., 2023. "Dynamic reliability and sensitivity analysis based on HMM models with Markovian signal process," Reliability Engineering and System Safety, Elsevier, vol. 239(C).
    6. Aizpurua, J.I. & Catterson, V.M. & Papadopoulos, Y. & Chiacchio, F. & D'Urso, D., 2017. "Supporting group maintenance through prognostics-enhanced dynamic dependability prediction," Reliability Engineering and System Safety, Elsevier, vol. 168(C), pages 171-188.
    7. Dinh, Duc-Hanh & Do, Phuc & Hoang, Van-Thanh & Vo, Nhu-Thanh & Bang, Tao Quang, 2024. "A predictive maintenance policy for manufacturing systems considering degradation of health monitoring device," Reliability Engineering and System Safety, Elsevier, vol. 248(C).
    8. Liu, Gehui & Chen, Shaokuan & Ho, Tinkin & Ran, Xinchen & Mao, Baohua & Lan, Zhen, 2022. "Optimum opportunistic maintenance schedule over variable horizons considering multi-stage degradation and dynamic strategy," Reliability Engineering and System Safety, Elsevier, vol. 225(C).
    9. Nguyen, Kim-Anh & Do, Phuc & Grall, Antoine, 2017. "Joint predictive maintenance and inventory strategy for multi-component systems using Birnbaum’s structural importance," Reliability Engineering and System Safety, Elsevier, vol. 168(C), pages 249-261.
    10. Lam, Ji Ye Janet & Banjevic, Dragan, 2015. "A myopic policy for optimal inspection scheduling for condition based maintenance," Reliability Engineering and System Safety, Elsevier, vol. 144(C), pages 1-11.
    11. Xinlong Li & Yan Ran & Fangming Wan & Hui Yu & Genbao Zhang & Yan He, 2022. "Condition-based maintenance strategy optimization of meta-action unit considering imperfect preventive maintenance based on Wiener process," Flexible Services and Manufacturing Journal, Springer, vol. 34(1), pages 204-233, March.
    12. Deep, Akash & Zhou, Shiyu & Veeramani, Dharmaraj & Chen, Yong, 2023. "Partially observable Markov decision process-based optimal maintenance planning with time-dependent observations," European Journal of Operational Research, Elsevier, vol. 311(2), pages 533-544.
    13. Nguyen, Khanh T. P. & Do, Phuc & Huynh, Khac Tuan & Bérenguer, Christophe & Grall, Antoine, 2019. "Joint optimization of monitoring quality and replacement decisions in condition-based maintenance," Reliability Engineering and System Safety, Elsevier, vol. 189(C), pages 177-195.
    14. Zhang, Zhengxin & Si, Xiaosheng & Hu, Changhua & Lei, Yaguo, 2018. "Degradation data analysis and remaining useful life estimation: A review on Wiener-process-based methods," European Journal of Operational Research, Elsevier, vol. 271(3), pages 775-796.
    15. Faddoul, R. & Raphael, W. & Chateauneuf, A., 2018. "Maintenance optimization of series systems subject to reliability constraints," Reliability Engineering and System Safety, Elsevier, vol. 180(C), pages 179-188.
    16. Do, Phuc & Assaf, Roy & Scarf, Phil & Iung, Benoit, 2019. "Modelling and application of condition-based maintenance for a two-component system with stochastic and economic dependencies," Reliability Engineering and System Safety, Elsevier, vol. 182(C), pages 86-97.
    17. Yaping Li & Enrico Zio & Ershun Pan, 2021. "An MEWMA-based segmental multivariate hidden Markov model for degradation assessment and prediction," Journal of Risk and Reliability, , vol. 235(5), pages 831-844, October.
    18. Song, Chaolin & Zhang, Chi & Shafieezadeh, Abdollah & Xiao, Rucheng, 2022. "Value of information analysis in non-stationary stochastic decision environments: A reliability-assisted POMDP approach," Reliability Engineering and System Safety, Elsevier, vol. 217(C).
    19. Vališ, David & Žák, Libor & Pokora, Ondřej & Lánský, Petr, 2016. "Perspective analysis outcomes of selected tribodiagnostic data used as input for condition based maintenance," Reliability Engineering and System Safety, Elsevier, vol. 145(C), pages 231-242.
    20. Memarzadeh, Milad & Pozzi, Matteo & Kolter, J. Zico, 2016. "Hierarchical modeling of systems with similar components: A framework for adaptive monitoring and control," Reliability Engineering and System Safety, Elsevier, vol. 153(C), pages 159-169.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:reensy:v:226:y:2022:i:c:s0951832022003167. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/reliability-engineering-and-system-safety .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.