IDEAS home Printed from https://ideas.repec.org/a/eee/reensy/v223y2022ics0951832022001351.html
   My bibliography  Save this article

Metrics for evaluating the performance of complex engineering system health monitoring models

Author

Listed:
  • Lewis, Austin D.
  • Groth, Katrina M.

Abstract

Recent efforts to apply prognostics and health management (PHM) practices towards the risk management of complex engineering systems (CES) traditionally structured using probabilistic risk assessments (PRA) provide more opportunities for modeling and dynamically monitoring system-level heath. However, it is unclear what metrics should be used to assess and compare the performance of these models. PHM performance metrics center around prediction accuracy without considering operational realities of the system, while risk management models are not easily compared. A new set of model metrics is needed for this evolving space.

Suggested Citation

  • Lewis, Austin D. & Groth, Katrina M., 2022. "Metrics for evaluating the performance of complex engineering system health monitoring models," Reliability Engineering and System Safety, Elsevier, vol. 223(C).
    • Handle: RePEc:eee:reensy:v:223:y:2022:i:c:s0951832022001351
    DOI: 10.1016/j.ress.2022.108473
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0951832022001351
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ress.2022.108473?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Zio, Enrico, 2022. "Prognostics and Health Management (PHM): Where are we and where do we (need to) go in theory and practice," Reliability Engineering and System Safety, Elsevier, vol. 218(PA).
    2. Zhou, Taotao & Modarres, Mohammad & Droguett, Enrique López, 2019. "Multi-unit risk aggregation with consideration of uncertainty and bias in risk metrics," Reliability Engineering and System Safety, Elsevier, vol. 188(C), pages 473-482.
    3. Vega, Manuel A. & Hu, Zhen & Fillmore, Travis B. & Smith, Matthew D. & Todd, Michael D., 2021. "A Novel Framework for Integration of Abstracted Inspection Data and Structural Health Monitoring for Damage Prognosis of Miter Gates," Reliability Engineering and System Safety, Elsevier, vol. 211(C).
    4. Jiao, Ruihua & Peng, Kaixiang & Dong, Jie & Zhang, Chuanfang, 2020. "Fault monitoring and remaining useful life prediction framework for multiple fault modes in prognostics," Reliability Engineering and System Safety, Elsevier, vol. 203(C).
    5. Katrina M Groth & Matthew R Denman & Michael C Darling & Thomas B Jones & George F Luger, 2020. "Building and using dynamic risk-informed diagnosis procedures for complex system accidents," Journal of Risk and Reliability, , vol. 234(1), pages 193-207, February.
    6. Arias Chao, Manuel & Kulkarni, Chetan & Goebel, Kai & Fink, Olga, 2022. "Fusing physics-based and deep learning models for prognostics," Reliability Engineering and System Safety, Elsevier, vol. 217(C).
    7. Xu, Zhaoyi & Saleh, Joseph Homer, 2021. "Machine learning for reliability engineering and safety applications: Review of current status and future opportunities," Reliability Engineering and System Safety, Elsevier, vol. 211(C).
    8. Gregory W. Vogl & Brian A. Weiss & Moneer Helu, 2019. "A review of diagnostic and prognostic capabilities and best practices for manufacturing," Journal of Intelligent Manufacturing, Springer, vol. 30(1), pages 79-95, January.
    9. Moradi, Ramin & Groth, Katrina M., 2020. "Modernizing risk assessment: A systematic integration of PRA and PHM techniques," Reliability Engineering and System Safety, Elsevier, vol. 204(C).
    10. Malings, C. & Pozzi, M., 2018. "Value-of-information in spatio-temporal systems: Sensor placement and scheduling," Reliability Engineering and System Safety, Elsevier, vol. 172(C), pages 45-57.
    11. Prabhakar V. Varde & Michael G. Pecht, 2018. "Prognostics and Health Management," Springer Series in Reliability Engineering, in: Risk-Based Engineering, chapter 0, pages 447-507, Springer.
    12. Rae, Andrew & Alexander, Rob & McDermid, John, 2014. "Fixing the cracks in the crystal ball: A maturity model for quantitative risk assessment," Reliability Engineering and System Safety, Elsevier, vol. 125(C), pages 67-81.
    13. Liu, Jie & Zio, Enrico, 2017. "System dynamic reliability assessment and failure prognostics," Reliability Engineering and System Safety, Elsevier, vol. 160(C), pages 21-36.
    14. Kim, Hyeonmin & Kim, Jung Taek & Heo, Gyunyoung, 2018. "Failure rate updates using condition-based prognostics in probabilistic safety assessments," Reliability Engineering and System Safety, Elsevier, vol. 175(C), pages 225-233.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Mandelli, Diego & Wang, Congjian & Agarwal, Vivek & Lin, Linyu & Manjunatha, Koushik A., 2024. "Reliability modeling in a predictive maintenance context: A margin-based approach," Reliability Engineering and System Safety, Elsevier, vol. 243(C).
    2. Kamariotis, Antonios & Tatsis, Konstantinos & Chatzi, Eleni & Goebel, Kai & Straub, Daniel, 2024. "A metric for assessing and optimizing data-driven prognostic algorithms for predictive maintenance," Reliability Engineering and System Safety, Elsevier, vol. 242(C).
    3. Lewis, Austin D. & Groth, Katrina M., 2023. "A comparison of DBN model performance in SIPPRA health monitoring based on different data stream discretization methods," Reliability Engineering and System Safety, Elsevier, vol. 236(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Xu, Yanwen & Kohtz, Sara & Boakye, Jessica & Gardoni, Paolo & Wang, Pingfeng, 2023. "Physics-informed machine learning for reliability and systems safety applications: State of the art and challenges," Reliability Engineering and System Safety, Elsevier, vol. 230(C).
    2. Lewis, Austin D. & Groth, Katrina M., 2023. "A comparison of DBN model performance in SIPPRA health monitoring based on different data stream discretization methods," Reliability Engineering and System Safety, Elsevier, vol. 236(C).
    3. Moradi, Ramin & Cofre-Martel, Sergio & Lopez Droguett, Enrique & Modarres, Mohammad & Groth, Katrina M., 2022. "Integration of deep learning and Bayesian networks for condition and operation risk monitoring of complex engineering systems," Reliability Engineering and System Safety, Elsevier, vol. 222(C).
    4. Gjorgiev, Blazhe & Das, Laya & Merkel, Seline & Rohrer, Martina & Auger, Etienne & Sansavini, Giovanni, 2023. "Simulation-driven deep learning for locating faulty insulators in a power line," Reliability Engineering and System Safety, Elsevier, vol. 231(C).
    5. Glavind, Sebastian T. & Sepulveda, Juan G. & Faber, Michael H., 2022. "On a simple scheme for systems modeling and identification using big data techniques," Reliability Engineering and System Safety, Elsevier, vol. 220(C).
    6. Xiong, Jiawei & Zhou, Jian & Ma, Yizhong & Zhang, Fengxia & Lin, Chenglong, 2023. "Adaptive deep learning-based remaining useful life prediction framework for systems with multiple failure patterns," Reliability Engineering and System Safety, Elsevier, vol. 235(C).
    7. Moradi, Ramin & Groth, Katrina M., 2020. "Modernizing risk assessment: A systematic integration of PRA and PHM techniques," Reliability Engineering and System Safety, Elsevier, vol. 204(C).
    8. Li, Yuanfu & Chen, Yao & Hu, Zhenchao & Zhang, Huisheng, 2023. "Remaining useful life prediction of aero-engine enabled by fusing knowledge and deep learning models," Reliability Engineering and System Safety, Elsevier, vol. 229(C).
    9. Costa, Nahuel & Sánchez, Luciano, 2022. "Variational encoding approach for interpretable assessment of remaining useful life estimation," Reliability Engineering and System Safety, Elsevier, vol. 222(C).
    10. Zheng, Shuwen & Wang, Chong & Zio, Enrico & Liu, Jie, 2024. "Fault detection in complex mechatronic systems by a hierarchical graph convolution attention network based on causal paths," Reliability Engineering and System Safety, Elsevier, vol. 243(C).
    11. Zaitseva, Elena & Levashenko, Vitaly & Rabcan, Jan, 2023. "A new method for analysis of Multi-State systems based on Multi-valued decision diagram under epistemic uncertainty," Reliability Engineering and System Safety, Elsevier, vol. 229(C).
    12. Lyu, Dongzhen & Niu, Guangxing & Liu, Enhui & Zhang, Bin & Chen, Gang & Yang, Tao & Zio, Enrico, 2022. "Time space modelling for fault diagnosis and prognosis with uncertainty management: A general theoretical formulation," Reliability Engineering and System Safety, Elsevier, vol. 226(C).
    13. Zou, Xinyu & Tao, Laifa & Sun, Lulu & Wang, Chao & Ma, Jian & Lu, Chen, 2023. "A case-learning-based paradigm for quantitative recommendation of fault diagnosis algorithms: A case study of gearbox," Reliability Engineering and System Safety, Elsevier, vol. 237(C).
    14. Vrignat, Pascal & Kratz, Frédéric & Avila, Manuel, 2022. "Sustainable manufacturing, maintenance policies, prognostics and health management: A literature review," Reliability Engineering and System Safety, Elsevier, vol. 218(PA).
    15. He, Yuxuan & Su, Huai & Zio, Enrico & Peng, Shiliang & Fan, Lin & Yang, Zhaoming & Yang, Zhe & Zhang, Jinjun, 2023. "A systematic method of remaining useful life estimation based on physics-informed graph neural networks with multisensor data," Reliability Engineering and System Safety, Elsevier, vol. 237(C).
    16. Mandelli, Diego & Wang, Congjian & Agarwal, Vivek & Lin, Linyu & Manjunatha, Koushik A., 2024. "Reliability modeling in a predictive maintenance context: A margin-based approach," Reliability Engineering and System Safety, Elsevier, vol. 243(C).
    17. Wang, Yilin & Li, Yuanxiang & Zhang, Yuxuan & Lei, Jia & Yu, Yifei & Zhang, Tongtong & Yang, Yongshen & Zhao, Honghua, 2024. "Incorporating prior knowledge into self-supervised representation learning for long PHM signal," Reliability Engineering and System Safety, Elsevier, vol. 241(C).
    18. Pan, Yongjun & Sun, Yu & Li, Zhixiong & Gardoni, Paolo, 2023. "Machine learning approaches to estimate suspension parameters for performance degradation assessment using accurate dynamic simulations," Reliability Engineering and System Safety, Elsevier, vol. 230(C).
    19. Liu, Lu & Song, Xiao & Zhou, Zhetao, 2022. "Aircraft engine remaining useful life estimation via a double attention-based data-driven architecture," Reliability Engineering and System Safety, Elsevier, vol. 221(C).
    20. Wang, Chu & Dou, Manfeng & Li, Zhongliang & Outbib, Rachid & Zhao, Dongdong & Zuo, Jian & Wang, Yuanlin & Liang, Bin & Wang, Peng, 2023. "Data-driven prognostics based on time-frequency analysis and symbolic recurrent neural network for fuel cells under dynamic load," Reliability Engineering and System Safety, Elsevier, vol. 233(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:reensy:v:223:y:2022:i:c:s0951832022001351. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/reliability-engineering-and-system-safety .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.