IDEAS home Printed from https://ideas.repec.org/a/eee/reensy/v180y2018icp201-210.html
   My bibliography  Save this article

Distinguishing between model- and data-driven inferences for high reliability statistical predictions

Author

Listed:
  • Hund, Lauren
  • Schroeder, Benjamin
  • Rumsey, Kellin
  • Huerta, Gabriel

Abstract

Estimating the tails of probability distributions plays a key role in complex engineering systems where the goal is characterization of low probability, high consequence events. When data are collected using physical experimentation, statistical distributional assumptions are often used to extrapolate tail behavior to assess reliability, introducing risk due to extrapolation from an unvalidated (statistical) model. Existing tools to evaluate statistical model fit, such as probability plots and goodness of fit tests, fail to communicate the risk associated with this extrapolation. In this work, we develop a new statistical model validation metric and relate this metric to engineering-driven model validation metrics. The metric measures how consistent the parametric tail estimates are with a more flexible model that makes weaker assumptions about the distribution tails. An extreme-value based generalized Pareto distribution is used for the more flexible model. Models are updated using a Bayesian inference procedure that defaults to reasonably conservative inferences when data are sparse. Properties of the estimation procedure are evaluated in statistical simulation, and the effectiveness of the proposed metrics relative to the standard-of-practice statistical metrics is illustrated using a pedagogical example related to a real, but proprietary, engineering example.

Suggested Citation

  • Hund, Lauren & Schroeder, Benjamin & Rumsey, Kellin & Huerta, Gabriel, 2018. "Distinguishing between model- and data-driven inferences for high reliability statistical predictions," Reliability Engineering and System Safety, Elsevier, vol. 180(C), pages 201-210.
    • Handle: RePEc:eee:reensy:v:180:y:2018:i:c:p:201-210
    DOI: 10.1016/j.ress.2018.07.017
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0951832018301534
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ress.2018.07.017?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. M. Ivette Gomes & Armelle Guillou, 2015. "Extreme Value Theory and Statistics of Univariate Extremes: A Review," International Statistical Review, International Statistical Institute, vol. 83(2), pages 263-292, August.
    2. Su, Steve, 2009. "Confidence intervals for quantiles using generalized lambda distributions," Computational Statistics & Data Analysis, Elsevier, vol. 53(9), pages 3324-3333, July.
    3. Derek S. Young & Thomas Mathew, 2014. "Improved nonparametric tolerance intervals based on interpolated and extrapolated order statistics," Journal of Nonparametric Statistics, Taylor & Francis Journals, vol. 26(3), pages 415-432, September.
    4. Wallstrom, Timothy C., 2011. "Quantification of margins and uncertainties: A probabilistic framework," Reliability Engineering and System Safety, Elsevier, vol. 96(9), pages 1053-1062.
    5. Pilch, Martin & Trucano, Timothy G. & Helton, Jon C., 2011. "Ideas underlying the Quantification of Margins and Uncertainties," Reliability Engineering and System Safety, Elsevier, vol. 96(9), pages 965-975.
    6. Lee, Dongjin & Pan, Rong, 2018. "A nonparametric Bayesian network approach to assessing system reliability at early design stages," Reliability Engineering and System Safety, Elsevier, vol. 171(C), pages 57-66.
    7. Mullins, Joshua & Ling, You & Mahadevan, Sankaran & Sun, Lin & Strachan, Alejandro, 2016. "Separation of aleatory and epistemic uncertainty in probabilistic model validation," Reliability Engineering and System Safety, Elsevier, vol. 147(C), pages 49-59.
    8. Yacov Y. Haimes & James H. Lambert, 1999. "When and How Can You Specify a Probability Distribution When You Don't Know Much? II," Risk Analysis, John Wiley & Sons, vol. 19(1), pages 43-46, February.
    9. Rebba, Ramesh & Mahadevan, Sankaran, 2008. "Computational methods for model reliability assessment," Reliability Engineering and System Safety, Elsevier, vol. 93(8), pages 1197-1207.
    10. Helton, Jon C., 2011. "Quantification of margins and uncertainties: Conceptual and computational basis," Reliability Engineering and System Safety, Elsevier, vol. 96(9), pages 976-1013.
    11. Alan Hutson, 1999. "Calculating nonparametric confidence intervals for quantiles using fractional order statistics," Journal of Applied Statistics, Taylor & Francis Journals, vol. 26(3), pages 343-353.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Hund, Lauren & Schroeder, Benjamin, 2020. "A causal perspective on reliability assessment," Reliability Engineering and System Safety, Elsevier, vol. 195(C).
    2. Dai, Baorui & Xia, Ye & Li, Qi, 2022. "An extreme value prediction method based on clustering algorithm," Reliability Engineering and System Safety, Elsevier, vol. 222(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Helton, Jon C. & Brooks, Dusty M. & Sallaberry, Cédric J., 2020. "Property values associated with the failure of individual links in a system with multiple weak and strong links," Reliability Engineering and System Safety, Elsevier, vol. 195(C).
    2. Shah, Harsheel & Hosder, Serhat & Winter, Tyler, 2015. "Quantification of margins and mixed uncertainties using evidence theory and stochastic expansions," Reliability Engineering and System Safety, Elsevier, vol. 138(C), pages 59-72.
    3. Helton, Jon C. & Brooks, Dusty M. & Sallaberry, Cédric J., 2020. "Margins associated with loss of assured safety for systems with multiple weak links and strong links," Reliability Engineering and System Safety, Elsevier, vol. 195(C).
    4. Vanslette, Kevin & Tohme, Tony & Youcef-Toumi, Kamal, 2020. "A general model validation and testing tool," Reliability Engineering and System Safety, Elsevier, vol. 195(C).
    5. Iaccarino, Gianluca & Sharp, David & Glimm, James, 2013. "Quantification of margins and uncertainties using multiple gates and conditional probabilities," Reliability Engineering and System Safety, Elsevier, vol. 114(C), pages 99-113.
    6. Ao, Dan & Hu, Zhen & Mahadevan, Sankaran, 2017. "Design of validation experiments for life prediction models," Reliability Engineering and System Safety, Elsevier, vol. 165(C), pages 22-33.
    7. Sankararaman, S. & Mahadevan, S., 2013. "Separating the contributions of variability and parameter uncertainty in probability distributions," Reliability Engineering and System Safety, Elsevier, vol. 112(C), pages 187-199.
    8. Hund, Lauren & Schroeder, Benjamin, 2020. "A causal perspective on reliability assessment," Reliability Engineering and System Safety, Elsevier, vol. 195(C).
    9. Di Maio, Francesco & Bandini, Alessandro & Zio, Enrico & Alberola, Sofia Carlos & Sanchez-Saez, Francisco & Martorell, Sebastián, 2016. "Bootstrapped-ensemble-based Sensitivity Analysis of a trace thermal-hydraulic model based on a limited number of PWR large break loca simulations," Reliability Engineering and System Safety, Elsevier, vol. 153(C), pages 122-134.
    10. Tohme, Tony & Vanslette, Kevin & Youcef-Toumi, Kamal, 2020. "A generalized Bayesian approach to model calibration," Reliability Engineering and System Safety, Elsevier, vol. 204(C).
    11. Strigini, Lorenzo & Wright, David, 2014. "Bounds on survival probability given mean probability of failure per demand; and the paradoxical advantages of uncertainty," Reliability Engineering and System Safety, Elsevier, vol. 128(C), pages 66-83.
    12. Teferra, Kirubel & Shields, Michael D. & Hapij, Adam & Daddazio, Raymond P., 2014. "Mapping model validation metrics to subject matter expert scores for model adequacy assessment," Reliability Engineering and System Safety, Elsevier, vol. 132(C), pages 9-19.
    13. Igor Fedotenkov, 2020. "A Review of More than One Hundred Pareto-Tail Index Estimators," Statistica, Department of Statistics, University of Bologna, vol. 80(3), pages 245-299.
    14. Zheng Liu & Xin Liu & Hong-Zhong Huang & Pingyu Zhu & Zhongwei Liang, 2022. "A new inherent reliability modeling and analysis method based on imprecise Dirichlet model for machine tool spindle," Annals of Operations Research, Springer, vol. 311(1), pages 295-310, April.
    15. Ivanilda Cabral & Frederico Caeiro & M. Ivette Gomes, 2022. "On the comparison of several classical estimators of the extreme value index," Communications in Statistics - Theory and Methods, Taylor & Francis Journals, vol. 51(1), pages 179-196, January.
    16. Cao, Jiaokun & Du, Farong & Ding, Shuiting, 2013. "Global sensitivity analysis for dynamic systems with stochastic input processes," Reliability Engineering and System Safety, Elsevier, vol. 118(C), pages 106-117.
    17. Certa, Antonella & Hopps, Fabrizio & Inghilleri, Roberta & La Fata, Concetta Manuela, 2017. "A Dempster-Shafer Theory-based approach to the Failure Mode, Effects and Criticality Analysis (FMECA) under epistemic uncertainty: application to the propulsion system of a fishing vessel," Reliability Engineering and System Safety, Elsevier, vol. 159(C), pages 69-79.
    18. Goldman, Matt & Kaplan, David M., 2017. "Fractional order statistic approximation for nonparametric conditional quantile inference," Journal of Econometrics, Elsevier, vol. 196(2), pages 331-346.
    19. Mathieu Balesdent & Jérôme Morio & Loïc Brevault, 2016. "Rare Event Probability Estimation in the Presence of Epistemic Uncertainty on Input Probability Distribution Parameters," Methodology and Computing in Applied Probability, Springer, vol. 18(1), pages 197-216, March.
    20. Wang, Wei & Cammi, Antonio & Di Maio, Francesco & Lorenzi, Stefano & Zio, Enrico, 2018. "A Monte Carlo-based exploration framework for identifying components vulnerable to cyber threats in nuclear power plants," Reliability Engineering and System Safety, Elsevier, vol. 175(C), pages 24-37.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:reensy:v:180:y:2018:i:c:p:201-210. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/reliability-engineering-and-system-safety .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.