IDEAS home Printed from https://ideas.repec.org/a/eee/reensy/v148y2016icp32-43.html
   My bibliography  Save this article

Role of calibration, validation, and relevance in multi-level uncertainty integration

Author

Listed:
  • Li, Chenzhao
  • Mahadevan, Sankaran

Abstract

Calibration of model parameters is an essential step in predicting the response of a complicated system, but the lack of data at the system level makes it impossible to conduct this quantification directly. In such a situation, system model parameters are estimated using tests at lower levels of complexity which share the same model parameters with the system. For such a multi-level problem, this paper proposes a methodology to quantify the uncertainty in the system level prediction by integrating calibration, validation and sensitivity analysis at different levels. The proposed approach considers the validity of the models used for parameter estimation at lower levels, as well as the relevance at the lower level to the prediction at the system level. The model validity is evaluated using a model reliability metric, and models with multivariate output are considered. The relevance is quantified by comparing Sobol indices at the lower level and system level, thus measuring the extent to which a lower level test represents the characteristics of the system so that the calibration results can be reliably used in the system level. Finally the results of calibration, validation and relevance analysis are integrated in a roll-up method to predict the system output.

Suggested Citation

  • Li, Chenzhao & Mahadevan, Sankaran, 2016. "Role of calibration, validation, and relevance in multi-level uncertainty integration," Reliability Engineering and System Safety, Elsevier, vol. 148(C), pages 32-43.
    • Handle: RePEc:eee:reensy:v:148:y:2016:i:c:p:32-43
    DOI: 10.1016/j.ress.2015.11.013
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0951832015003403
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ress.2015.11.013?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Marc C. Kennedy & Anthony O'Hagan, 2001. "Bayesian calibration of computer models," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 63(3), pages 425-464.
    2. Rebba, Ramesh & Mahadevan, Sankaran, 2008. "Computational methods for model reliability assessment," Reliability Engineering and System Safety, Elsevier, vol. 93(8), pages 1197-1207.
    3. Scott Ferson & William L. Oberkampf, 2009. "Validation of imprecise probability models," International Journal of Reliability and Safety, Inderscience Enterprises Ltd, vol. 3(1/2/3), pages 3-22.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Vanslette, Kevin & Tohme, Tony & Youcef-Toumi, Kamal, 2020. "A general model validation and testing tool," Reliability Engineering and System Safety, Elsevier, vol. 195(C).
    2. Jung, Yongsu & Lee, Ikjin, 2021. "Optimal design of experiments for optimization-based model calibration using Fisher information matrix," Reliability Engineering and System Safety, Elsevier, vol. 216(C).
    3. Schöbi, Roland & Sudret, Bruno, 2019. "Global sensitivity analysis in the context of imprecise probabilities (p-boxes) using sparse polynomial chaos expansions," Reliability Engineering and System Safety, Elsevier, vol. 187(C), pages 129-141.
    4. Yoo, Yeongmin & Jung, Ui-Jin & Han, Yong Ha & Lee, Jongsoo, 2021. "Data Augmentation-Based Prediction of System Level Performance under Model and Parameter Uncertainties: Role of Designable Generative Adversarial Networks (DGAN)," Reliability Engineering and System Safety, Elsevier, vol. 206(C).
    5. Fei, Liguo & Deng, Yong, 2017. "A new method to identify influential nodes based on relative entropy," Chaos, Solitons & Fractals, Elsevier, vol. 104(C), pages 257-267.
    6. Kim, Wongon & Yoon, Heonjun & Lee, Guesuk & Kim, Taejin & Youn, Byeng D., 2020. "A new calibration metric that considers statistical correlation: Marginal Probability and Correlation Residuals," Reliability Engineering and System Safety, Elsevier, vol. 195(C).
    7. Li, Chenzhao & Mahadevan, Sankaran, 2016. "An efficient modularized sample-based method to estimate the first-order Sobol׳ index," Reliability Engineering and System Safety, Elsevier, vol. 153(C), pages 110-121.
    8. Jung, Yongsu & Jo, Hwisang & Choo, Jeonghwan & Lee, Ikjin, 2022. "Statistical model calibration and design optimization under aleatory and epistemic uncertainty," Reliability Engineering and System Safety, Elsevier, vol. 222(C).
    9. Zhang, Xiaoge & Mahadevan, Sankaran & Deng, Xinyang, 2017. "Reliability analysis with linguistic data: An evidential network approach," Reliability Engineering and System Safety, Elsevier, vol. 162(C), pages 111-121.
    10. Jiang, Chen & Vega, Manuel A. & Todd, Michael D. & Hu, Zhen, 2022. "Model correction and updating of a stochastic degradation model for failure prognostics of miter gates," Reliability Engineering and System Safety, Elsevier, vol. 218(PA).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kwag, Shinyoung & Gupta, Abhinav & Dinh, Nam, 2018. "Probabilistic risk assessment based model validation method using Bayesian network," Reliability Engineering and System Safety, Elsevier, vol. 169(C), pages 380-393.
    2. Mullins, Joshua & Ling, You & Mahadevan, Sankaran & Sun, Lin & Strachan, Alejandro, 2016. "Separation of aleatory and epistemic uncertainty in probabilistic model validation," Reliability Engineering and System Safety, Elsevier, vol. 147(C), pages 49-59.
    3. Vanslette, Kevin & Tohme, Tony & Youcef-Toumi, Kamal, 2020. "A general model validation and testing tool," Reliability Engineering and System Safety, Elsevier, vol. 195(C).
    4. Ao, Dan & Hu, Zhen & Mahadevan, Sankaran, 2017. "Design of validation experiments for life prediction models," Reliability Engineering and System Safety, Elsevier, vol. 165(C), pages 22-33.
    5. Li, Wei & Chen, Wei & Jiang, Zhen & Lu, Zhenzhou & Liu, Yu, 2014. "New validation metrics for models with multiple correlated responses," Reliability Engineering and System Safety, Elsevier, vol. 127(C), pages 1-11.
    6. Wang, Chong & Matthies, Hermann G., 2019. "Novel model calibration method via non-probabilistic interval characterization and Bayesian theory," Reliability Engineering and System Safety, Elsevier, vol. 183(C), pages 84-92.
    7. Wu, Danqing & Lu, Zhenzhou & Wang, Yanping & Cheng, Lei, 2015. "Model validation and calibration based on component functions of model output," Reliability Engineering and System Safety, Elsevier, vol. 140(C), pages 59-70.
    8. Liu, Yang & Wang, Dewei & Sun, Xiaodong & Liu, Yang & Dinh, Nam & Hu, Rui, 2021. "Uncertainty quantification for Multiphase-CFD simulations of bubbly flows: a machine learning-based Bayesian approach supported by high-resolution experiments," Reliability Engineering and System Safety, Elsevier, vol. 212(C).
    9. Ling, You & Mahadevan, Sankaran, 2013. "Quantitative model validation techniques: New insights," Reliability Engineering and System Safety, Elsevier, vol. 111(C), pages 217-231.
    10. Sankararaman, Shankar & Mahadevan, Sankaran, 2015. "Integration of model verification, validation, and calibration for uncertainty quantification in engineering systems," Reliability Engineering and System Safety, Elsevier, vol. 138(C), pages 194-209.
    11. Zhao, Lufeng & Lu, Zhenzhou & Yun, Wanying & Wang, Wenjin, 2017. "Validation metric based on Mahalanobis distance for models with multiple correlated responses," Reliability Engineering and System Safety, Elsevier, vol. 159(C), pages 80-89.
    12. Matthias Katzfuss & Joseph Guinness & Wenlong Gong & Daniel Zilber, 2020. "Vecchia Approximations of Gaussian-Process Predictions," Journal of Agricultural, Biological and Environmental Statistics, Springer;The International Biometric Society;American Statistical Association, vol. 25(3), pages 383-414, September.
    13. Jakub Bijak & Jason D. Hilton & Eric Silverman & Viet Dung Cao, 2013. "Reforging the Wedding Ring," Demographic Research, Max Planck Institute for Demographic Research, Rostock, Germany, vol. 29(27), pages 729-766.
    14. Hao Wu & Michael Browne, 2015. "Random Model Discrepancy: Interpretations and Technicalities (A Rejoinder)," Psychometrika, Springer;The Psychometric Society, vol. 80(3), pages 619-624, September.
    15. Villez, Kris & Del Giudice, Dario & Neumann, Marc B. & Rieckermann, Jörg, 2020. "Accounting for erroneous model structures in biokinetic process models," Reliability Engineering and System Safety, Elsevier, vol. 203(C).
    16. Xiaoyu Xiong & Benjamin D. Youngman & Theodoros Economou, 2021. "Data fusion with Gaussian processes for estimation of environmental hazard events," Environmetrics, John Wiley & Sons, Ltd., vol. 32(3), May.
    17. Petropoulos, G. & Wooster, M.J. & Carlson, T.N. & Kennedy, M.C. & Scholze, M., 2009. "A global Bayesian sensitivity analysis of the 1d SimSphere soil–vegetation–atmospheric transfer (SVAT) model using Gaussian model emulation," Ecological Modelling, Elsevier, vol. 220(19), pages 2427-2440.
    18. David Breitenmoser & Francesco Cerutti & Gernot Butterweck & Malgorzata Magdalena Kasprzak & Sabine Mayer, 2023. "Emulator-based Bayesian inference on non-proportional scintillation models by compton-edge probing," Nature Communications, Nature, vol. 14(1), pages 1-12, December.
    19. Mathieu Balesdent & Jérôme Morio & Loïc Brevault, 2016. "Rare Event Probability Estimation in the Presence of Epistemic Uncertainty on Input Probability Distribution Parameters," Methodology and Computing in Applied Probability, Springer, vol. 18(1), pages 197-216, March.
    20. Drignei, Dorin, 2011. "A general statistical model for computer experiments with time series output," Reliability Engineering and System Safety, Elsevier, vol. 96(4), pages 460-467.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:reensy:v:148:y:2016:i:c:p:32-43. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/reliability-engineering-and-system-safety .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.