IDEAS home Printed from https://ideas.repec.org/a/eee/reensy/v159y2017icp80-89.html
   My bibliography  Save this article

Validation metric based on Mahalanobis distance for models with multiple correlated responses

Author

Listed:
  • Zhao, Lufeng
  • Lu, Zhenzhou
  • Yun, Wanying
  • Wang, Wenjin

Abstract

In the probabilistic context, validation metric for models with multiple responses is essentially used to measure the difference between joint statistical distributions resulting from simulation predictions and experimental observations respectively. Considering both uncertainty and correlation, existing validation metrics either ignore correlations among responses or have a relatively huge computational cost. In this paper, by extending the concept of “area metric†and “u-pooling method†developed for validating a scalar response, two new metrics are proposed to validate models with multiple correlated responses using Mahalanobis distance (MD). One new metric is the MD area metric for validating multi-responses at a single validation site. The other is the MD-pooling metric, and it allows for pooling the evidence from all relevant data of multi-response over the intended validation domain into a scalar measure to assess the global predictive capability of computational models. The proposed metrics are applicable to validation for models with multiple correlated responses. Their several favorable properties include objectiveness, affordability, unboundedness, and determinacy of the results. Compared with the existing validation metrics, the feasibility, effectiveness and efficiency of our proposed two metrics are illustrated by a numerical test example and an engineering example.

Suggested Citation

  • Zhao, Lufeng & Lu, Zhenzhou & Yun, Wanying & Wang, Wenjin, 2017. "Validation metric based on Mahalanobis distance for models with multiple correlated responses," Reliability Engineering and System Safety, Elsevier, vol. 159(C), pages 80-89.
  • Handle: RePEc:eee:reensy:v:159:y:2017:i:c:p:80-89
    DOI: 10.1016/j.ress.2016.10.016
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0951832016307116
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.ress.2016.10.016?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Rebba, Ramesh & Mahadevan, Sankaran, 2006. "Validation of models with multivariate output," Reliability Engineering and System Safety, Elsevier, vol. 91(8), pages 861-871.
    2. Scott Ferson & William L. Oberkampf, 2009. "Validation of imprecise probability models," International Journal of Reliability and Safety, Inderscience Enterprises Ltd, vol. 3(1/2/3), pages 3-22.
    3. Li, Wei & Chen, Wei & Jiang, Zhen & Lu, Zhenzhou & Liu, Yu, 2014. "New validation metrics for models with multiple correlated responses," Reliability Engineering and System Safety, Elsevier, vol. 127(C), pages 1-11.
    4. Marc C. Kennedy & Anthony O'Hagan, 2001. "Bayesian calibration of computer models," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 63(3), pages 425-464.
    5. Genest, Christian & Rivest, Louis-Paul, 2001. "On the multivariate probability integral transformation," Statistics & Probability Letters, Elsevier, vol. 53(4), pages 391-399, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Vanslette, Kevin & Tohme, Tony & Youcef-Toumi, Kamal, 2020. "A general model validation and testing tool," Reliability Engineering and System Safety, Elsevier, vol. 195(C).
    2. Kim, Wongon & Yoon, Heonjun & Lee, Guesuk & Kim, Taejin & Youn, Byeng D., 2020. "A new calibration metric that considers statistical correlation: Marginal Probability and Correlation Residuals," Reliability Engineering and System Safety, Elsevier, vol. 195(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Li, Wei & Chen, Wei & Jiang, Zhen & Lu, Zhenzhou & Liu, Yu, 2014. "New validation metrics for models with multiple correlated responses," Reliability Engineering and System Safety, Elsevier, vol. 127(C), pages 1-11.
    2. Li, Luyi & Lu, Zhenzhou, 2018. "A new method for model validation with multivariate output," Reliability Engineering and System Safety, Elsevier, vol. 169(C), pages 579-592.
    3. Ao, Dan & Hu, Zhen & Mahadevan, Sankaran, 2017. "Design of validation experiments for life prediction models," Reliability Engineering and System Safety, Elsevier, vol. 165(C), pages 22-33.
    4. Mullins, Joshua & Ling, You & Mahadevan, Sankaran & Sun, Lin & Strachan, Alejandro, 2016. "Separation of aleatory and epistemic uncertainty in probabilistic model validation," Reliability Engineering and System Safety, Elsevier, vol. 147(C), pages 49-59.
    5. Wang, Chong & Matthies, Hermann G., 2019. "Novel model calibration method via non-probabilistic interval characterization and Bayesian theory," Reliability Engineering and System Safety, Elsevier, vol. 183(C), pages 84-92.
    6. Kwag, Shinyoung & Gupta, Abhinav & Dinh, Nam, 2018. "Probabilistic risk assessment based model validation method using Bayesian network," Reliability Engineering and System Safety, Elsevier, vol. 169(C), pages 380-393.
    7. Wu, Danqing & Lu, Zhenzhou & Wang, Yanping & Cheng, Lei, 2015. "Model validation and calibration based on component functions of model output," Reliability Engineering and System Safety, Elsevier, vol. 140(C), pages 59-70.
    8. Liu, Yang & Wang, Dewei & Sun, Xiaodong & Liu, Yang & Dinh, Nam & Hu, Rui, 2021. "Uncertainty quantification for Multiphase-CFD simulations of bubbly flows: a machine learning-based Bayesian approach supported by high-resolution experiments," Reliability Engineering and System Safety, Elsevier, vol. 212(C).
    9. Ling, You & Mahadevan, Sankaran, 2013. "Quantitative model validation techniques: New insights," Reliability Engineering and System Safety, Elsevier, vol. 111(C), pages 217-231.
    10. Vanslette, Kevin & Tohme, Tony & Youcef-Toumi, Kamal, 2020. "A general model validation and testing tool," Reliability Engineering and System Safety, Elsevier, vol. 195(C).
    11. Bin Suo & Yang Qi & Kai Sun & Jingyuan Xu, 2023. "A Novel Model Validation Method Based on Area Metric Disagreement between Accelerated Storage Distributions and Natural Storage Data," Mathematics, MDPI, vol. 11(11), pages 1-18, May.
    12. Li, Chenzhao & Mahadevan, Sankaran, 2016. "Role of calibration, validation, and relevance in multi-level uncertainty integration," Reliability Engineering and System Safety, Elsevier, vol. 148(C), pages 32-43.
    13. Kim, Wongon & Yoon, Heonjun & Lee, Guesuk & Kim, Taejin & Youn, Byeng D., 2020. "A new calibration metric that considers statistical correlation: Marginal Probability and Correlation Residuals," Reliability Engineering and System Safety, Elsevier, vol. 195(C).
    14. Park, Inseok & Grandhi, Ramana V., 2014. "A Bayesian statistical method for quantifying model form uncertainty and two model combination methods," Reliability Engineering and System Safety, Elsevier, vol. 129(C), pages 46-56.
    15. Matthias Katzfuss & Joseph Guinness & Wenlong Gong & Daniel Zilber, 2020. "Vecchia Approximations of Gaussian-Process Predictions," Journal of Agricultural, Biological and Environmental Statistics, Springer;The International Biometric Society;American Statistical Association, vol. 25(3), pages 383-414, September.
    16. Hao Wu & Michael Browne, 2015. "Random Model Discrepancy: Interpretations and Technicalities (A Rejoinder)," Psychometrika, Springer;The Psychometric Society, vol. 80(3), pages 619-624, September.
    17. Xiaoyu Xiong & Benjamin D. Youngman & Theodoros Economou, 2021. "Data fusion with Gaussian processes for estimation of environmental hazard events," Environmetrics, John Wiley & Sons, Ltd., vol. 32(3), May.
    18. Petropoulos, G. & Wooster, M.J. & Carlson, T.N. & Kennedy, M.C. & Scholze, M., 2009. "A global Bayesian sensitivity analysis of the 1d SimSphere soil–vegetation–atmospheric transfer (SVAT) model using Gaussian model emulation," Ecological Modelling, Elsevier, vol. 220(19), pages 2427-2440.
    19. Drignei, Dorin, 2011. "A general statistical model for computer experiments with time series output," Reliability Engineering and System Safety, Elsevier, vol. 96(4), pages 460-467.
    20. Yuan, Jun & Ng, Szu Hui, 2013. "A sequential approach for stochastic computer model calibration and prediction," Reliability Engineering and System Safety, Elsevier, vol. 111(C), pages 273-286.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:reensy:v:159:y:2017:i:c:p:80-89. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/reliability-engineering-and-system-safety .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.