IDEAS home Printed from https://ideas.repec.org/a/eee/reensy/v169y2018icp579-592.html
   My bibliography  Save this article

A new method for model validation with multivariate output

Author

Listed:
  • Li, Luyi
  • Lu, Zhenzhou

Abstract

Traditional methods for model validation assessment mainly focus on validating a single response. However, for many applications joint predictions of the multiple responses are needed. It is thereby not sufficient to validate the individual responses separately, which ignores correlation among multiple responses. Validation assessment for multiple responses involves comparison with multiple experimental measurements, which makes it much more complicated than that for single response. With considering both the uncertainty and correlation of multiple responses, this paper presents a new method for validation assessment of models with multivariate output. The new method is based on principal component analysis and the concept of area metric. The method is innovative in that it can eliminate the redundant part of multiple responses while reserving their main variability information in the assessment process. This avoids directly comparing the joint distributions of computational and experimental responses. It not only can be used for validating multiple responses at a single validation site, but also is capable of dealing with the case where observations of multiple responses are collected at multiple validation sites. The new method is examined and compared with the existing u-pooling and t-pooling methods through numerical and engineering examples to illustrate its validity and potential benefits.

Suggested Citation

  • Li, Luyi & Lu, Zhenzhou, 2018. "A new method for model validation with multivariate output," Reliability Engineering and System Safety, Elsevier, vol. 169(C), pages 579-592.
    • Handle: RePEc:eee:reensy:v:169:y:2018:i:c:p:579-592
    DOI: 10.1016/j.ress.2017.10.005
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0951832016302149
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ress.2017.10.005?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Rebba, Ramesh & Mahadevan, Sankaran, 2006. "Validation of models with multivariate output," Reliability Engineering and System Safety, Elsevier, vol. 91(8), pages 861-871.
    2. Li, Baibing & Martin, Elaine B. & Morris, A. Julian, 2002. "On principal component analysis in L1," Computational Statistics & Data Analysis, Elsevier, vol. 40(3), pages 471-474, September.
    3. Besse, Philippe, 1992. "PCA stability and choice of dimensionality," Statistics & Probability Letters, Elsevier, vol. 13(5), pages 405-410, April.
    4. Xiaomo Jiang & Sankaran Mahadevan, 2008. "Bayesian validation assessment of multivariate computational models," Journal of Applied Statistics, Taylor & Francis Journals, vol. 35(1), pages 49-65.
    5. Li, Wei & Chen, Wei & Jiang, Zhen & Lu, Zhenzhou & Liu, Yu, 2014. "New validation metrics for models with multiple correlated responses," Reliability Engineering and System Safety, Elsevier, vol. 127(C), pages 1-11.
    6. Marc C. Kennedy & Anthony O'Hagan, 2001. "Bayesian calibration of computer models," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 63(3), pages 425-464.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Xiang Peng & Xiaoqing Xu & Jiquan Li & Shaofei Jiang, 2021. "A Sampling-Based Sensitivity Analysis Method Considering the Uncertainties of Input Variables and Their Distribution Parameters," Mathematics, MDPI, vol. 9(10), pages 1-18, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zhao, Lufeng & Lu, Zhenzhou & Yun, Wanying & Wang, Wenjin, 2017. "Validation metric based on Mahalanobis distance for models with multiple correlated responses," Reliability Engineering and System Safety, Elsevier, vol. 159(C), pages 80-89.
    2. Li, Wei & Chen, Wei & Jiang, Zhen & Lu, Zhenzhou & Liu, Yu, 2014. "New validation metrics for models with multiple correlated responses," Reliability Engineering and System Safety, Elsevier, vol. 127(C), pages 1-11.
    3. Lamboni, Matieyendou & Monod, Hervé & Makowski, David, 2011. "Multivariate sensitivity analysis to measure global contribution of input factors in dynamic models," Reliability Engineering and System Safety, Elsevier, vol. 96(4), pages 450-459.
    4. Vanslette, Kevin & Tohme, Tony & Youcef-Toumi, Kamal, 2020. "A general model validation and testing tool," Reliability Engineering and System Safety, Elsevier, vol. 195(C).
    5. Ao, Dan & Hu, Zhen & Mahadevan, Sankaran, 2017. "Design of validation experiments for life prediction models," Reliability Engineering and System Safety, Elsevier, vol. 165(C), pages 22-33.
    6. Dray, Stephane, 2008. "On the number of principal components: A test of dimensionality based on measurements of similarity between matrices," Computational Statistics & Data Analysis, Elsevier, vol. 52(4), pages 2228-2237, January.
    7. Nagel, Joseph B. & Rieckermann, Jörg & Sudret, Bruno, 2020. "Principal component analysis and sparse polynomial chaos expansions for global sensitivity analysis and model calibration: Application to urban drainage simulation," Reliability Engineering and System Safety, Elsevier, vol. 195(C).
    8. Kim, Wongon & Yoon, Heonjun & Lee, Guesuk & Kim, Taejin & Youn, Byeng D., 2020. "A new calibration metric that considers statistical correlation: Marginal Probability and Correlation Residuals," Reliability Engineering and System Safety, Elsevier, vol. 195(C).
    9. Park, Inseok & Grandhi, Ramana V., 2014. "A Bayesian statistical method for quantifying model form uncertainty and two model combination methods," Reliability Engineering and System Safety, Elsevier, vol. 129(C), pages 46-56.
    10. Teferra, Kirubel & Shields, Michael D. & Hapij, Adam & Daddazio, Raymond P., 2014. "Mapping model validation metrics to subject matter expert scores for model adequacy assessment," Reliability Engineering and System Safety, Elsevier, vol. 132(C), pages 9-19.
    11. Mullins, Joshua & Ling, You & Mahadevan, Sankaran & Sun, Lin & Strachan, Alejandro, 2016. "Separation of aleatory and epistemic uncertainty in probabilistic model validation," Reliability Engineering and System Safety, Elsevier, vol. 147(C), pages 49-59.
    12. Plat, Richard, 2009. "Stochastic portfolio specific mortality and the quantification of mortality basis risk," Insurance: Mathematics and Economics, Elsevier, vol. 45(1), pages 123-132, August.
    13. Kondylis, Athanassios & Whittaker, Joe, 2008. "Spectral preconditioning of Krylov spaces: Combining PLS and PC regression," Computational Statistics & Data Analysis, Elsevier, vol. 52(5), pages 2588-2603, January.
    14. Ouyang, Yaofu & Li, Peng, 2018. "On the nexus of financial development, economic growth, and energy consumption in China: New perspective from a GMM panel VAR approach," Energy Economics, Elsevier, vol. 71(C), pages 238-252.
    15. Matthias Katzfuss & Joseph Guinness & Wenlong Gong & Daniel Zilber, 2020. "Vecchia Approximations of Gaussian-Process Predictions," Journal of Agricultural, Biological and Environmental Statistics, Springer;The International Biometric Society;American Statistical Association, vol. 25(3), pages 383-414, September.
    16. Paschalis Arvanitidis & Athina Economou & Christos Kollias, 2016. "Terrorism’s effects on social capital in European countries," Public Choice, Springer, vol. 169(3), pages 231-250, December.
    17. Rizvi, Syed Kumail Abbas & Rahat, Birjees & Naqvi, Bushra & Umar, Muhammad, 2024. "Revolutionizing finance: The synergy of fintech, digital adoption, and innovation," Technological Forecasting and Social Change, Elsevier, vol. 200(C).
    18. Teerachai Amnuaylojaroen & Pavinee Chanvichit, 2024. "Historical Analysis of the Effects of Drought on Rice and Maize Yields in Southeast Asia," Resources, MDPI, vol. 13(3), pages 1-18, March.
    19. Hao Wu & Michael Browne, 2015. "Random Model Discrepancy: Interpretations and Technicalities (A Rejoinder)," Psychometrika, Springer;The Psychometric Society, vol. 80(3), pages 619-624, September.
    20. Weili Duan & Bin He & Daniel Nover & Guishan Yang & Wen Chen & Huifang Meng & Shan Zou & Chuanming Liu, 2016. "Water Quality Assessment and Pollution Source Identification of the Eastern Poyang Lake Basin Using Multivariate Statistical Methods," Sustainability, MDPI, vol. 8(2), pages 1-15, January.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:reensy:v:169:y:2018:i:c:p:579-592. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/reliability-engineering-and-system-safety .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.