IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2211.16362.html
   My bibliography  Save this paper

Score-based calibration testing for multivariate forecast distributions

Author

Listed:
  • Malte Knuppel
  • Fabian Kruger
  • Marc-Oliver Pohle

Abstract

Calibration tests based on the probability integral transform (PIT) are routinely used to assess the quality of univariate distributional forecasts. However, PIT-based calibration tests for multivariate distributional forecasts face various challenges. We propose two new types of tests based on proper scoring rules, which overcome these challenges. They arise from a general framework for calibration testing in the multivariate case, introduced in this work. The new tests have good size and power properties in simulations and solve various problems of existing tests. We apply the tests to forecast distributions for macroeconomic and financial time series data.

Suggested Citation

  • Malte Knuppel & Fabian Kruger & Marc-Oliver Pohle, 2022. "Score-based calibration testing for multivariate forecast distributions," Papers 2211.16362, arXiv.org, revised Dec 2023.
    • Handle: RePEc:arx:papers:2211.16362
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2211.16362
    File Function: Latest version
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Gary M. Koop, 2013. "Forecasting with Medium and Large Bayesian VARS," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 28(2), pages 177-203, March.
    2. Jacob A. Mincer & Victor Zarnowitz, 1969. "The Evaluation of Economic Forecasts," NBER Chapters, in: Economic Forecasts and Expectations: Analysis of Forecasting Behavior and Performance, pages 3-46, National Bureau of Economic Research, Inc.
    3. Raffaella Giacomini & Halbert White, 2006. "Tests of Conditional Predictive Ability," Econometrica, Econometric Society, vol. 74(6), pages 1545-1578, November.
    4. Victor Chernozhukov & Iv·n Fern·ndez-Val & Alfred Galichon, 2010. "Quantile and Probability Curves Without Crossing," Econometrica, Econometric Society, vol. 78(3), pages 1093-1125, May.
    5. repec:hal:wpspec:info:hdl:2441/5rkqqmvrn4tl22s9mc4b6ga2g is not listed on IDEAS
    6. Gupta, Rangan & Huber, Florian & Piribauer, Philipp, 2020. "Predicting international equity returns: Evidence from time-varying parameter vector autoregressive models," International Review of Financial Analysis, Elsevier, vol. 68(C).
    7. Fabian Krüger & Sebastian Lerch & Thordis Thorarinsdottir & Tilmann Gneiting, 2021. "Predictive Inference Based on Markov Chain Monte Carlo Output," International Statistical Review, International Statistical Institute, vol. 89(2), pages 274-301, August.
    8. Tilmann Gneiting, 2008. "Editorial: Probabilistic forecasting," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 171(2), pages 319-321, April.
    9. Jonas Dovern & Hans Manner, 2020. "Order‐invariant tests for proper calibration of multivariate density forecasts," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 35(4), pages 440-456, June.
    10. West, Kenneth D, 1996. "Asymptotic Inference about Predictive Ability," Econometrica, Econometric Society, vol. 64(5), pages 1067-1084, September.
    11. Catania, Leopoldo & Grassi, Stefano & Ravazzolo, Francesco, 2019. "Forecasting cryptocurrencies under model and parameter instability," International Journal of Forecasting, Elsevier, vol. 35(2), pages 485-501.
    12. Todd E. Clark, 2011. "Real-Time Density Forecasts From Bayesian Vector Autoregressions With Stochastic Volatility," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 29(3), pages 327-341, July.
    13. Raffaella Giacomini & Barbara Rossi, 2010. "Forecast comparisons in unstable environments," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(4), pages 595-620.
    14. Tilmann Gneiting & Fadoua Balabdaoui & Adrian E. Raftery, 2007. "Probabilistic forecasts, calibration and sharpness," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 69(2), pages 243-268, April.
    15. Andrea Carriero & Todd E. Clark & Massimiliano Marcellino & Elmar Mertens, 2024. "Addressing COVID-19 Outliers in BVARs with Stochastic Volatility," The Review of Economics and Statistics, MIT Press, vol. 106(5), pages 1403-1417, September.
    16. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    17. L. Held & K. Rufibach & F. Balabdaoui, 2010. "A Score Regression Approach to Assess Calibration of Continuous Probabilistic Predictions," Biometrics, The International Biometric Society, vol. 66(4), pages 1295-1305, December.
    18. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    19. Yi‐Ting Chen, 2011. "Moment tests for density forecast evaluation in the presence of parameter estimation uncertainty," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 30(4), pages 409-450, July.
    20. Oliver Grothe & Fabian Kachele & Fabian Kruger, 2022. "From point forecasts to multivariate probabilistic forecasts: The Schaake shuffle for day-ahead electricity price forecasting," Papers 2204.10154, arXiv.org.
    21. Rossi, Barbara & Sekhposyan, Tatevik, 2019. "Alternative tests for correct specification of conditional predictive densities," Journal of Econometrics, Elsevier, vol. 208(2), pages 638-657.
    22. Michael P. Clements, 2014. "Forecast Uncertainty- Ex Ante and Ex Post : U.S. Inflation and Output Growth," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 32(2), pages 206-216, April.
    23. Filip Žikeš & Jozef Baruník, 2015. "Semi-parametric Conditional Quantile Models for Financial Returns and Realized Volatility," Journal of Financial Econometrics, Society for Financial Econometrics, vol. 14(1), pages 185-226.
    24. Clements, Michael P. & Smith, Jeremy, 2002. "Evaluating multivariate forecast densities: a comparison of two approaches," International Journal of Forecasting, Elsevier, vol. 18(3), pages 397-407.
    25. Ole E. Barndorff-Nielsen & Peter Reinhard Hansen & Asger Lunde & Neil Shephard, 2008. "Designing Realized Kernels to Measure the ex post Variation of Equity Prices in the Presence of Noise," Econometrica, Econometric Society, vol. 76(6), pages 1481-1536, November.
    26. Kenichiro McAlinn & Knut Are Aastveit & Jouchi Nakajima & Mike West, 2020. "Multivariate Bayesian Predictive Synthesis in Macroeconomic Forecasting," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 115(531), pages 1092-1110, July.
    27. repec:wrk:wrkemf:24 is not listed on IDEAS
    28. Diebold, Francis X & Gunther, Todd A & Tay, Anthony S, 1998. "Evaluating Density Forecasts with Applications to Financial Risk Management," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 39(4), pages 863-883, November.
    29. O. E. Barndorff-Nielsen & P. Reinhard Hansen & A. Lunde & N. Shephard, 2009. "Realized kernels in practice: trades and quotes," Econometrics Journal, Royal Economic Society, vol. 12(3), pages 1-32, November.
    30. Diks, Cees & Panchenko, Valentyn & Sokolinskiy, Oleg & van Dijk, Dick, 2014. "Comparing the accuracy of multivariate density forecasts in selected regions of the copula support," Journal of Economic Dynamics and Control, Elsevier, vol. 48(C), pages 79-94.
    31. González-Rivera, Gloria & Yoldas, Emre, 2012. "Autocontour-based evaluation of multivariate predictive densities," International Journal of Forecasting, Elsevier, vol. 28(2), pages 328-342.
    32. Giorgio E. Primiceri, 2005. "Time Varying Structural Vector Autoregressions and Monetary Policy," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 72(3), pages 821-852.
    33. Jacob A. Mincer, 1969. "Economic Forecasts and Expectations: Analysis of Forecasting Behavior and Performance," NBER Books, National Bureau of Economic Research, Inc, number minc69-1, June.
    34. Marco Del Negro & Giorgio E. Primiceri, 2015. "Time Varying Structural Vector Autoregressions and Monetary Policy: A Corrigendum," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 82(4), pages 1342-1345.
    35. Tsyplakov, Alexander, 2014. "Theoretical guidelines for a partially informed forecast examiner," MPRA Paper 55017, University Library of Munich, Germany.
    36. repec:hal:spmain:info:hdl:2441/5rkqqmvrn4tl22s9mc4b6ga2g is not listed on IDEAS
    37. Fulvio Corsi, 2009. "A Simple Approximate Long-Memory Model of Realized Volatility," Journal of Financial Econometrics, Oxford University Press, vol. 7(2), pages 174-196, Spring.
    38. Claudio Heinrich & Kristoffer H. Hellton & Alex Lenkoski & Thordis L. Thorarinsdottir, 2021. "Multivariate Postprocessing Methods for High-Dimensional Seasonal Weather Forecasts," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(535), pages 1048-1059, July.
    39. Bollerslev, Tim, 1990. "Modelling the Coherence in Short-run Nominal Exchange Rates: A Multivariate Generalized ARCH Model," The Review of Economics and Statistics, MIT Press, vol. 72(3), pages 498-505, August.
    40. Joshua C. C. Chan, 2020. "Large Bayesian VARs: A Flexible Kronecker Error Covariance Structure," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 38(1), pages 68-79, January.
    41. Tilmann Gneiting & Larissa Stanberry & Eric Grimit & Leonhard Held & Nicholas Johnson, 2008. "Rejoinder on: Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 17(2), pages 256-264, August.
    42. Jonah Gabry & Daniel Simpson & Aki Vehtari & Michael Betancourt & Andrew Gelman, 2019. "Visualization in Bayesian workflow," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 182(2), pages 389-402, February.
    43. Ana Beatriz Galvão & James Mitchell, 2019. "Measuring Data Uncertainty: An Application using the Bank of England's "Fan Charts" for Historical GDP Growth," Economic Statistics Centre of Excellence (ESCoE) Discussion Papers ESCoE DP-2019-08, Economic Statistics Centre of Excellence (ESCoE).
    44. Marc-Oliver Pohle, 2020. "The Murphy Decomposition and the Calibration-Resolution Principle: A New Perspective on Forecast Evaluation," Papers 2005.01835, arXiv.org.
    45. Ko, Stanley I.M. & Park, Sung Y., 2013. "Multivariate density forecast evaluation: A modified approach," International Journal of Forecasting, Elsevier, vol. 29(3), pages 431-441.
    46. Francis X. Diebold & Jinyong Hahn & Anthony S. Tay, 1999. "Multivariate Density Forecast Evaluation And Calibration In Financial Risk Management: High-Frequency Returns On Foreign Exchange," The Review of Economics and Statistics, MIT Press, vol. 81(4), pages 661-673, November.
    47. Tsyplakov, Alexander, 2011. "Evaluating density forecasts: a comment," MPRA Paper 31184, University Library of Munich, Germany.
    48. Wei, Wei & Balabdaoui, Fadoua & Held, Leonhard, 2017. "Calibration tests for multivariate Gaussian forecasts," Journal of Multivariate Analysis, Elsevier, vol. 154(C), pages 216-233.
    49. Tilmann Gneiting & Larissa Stanberry & Eric Grimit & Leonhard Held & Nicholas Johnson, 2008. "Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 17(2), pages 211-235, August.
    50. Szekely, Gábor J. & Rizzo, Maria L., 2005. "A new test for multivariate normality," Journal of Multivariate Analysis, Elsevier, vol. 93(1), pages 58-80, March.
    51. Victor Chernozhukov & Ivan Fernandez-Val & Alfred Galichon, 2007. "Quantile and probability curves without crossing," CeMMAP working papers CWP10/07, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    52. Florian Ziel & Kevin Berk, 2019. "Multivariate Forecasting Evaluation: On Sensitive and Strictly Proper Scoring Rules," Papers 1910.07325, arXiv.org.
    53. Genest, Christian & Rivest, Louis-Paul, 2001. "On the multivariate probability integral transformation," Statistics & Probability Letters, Elsevier, vol. 53(4), pages 391-399, July.
    54. Kemal Guler & Pin T. Ng & Zhijie Xiao, 2017. "Mincer–Zarnowitz quantile and expectile regressions for forecast evaluations under aysmmetric loss functions," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 36(6), pages 651-679, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    2. Barbara Rossi, 2019. "Forecasting in the presence of instabilities: How do we know whether models predict well and how to improve them," Economics Working Papers 1711, Department of Economics and Business, Universitat Pompeu Fabra, revised Jul 2021.
    3. Jonas Dovern & Hans Manner, 2020. "Order‐invariant tests for proper calibration of multivariate density forecasts," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 35(4), pages 440-456, June.
    4. Dovern, Jonas & Manner, Hans, 2016. "Robust Evaluation of Multivariate Density Forecasts," VfS Annual Conference 2016 (Augsburg): Demographic Change 145547, Verein für Socialpolitik / German Economic Association.
    5. Tsyplakov, Alexander, 2014. "Theoretical guidelines for a partially informed forecast examiner," MPRA Paper 55017, University Library of Munich, Germany.
    6. Dovern, Jonas & Manner, Hans, 2016. "Order Invariant Evaluation of Multivariate Density Forecasts," Working Papers 0608, University of Heidelberg, Department of Economics.
    7. Carlos Henrique Dias Cordeiro de Castro & Fernando Antonio Lucena Aiube, 2023. "Forecasting inflation time series using score‐driven dynamic models and combination methods: The case of Brazil," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 42(2), pages 369-401, March.
    8. Gaglianone, Wagner Piazza & Marins, Jaqueline Terra Moura, 2017. "Evaluation of exchange rate point and density forecasts: An application to Brazil," International Journal of Forecasting, Elsevier, vol. 33(3), pages 707-728.
    9. Todd E. Clark & Florian Huber & Gary Koop & Massimiliano Marcellino & Michael Pfarrhofer, 2023. "Tail Forecasting With Multivariate Bayesian Additive Regression Trees," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 64(3), pages 979-1022, August.
    10. Allen, Sam & Koh, Jonathan & Segers, Johan & Ziegel, Johanna, 2024. "Tail calibration of probabilistic forecasts," LIDAM Discussion Papers ISBA 2024018, Université catholique de Louvain, Institute of Statistics, Biostatistics and Actuarial Sciences (ISBA).
    11. Tallman, Ellis W. & Zaman, Saeed, 2020. "Combining survey long-run forecasts and nowcasts with BVAR forecasts using relative entropy," International Journal of Forecasting, Elsevier, vol. 36(2), pages 373-398.
    12. Fabian Krüger & Todd E. Clark & Francesco Ravazzolo, 2017. "Using Entropic Tilting to Combine BVAR Forecasts With External Nowcasts," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 35(3), pages 470-485, July.
    13. Marc-Oliver Pohle, 2020. "The Murphy Decomposition and the Calibration-Resolution Principle: A New Perspective on Forecast Evaluation," Papers 2005.01835, arXiv.org.
    14. Alexander, Carol & Han, Yang & Meng, Xiaochun, 2023. "Static and dynamic models for multivariate distribution forecasts: Proper scoring rule tests of factor-quantile versus multivariate GARCH models," International Journal of Forecasting, Elsevier, vol. 39(3), pages 1078-1096.
    15. Taylor, James W., 2020. "A strategic predictive distribution for tests of probabilistic calibration," International Journal of Forecasting, Elsevier, vol. 36(4), pages 1380-1388.
    16. Warne, Anders, 2023. "DSGE model forecasting: rational expectations vs. adaptive learning," Working Paper Series 2768, European Central Bank.
    17. Rossi, Barbara, 2013. "Advances in Forecasting under Instability," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1203-1324, Elsevier.
    18. Berg, Tim O. & Henzel, Steffen R., 2015. "Point and density forecasts for the euro area using Bayesian VARs," International Journal of Forecasting, Elsevier, vol. 31(4), pages 1067-1095.
    19. Wei Wei & Leonhard Held, 2014. "Calibration tests for count data," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 23(4), pages 787-805, December.
    20. Gneiting, Tilmann, 2011. "Making and Evaluating Point Forecasts," Journal of the American Statistical Association, American Statistical Association, vol. 106(494), pages 746-762.

    More about this item

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2211.16362. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.