IDEAS home Printed from https://ideas.repec.org/a/eee/intfor/v36y2020i4p1380-1388.html
   My bibliography  Save this article

A strategic predictive distribution for tests of probabilistic calibration

Author

Listed:
  • Taylor, James W.

Abstract

Forecasts of probability distributions are needed to support decision making in many applications. The accuracy of predictive distributions should be evaluated by maximising sharpness subject to calibration. Sharpness relates to the concentration of the predictive distributions, while calibration concerns their statistical consistency with the data. This paper focuses on calibration testing. It is important that a calibration test cannot be gamed by forecasts that have been strategically designed to pass the test. The widely used tests of probabilistic calibration for predictive distributions are based on the probability integral transform. Drawing on previous results for quantile prediction, we show that strategic distributional forecasting is a concern for these tests. To address this, we provide a simple extension of one of the tests. We illustrate ideas using simulated data.

Suggested Citation

  • Taylor, James W., 2020. "A strategic predictive distribution for tests of probabilistic calibration," International Journal of Forecasting, Elsevier, vol. 36(4), pages 1380-1388.
    • Handle: RePEc:eee:intfor:v:36:y:2020:i:4:p:1380-1388
    DOI: 10.1016/j.ijforecast.2020.02.001
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0169207020300182
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ijforecast.2020.02.001?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Natalia Nolde & Johanna F. Ziegel, 2016. "Elicitability and backtesting: Perspectives for banking regulation," Papers 1608.05498, arXiv.org, revised Feb 2017.
    2. Ottaviani, Marco & Sorensen, Peter Norman, 2006. "The strategy of professional forecasting," Journal of Financial Economics, Elsevier, vol. 81(2), pages 441-466, August.
    3. Jacob A. Mincer & Victor Zarnowitz, 1969. "The Evaluation of Economic Forecasts," NBER Chapters, in: Economic Forecasts and Expectations: Analysis of Forecasting Behavior and Performance, pages 3-46, National Bureau of Economic Research, Inc.
    4. Berkowitz, Jeremy, 2001. "Testing Density Forecasts, with Applications to Risk Management," Journal of Business & Economic Statistics, American Statistical Association, vol. 19(4), pages 465-474, October.
    5. Tommaso Proietti & Martyna Marczak & Gianluigi Mazzi, 2017. "Euromind‐ D : A Density Estimate of Monthly Gross Domestic Product for the Euro Area," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(3), pages 683-703, April.
    6. West, Kenneth D, 1996. "Asymptotic Inference about Predictive Ability," Econometrica, Econometric Society, vol. 64(5), pages 1067-1084, September.
    7. Denis Pelletier & Wei Wei, 2016. "The Geometric-VaR Backtesting Method," Journal of Financial Econometrics, Oxford University Press, vol. 14(4), pages 725-745.
    8. Tilmann Gneiting & Fadoua Balabdaoui & Adrian E. Raftery, 2007. "Probabilistic forecasts, calibration and sharpness," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 69(2), pages 243-268, April.
    9. Angelica Gianfreda & Derek Bunn, 2018. "A Stochastic Latent Moment Model for Electricity Price Formation," BEMPS - Bozen Economics & Management Paper Series BEMPS46, Faculty of Economics and Management at the Free University of Bozen.
    10. Michael P. Clements, 2004. "Evaluating the Bank of England Density Forecasts of Inflation," Economic Journal, Royal Economic Society, vol. 114(498), pages 844-866, October.
    11. Rossi, Barbara & Sekhposyan, Tatevik, 2014. "Evaluating predictive densities of US output growth and inflation in a large macroeconomic data set," International Journal of Forecasting, Elsevier, vol. 30(3), pages 662-682.
    12. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    13. Berrocal, Veronica J. & Raftery, Adrian E. & Gneiting, Tilmann & Steed, Richard C., 2010. "Probabilistic Weather Forecasting for Winter Road Maintenance," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 522-537.
    14. Rossi, Barbara & Sekhposyan, Tatevik, 2019. "Alternative tests for correct specification of conditional predictive densities," Journal of Econometrics, Elsevier, vol. 208(2), pages 638-657.
    15. Robert F. Engle & Simone Manganelli, 2004. "CAViaR: Conditional Autoregressive Value at Risk by Regression Quantiles," Journal of Business & Economic Statistics, American Statistical Association, vol. 22, pages 367-381, October.
    16. Diebold, Francis X & Gunther, Todd A & Tay, Anthony S, 1998. "Evaluating Density Forecasts with Applications to Financial Risk Management," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 39(4), pages 863-883, November.
    17. Kolassa, Stephan, 2016. "Evaluating predictive count data distributions in retail sales forecasting," International Journal of Forecasting, Elsevier, vol. 32(3), pages 788-803.
    18. Kenneth C. Lichtendahl & Yael Grushka-Cockayne & Phillip E. Pfeifer, 2013. "The Wisdom of Competitive Crowds," Operations Research, INFORMS, vol. 61(6), pages 1383-1398, December.
    19. Nieto, Maria Rosa & Ruiz, Esther, 2016. "Frontiers in VaR forecasting and backtesting," International Journal of Forecasting, Elsevier, vol. 32(2), pages 475-501.
    20. Tsyplakov, Alexander, 2014. "Theoretical guidelines for a partially informed forecast examiner," MPRA Paper 55017, University Library of Munich, Germany.
    21. West, Kenneth D & McCracken, Michael W, 1998. "Regression-Based Tests of Predictive Ability," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 39(4), pages 817-840, November.
    22. Christoffersen, Peter F, 1998. "Evaluating Interval Forecasts," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 39(4), pages 841-862, November.
    23. Olszewski, Wojciech, 2015. "Calibration and Expert Testing," Handbook of Game Theory with Economic Applications,, Elsevier.
    24. James Mitchell & Kenneth F. Wallis, 2011. "Evaluating density forecasts: forecast combinations, model mixtures, calibration and sharpness," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 26(6), pages 1023-1040, September.
    25. Tsyplakov, Alexander, 2011. "Evaluating density forecasts: a comment," MPRA Paper 31184, University Library of Munich, Germany.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Taylor, James W., 2021. "Evaluating quantile-bounded and expectile-bounded interval forecasts," International Journal of Forecasting, Elsevier, vol. 37(2), pages 800-811.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Barbara Rossi, 2019. "Forecasting in the presence of instabilities: How do we know whether models predict well and how to improve them," Economics Working Papers 1711, Department of Economics and Business, Universitat Pompeu Fabra, revised Jul 2021.
    2. Marc-Oliver Pohle, 2020. "The Murphy Decomposition and the Calibration-Resolution Principle: A New Perspective on Forecast Evaluation," Papers 2005.01835, arXiv.org.
    3. Tsyplakov, Alexander, 2014. "Theoretical guidelines for a partially informed forecast examiner," MPRA Paper 55017, University Library of Munich, Germany.
    4. Malte Knuppel & Fabian Kruger & Marc-Oliver Pohle, 2022. "Score-based calibration testing for multivariate forecast distributions," Papers 2211.16362, arXiv.org, revised Dec 2023.
    5. Gaglianone, Wagner Piazza & Marins, Jaqueline Terra Moura, 2017. "Evaluation of exchange rate point and density forecasts: An application to Brazil," International Journal of Forecasting, Elsevier, vol. 33(3), pages 707-728.
    6. Tsyplakov, Alexander, 2013. "Evaluation of Probabilistic Forecasts: Proper Scoring Rules and Moments," MPRA Paper 45186, University Library of Munich, Germany.
    7. Marcus P. A. Cobb, 2020. "Aggregate density forecasting from disaggregate components using Bayesian VARs," Empirical Economics, Springer, vol. 58(1), pages 287-312, January.
    8. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    9. Torben G. Andersen & Tim Bollerslev & Peter F. Christoffersen & Francis X. Diebold, 2005. "Volatility Forecasting," PIER Working Paper Archive 05-011, Penn Institute for Economic Research, Department of Economics, University of Pennsylvania.
    10. Andersen, Torben G. & Bollerslev, Tim & Christoffersen, Peter F. & Diebold, Francis X., 2006. "Volatility and Correlation Forecasting," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 15, pages 777-878, Elsevier.
    11. Tsyplakov, Alexander, 2012. "Assessment of probabilistic forecasts: Proper scoring rules and moments," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 27(3), pages 115-132.
    12. Diks, Cees & Panchenko, Valentyn & van Dijk, Dick, 2011. "Likelihood-based scoring rules for comparing density forecasts in tails," Journal of Econometrics, Elsevier, vol. 163(2), pages 215-230, August.
    13. Gordy, Michael B. & McNeil, Alexander J., 2020. "Spectral backtests of forecast distributions with application to risk management," Journal of Banking & Finance, Elsevier, vol. 116(C).
    14. Jonas Dovern & Hans Manner, 2020. "Order‐invariant tests for proper calibration of multivariate density forecasts," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 35(4), pages 440-456, June.
    15. Sander Barendse & Erik Kole & Dick van Dijk, 2023. "Backtesting Value-at-Risk and Expected Shortfall in the Presence of Estimation Error," Journal of Financial Econometrics, Oxford University Press, vol. 21(2), pages 528-568.
    16. Weron, Rafał, 2014. "Electricity price forecasting: A review of the state-of-the-art with a look into the future," International Journal of Forecasting, Elsevier, vol. 30(4), pages 1030-1081.
    17. Allen, Sam & Koh, Jonathan & Segers, Johan & Ziegel, Johanna, 2024. "Tail calibration of probabilistic forecasts," LIDAM Discussion Papers ISBA 2024018, Université catholique de Louvain, Institute of Statistics, Biostatistics and Actuarial Sciences (ISBA).
    18. Federico Bassetti & Roberto Casarin & Francesco Ravazzolo, 2019. "Density Forecasting," BEMPS - Bozen Economics & Management Paper Series BEMPS59, Faculty of Economics and Management at the Free University of Bozen.
    19. Paul Labonne, 2022. "Asymmetric Uncertainty: Nowcasting Using Skewness in Real-time Data," Economic Statistics Centre of Excellence (ESCoE) Discussion Papers ESCoE DP-2022-23, Economic Statistics Centre of Excellence (ESCoE).
    20. Taylor, James W., 2021. "Evaluating quantile-bounded and expectile-bounded interval forecasts," International Journal of Forecasting, Elsevier, vol. 37(2), pages 800-811.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:intfor:v:36:y:2020:i:4:p:1380-1388. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/ijforecast .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.