IDEAS home Printed from https://ideas.repec.org/a/bla/jtsera/v39y2018i6p953-965.html
   My bibliography  Save this article

On the Comparison of Interval Forecasts

Author

Listed:
  • Ross Askanazi
  • Francis X. Diebold
  • Frank Schorfheide
  • Minchul Shin

Abstract

We explore interval forecast comparison when the nominal confidence level is specified, but the quantiles on which intervals are based are not specified. It turns out that the problem is difficult, and perhaps unsolvable. We first consider a situation where intervals meet the Christoffersen conditions (in particular, where they are correctly calibrated), in which case the common prescription, which we rationalize and explore, is to prefer the interval of shortest length. We then allow for mis‐calibrated intervals, in which case there is a calibration‐length tradeoff. We propose two natural conditions that interval forecast loss functions should meet in such environments, and we show that a variety of popular approaches to interval forecast comparison fail them. Our negative results strengthen the case for abandoning interval forecasts in favor of density forecasts: Density forecasts not only provide richer information, but also can be readily compared using known proper scoring rules like the log predictive score, whereas interval forecasts cannot.

Suggested Citation

  • Ross Askanazi & Francis X. Diebold & Frank Schorfheide & Minchul Shin, 2018. "On the Comparison of Interval Forecasts," Journal of Time Series Analysis, Wiley Blackwell, vol. 39(6), pages 953-965, November.
    • Handle: RePEc:bla:jtsera:v:39:y:2018:i:6:p:953-965
    DOI: 10.1111/jtsa.12426
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/jtsa.12426
    Download Restriction: no
    File URL: https://libkey.io/10.1111/jtsa.12426?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Christoffersen, Peter F, 1998. "Evaluating Interval Forecasts," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 39(4), pages 841-862, November.
    2. Gneiting, Tilmann, 2011. "Quantiles as optimal point forecasts," International Journal of Forecasting, Elsevier, vol. 27(2), pages 197-207.
    3. Gianni Amisano & John Geweke, 2017. "Prediction Using Several Macroeconomic Models," The Review of Economics and Statistics, MIT Press, vol. 99(5), pages 912-925, December.
    4. Giacomini, Raffaella & Komunjer, Ivana, 2005. "Evaluation and Combination of Conditional Quantile Forecasts," Journal of Business & Economic Statistics, American Statistical Association, vol. 23, pages 416-431, October.
    5. Jason J. Wu, 2012. "Semiparametric forecast intervals," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 31(3), pages 189-228, April.
    6. Edward P. Herbst & Frank Schorfheide, 2016. "Bayesian Estimation of DSGE Models," Economics Books, Princeton University Press, edition 1, number 10612.
    7. Gneiting, Tilmann, 2011. "Making and Evaluating Point Forecasts," Journal of the American Statistical Association, American Statistical Association, vol. 106(494), pages 746-762.
    8. Granger, C. W. J. & White, Halbert & Kamstra, Mark, 1989. "Interval forecasting : An analysis based upon ARCH-quantile estimators," Journal of Econometrics, Elsevier, vol. 40(1), pages 87-96, January.
    9. Schmalensee, Richard, 1976. "An Experimental Study of Expectation Formation," Econometrica, Econometric Society, vol. 44(1), pages 17-41, January.
    10. Diebold, Francis X & Gunther, Todd A & Tay, Anthony S, 1998. "Evaluating Density Forecasts with Applications to Financial Risk Management," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 39(4), pages 863-883, November.
    11. Gneiting, Tilmann, 2011. "Quantiles as optimal point forecasts," International Journal of Forecasting, Elsevier, vol. 27(2), pages 197-207, April.
    12. Hyndman, Rob J. & Koehler, Anne B., 2006. "Another look at measures of forecast accuracy," International Journal of Forecasting, Elsevier, vol. 22(4), pages 679-688.
    13. Polonik, Wolfgang & Yao, Qiwei, 2000. "Conditional minimum volume predictive regions for stochastic processes," LSE Research Online Documents on Economics 6311, London School of Economics and Political Science, LSE Library.
    14. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    15. repec:cup:judgdm:v:10:y:2015:i:5:p:456-468 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    2. Laura Liu & Hyungsik Roger Moon & Frank Schorfheide, 2023. "Forecasting with a panel Tobit model," Quantitative Economics, Econometric Society, vol. 14(1), pages 117-159, January.
    3. James Mitchell & Martin Weale, 2023. "Censored density forecasts: Production and evaluation," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(5), pages 714-734, August.
    4. Diebold, Francis X. & Shin, Minchul & Zhang, Boyuan, 2023. "On the aggregation of probability assessments: Regularized mixtures of predictive densities for Eurozone inflation and real interest rates," Journal of Econometrics, Elsevier, vol. 237(2).
    5. Liu, Laura & Moon, Hyungsik Roger & Schorfheide, Frank, 2021. "Panel forecasts of country-level Covid-19 infections," Journal of Econometrics, Elsevier, vol. 220(1), pages 2-22.
    6. Hyndman, Rob J., 2020. "A brief history of forecasting competitions," International Journal of Forecasting, Elsevier, vol. 36(1), pages 7-14.
    7. Makridakis, Spyros & Hyndman, Rob J. & Petropoulos, Fotios, 2020. "Forecasting in social settings: The state of the art," International Journal of Forecasting, Elsevier, vol. 36(1), pages 15-28.
    8. Francisca Corpas-Burgos & Miguel A. Martinez-Beneito, 2021. "An Autoregressive Disease Mapping Model for Spatio-Temporal Forecasting," Mathematics, MDPI, vol. 9(4), pages 1-17, February.
    9. Sayar Karmakar & Marek Chudý & Wei Biao Wu, 2022. "Long‐term prediction intervals with many covariates," Journal of Time Series Analysis, Wiley Blackwell, vol. 43(4), pages 587-609, July.
    10. Arias, Jonas E. & Rubio-Ramírez, Juan F. & Shin, Minchul, 2023. "Macroeconomic forecasting and variable ordering in multivariate stochastic volatility models," Journal of Econometrics, Elsevier, vol. 235(2), pages 1054-1086.
    11. Nico Keilman, 2020. "Evaluating Probabilistic Population Forecasts," Economie et Statistique / Economics and Statistics, Institut National de la Statistique et des Etudes Economiques (INSEE), issue 520-521, pages 49-64.
    12. Spyros Makridakis & Chris Fry & Fotios Petropoulos & Evangelos Spiliotis, 2022. "The Future of Forecasting Competitions: Design Attributes and Principles," INFORMS Joural on Data Science, INFORMS, vol. 1(1), pages 96-113, April.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Gensler, André & Sick, Bernhard & Vogt, Stephan, 2018. "A review of uncertainty representations and metaverification of uncertainty assessment techniques for renewable energies," Renewable and Sustainable Energy Reviews, Elsevier, vol. 96(C), pages 352-379.
    2. Gneiting, Tilmann, 2011. "Making and Evaluating Point Forecasts," Journal of the American Statistical Association, American Statistical Association, vol. 106(494), pages 746-762.
    3. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    4. Ruben Loaiza‐Maya & Gael M. Martin & David T. Frazier, 2021. "Focused Bayesian prediction," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 36(5), pages 517-543, August.
    5. George Athanasopoulos & Nikolaos Kourentzes, 2021. "On the Evaluation of Hierarchical Forecasts," Monash Econometrics and Business Statistics Working Papers 10/21, Monash University, Department of Econometrics and Business Statistics.
    6. Athanasopoulos, George & Kourentzes, Nikolaos, 2023. "On the evaluation of hierarchical forecasts," International Journal of Forecasting, Elsevier, vol. 39(4), pages 1502-1511.
    7. Komunjer, Ivana, 2013. "Quantile Prediction," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 961-994, Elsevier.
    8. Timo Dimitriadis & Tobias Fissler & Johanna Ziegel, 2020. "The Efficiency Gap," Papers 2010.14146, arXiv.org, revised Sep 2022.
    9. Gaglianone, Wagner Piazza & Marins, Jaqueline Terra Moura, 2017. "Evaluation of exchange rate point and density forecasts: An application to Brazil," International Journal of Forecasting, Elsevier, vol. 33(3), pages 707-728.
    10. Nowotarski, Jakub & Weron, Rafał, 2018. "Recent advances in electricity price forecasting: A review of probabilistic forecasting," Renewable and Sustainable Energy Reviews, Elsevier, vol. 81(P1), pages 1548-1568.
    11. Trapero, Juan R. & Cardós, Manuel & Kourentzes, Nikolaos, 2019. "Quantile forecast optimal combination to enhance safety stock estimation," International Journal of Forecasting, Elsevier, vol. 35(1), pages 239-250.
    12. Li, Yushu & Andersson, Jonas, 2014. "A Likelihood Ratio and Markov Chain Based Method to Evaluate Density Forecasting," Discussion Papers 2014/12, Norwegian School of Economics, Department of Business and Management Science.
    13. Yushu Li & Jonas Andersson, 2020. "A likelihood ratio and Markov chain‐based method to evaluate density forecasting," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 39(1), pages 47-55, January.
    14. Geweke, John & Amisano, Gianni, 2011. "Optimal prediction pools," Journal of Econometrics, Elsevier, vol. 164(1), pages 130-141, September.
    15. Antonio Bracale & Pasquale De Falco, 2015. "An Advanced Bayesian Method for Short-Term Probabilistic Forecasting of the Generation of Wind Power," Energies, MDPI, vol. 8(9), pages 1-22, September.
    16. Diks, Cees & Panchenko, Valentyn & van Dijk, Dick, 2011. "Likelihood-based scoring rules for comparing density forecasts in tails," Journal of Econometrics, Elsevier, vol. 163(2), pages 215-230, August.
    17. Tsyplakov, Alexander, 2013. "Evaluation of Probabilistic Forecasts: Proper Scoring Rules and Moments," MPRA Paper 45186, University Library of Munich, Germany.
    18. Makridakis, Spyros & Spiliotis, Evangelos & Assimakopoulos, Vassilios & Chen, Zhi & Gaba, Anil & Tsetlin, Ilia & Winkler, Robert L., 2022. "The M5 uncertainty competition: Results, findings and conclusions," International Journal of Forecasting, Elsevier, vol. 38(4), pages 1365-1385.
    19. Martin, Gael M. & Loaiza-Maya, Rubén & Maneesoonthorn, Worapree & Frazier, David T. & Ramírez-Hassan, Andrés, 2022. "Optimal probabilistic forecasts: When do they work?," International Journal of Forecasting, Elsevier, vol. 38(1), pages 384-406.
    20. Berrisch, Jonathan & Ziel, Florian, 2023. "CRPS learning," Journal of Econometrics, Elsevier, vol. 237(2).

    More about this item

    JEL classification:

    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jtsera:v:39:y:2018:i:6:p:953-965. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.blackwellpublishing.com/journal.asp?ref=0143-9782 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.