IDEAS home Printed from https://ideas.repec.org/a/wly/japmet/v36y2021i4p369-392.html
   My bibliography  Save this article

How far can we forecast? Statistical tests of the predictive content

Author

Listed:
  • Jörg Breitung
  • Malte Knüppel

Abstract

We develop tests for the null hypothesis that forecasts become uninformative beyond some maximum forecast horizon h∗. The forecast may result from a survey of forecasters or from an estimated parametric model. The first class of tests compares the mean‐squared prediction error of the forecast to the variance of the evaluation sample, whereas the second class of tests compares it with the mean‐squared prediction error of the recursive mean. We show that the forecast comparison may easily be performed by adopting the encompassing principle, which results in simple regression tests with standard asymptotic inference. Our tests are applied to forecasts of macroeconomic key variables from the survey of Consensus Economics. The results suggest that these forecasts are barely informative beyond two to four quarters ahead.

Suggested Citation

  • Jörg Breitung & Malte Knüppel, 2021. "How far can we forecast? Statistical tests of the predictive content," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 36(4), pages 369-392, June.
    • Handle: RePEc:wly:japmet:v:36:y:2021:i:4:p:369-392
    DOI: 10.1002/jae.2817
    as

    Download full text from publisher

    File URL: https://doi.org/10.1002/jae.2817
    Download Restriction: no
    File URL: https://libkey.io/10.1002/jae.2817?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. West, Kenneth D, 1996. "Asymptotic Inference about Predictive Ability," Econometrica, Econometric Society, vol. 64(5), pages 1067-1084, September.
    2. Andrew Patton & Allan Timmermann, 2012. "Forecast Rationality Tests Based on Multi-Horizon Bounds," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 30(1), pages 1-17.
    3. John W. Galbraith & Greg Tkacz, 2007. "Forecast content and content horizons for some important macroeconomic time series," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 40(3), pages 935-953, August.
    4. Francis X. Diebold & Lutz Kilian, 2001. "Measuring predictability: theory and macroeconomic applications," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 16(6), pages 657-669.
    5. Clark, Todd E. & West, Kenneth D., 2007. "Approximately normal tests for equal predictive accuracy in nested models," Journal of Econometrics, Elsevier, vol. 138(1), pages 291-311, May.
    6. Ang, Andrew & Bekaert, Geert & Wei, Min, 2007. "Do macro variables, asset markets, or surveys forecast inflation better?," Journal of Monetary Economics, Elsevier, vol. 54(4), pages 1163-1212, May.
    7. Jörg Breitung & Malte Knüppel, 2021. "How far can we forecast? Statistical tests of the predictive content," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 36(4), pages 369-392, June.
    8. Fama, Eugene F. & French, Kenneth R., 1988. "Dividend yields and expected stock returns," Journal of Financial Economics, Elsevier, vol. 22(1), pages 3-25, October.
    9. Isiklar, Gultekin & Lahiri, Kajal, 2007. "How far ahead can we forecast? Evidence from cross-country surveys," International Journal of Forecasting, Elsevier, vol. 23(2), pages 167-187.
    10. James H. Stock & Mark W. Watson, 2007. "Why Has U.S. Inflation Become Harder to Forecast?," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 39(s1), pages 3-33, February.
    11. Newey, Whitney & West, Kenneth, 2014. "A simple, positive semi-definite, heteroscedasticity and autocorrelation consistent covariance matrix," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 33(1), pages 125-132.
    12. Clements,Michael & Hendry,David, 1998. "Forecasting Economic Time Series," Cambridge Books, Cambridge University Press, number 9780521634809.
    13. Granger, C. W. J. & Newbold, Paul, 1986. "Forecasting Economic Time Series," Elsevier Monographs, Elsevier, edition 2, number 9780122951831 edited by Shell, Karl.
    14. Knüppel, Malte, 2018. "Forecast-error-based estimation of forecast uncertainty when the horizon is increased," International Journal of Forecasting, Elsevier, vol. 34(1), pages 105-116.
    15. Jacob A. Mincer & Victor Zarnowitz, 1969. "The Evaluation of Economic Forecasts," NBER Chapters, in: Economic Forecasts and Expectations: Analysis of Forecasting Behavior and Performance, pages 3-46, National Bureau of Economic Research, Inc.
    16. Raffaella Giacomini & Halbert White, 2006. "Tests of Conditional Predictive Ability," Econometrica, Econometric Society, vol. 74(6), pages 1545-1578, November.
    17. Clark, Todd E. & McCracken, Michael W., 2001. "Tests of equal forecast accuracy and encompassing for nested models," Journal of Econometrics, Elsevier, vol. 105(1), pages 85-110, November.
    18. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    19. Andrews, Donald W K, 1991. "Heteroskedasticity and Autocorrelation Consistent Covariance Matrix Estimation," Econometrica, Econometric Society, vol. 59(3), pages 817-858, May.
    20. Jacob A. Mincer, 1969. "Economic Forecasts and Expectations: Analysis of Forecasting Behavior and Performance," NBER Books, National Bureau of Economic Research, Inc, number minc69-1.
    21. Michael W. McCracken, 2019. "Tests of Conditional Predictive Ability: Some Simulation Evidence," Working Papers 2019-11, Federal Reserve Bank of St. Louis.
    22. Harvey, David & Leybourne, Stephen & Newbold, Paul, 1997. "Testing the equality of prediction mean squared errors," International Journal of Forecasting, Elsevier, vol. 13(2), pages 281-291, June.
    23. repec:bla:econom:v:42:y:1975:i:166:p:123-38 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Stamer, Vincent, 2022. "Thinking Outside the Container: A Sparse Partial Least Squares Approach to Forecasting Trade Flows," VfS Annual Conference 2022 (Basel): Big Data in Economics 264096, Verein für Socialpolitik / German Economic Association.
    2. Regmi, Hari & Kuethe, Todd H. & Foster, Kenneth A., 2022. "Evaluation of USDA's Agricultural Exports Projections," 2022 Annual Meeting, July 31-August 2, Anaheim, California 322363, Agricultural and Applied Economics Association.
    3. Daniel Borup & Bent Jesper Christensen & Yunus Emre Ergemen, 2019. "Assessing predictive accuracy in panel data models with long-range dependence," CREATES Research Papers 2019-04, Department of Economics and Business Economics, Aarhus University.
    4. Jean-Yves Pitarakis, 2023. "Direct Multi-Step Forecast based Comparison of Nested Models via an Encompassing Test," Papers 2312.16099, arXiv.org.
    5. Zhu, Tiantian & Haugen, Stein & Liu, Yiliu & Yang, Xue, 2023. "A value of prediction model to estimate optimal response time to threats for accident prevention," Reliability Engineering and System Safety, Elsevier, vol. 232(C).
    6. Carola Binder & Tucker S. Mcelroy & Xuguang S. Sheng, 2022. "The Term Structure of Uncertainty: New Evidence from Survey Expectations," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 54(1), pages 39-71, February.
    7. Fabian Kruger & Hendrik Plett, 2022. "Prediction intervals for economic fixed-event forecasts," Papers 2210.13562, arXiv.org, revised Mar 2024.
    8. Cepni, Oguzhan & Clements, Michael P., 2024. "How local is the local inflation factor? Evidence from emerging European countries," International Journal of Forecasting, Elsevier, vol. 40(1), pages 160-183.
    9. Jörg Breitung & Malte Knüppel, 2021. "How far can we forecast? Statistical tests of the predictive content," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 36(4), pages 369-392, June.
    10. Jorge Abad & Javier Suarez, 2018. "The Procyclicality of Expected Credit Loss Provisions," Working Papers wp2018_1806, CEMFI.
    11. Marc-Oliver Pohle, 2020. "The Murphy Decomposition and the Calibration-Resolution Principle: A New Perspective on Forecast Evaluation," Papers 2005.01835, arXiv.org.
    12. M. Chudý & S. Karmakar & W. B. Wu, 2020. "Long-term prediction intervals of economic time series," Empirical Economics, Springer, vol. 58(1), pages 191-222, January.
    13. repec:ags:aaea22:335690 is not listed on IDEAS
    14. Jörg Döpke & Karsten Müller & Lars Tegtmeier, 2023. "Moments of cross‐sectional stock market returns and the German business cycle," Economic Notes, Banca Monte dei Paschi di Siena SpA, vol. 52(2), July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rossi, Barbara, 2013. "Advances in Forecasting under Instability," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1203-1324, Elsevier.
    2. Pablo Pincheira Brown & Nicolás Hardy, 2024. "Correlation‐based tests of predictability," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 43(6), pages 1835-1858, September.
    3. Raffaella Giacomini & Barbara Rossi, 2013. "Forecasting in macroeconomics," Chapters, in: Nigar Hashimzade & Michael A. Thornton (ed.), Handbook of Research Methods and Applications in Empirical Macroeconomics, chapter 17, pages 381-408, Edward Elgar Publishing.
    4. Marc-Oliver Pohle, 2020. "The Murphy Decomposition and the Calibration-Resolution Principle: A New Perspective on Forecast Evaluation," Papers 2005.01835, arXiv.org.
    5. Barbara Rossi & Atsushi Inoue, 2012. "Out-of-Sample Forecast Tests Robust to the Choice of Window Size," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 30(3), pages 432-453, April.
    6. Clark, Todd & McCracken, Michael, 2013. "Advances in Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1107-1201, Elsevier.
    7. Cepni, Oguzhan & Clements, Michael P., 2024. "How local is the local inflation factor? Evidence from emerging European countries," International Journal of Forecasting, Elsevier, vol. 40(1), pages 160-183.
    8. Rossi, Barbara & Sekhposyan, Tatevik, 2011. "Understanding models' forecasting performance," Journal of Econometrics, Elsevier, vol. 164(1), pages 158-172, September.
    9. Busetti, Fabio & Marcucci, Juri, 2013. "Comparing forecast accuracy: A Monte Carlo investigation," International Journal of Forecasting, Elsevier, vol. 29(1), pages 13-27.
    10. Nathan Goldstein & Ben‐Zion Zilberfarb, 2023. "The closer we get, the better we are?," Economic Inquiry, Western Economic Association International, vol. 61(2), pages 364-376, April.
    11. Barbara Rossi, 2019. "Forecasting in the presence of instabilities: How do we know whether models predict well and how to improve them," Economics Working Papers 1711, Department of Economics and Business, Universitat Pompeu Fabra, revised Jul 2021.
    12. Pablo Pincheira Brown & Nicolás Hardy, 2024. "The mean squared prediction error paradox," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 43(6), pages 2298-2321, September.
    13. Matei Demetrescu & Christoph Hanck & Robinson Kruse‐Becher, 2022. "Robust inference under time‐varying volatility: A real‐time evaluation of professional forecasters," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(5), pages 1010-1030, August.
    14. West, Kenneth D., 2006. "Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 3, pages 99-134, Elsevier.
    15. Raffaella Giacomini & Halbert White, 2006. "Tests of Conditional Predictive Ability," Econometrica, Econometric Society, vol. 74(6), pages 1545-1578, November.
    16. Firmin Doko Tchatoka & Qazi Haque, 2023. "On bootstrapping tests of equal forecast accuracy for nested models," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 42(7), pages 1844-1864, November.
    17. Kontogeorgos, Georgios & Lambrias, Kyriacos, 2019. "An analysis of the Eurosystem/ECB projections," Working Paper Series 2291, European Central Bank.
    18. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    19. Granziera, Eleonora & Sekhposyan, Tatevik, 2019. "Predicting relative forecasting performance: An empirical investigation," International Journal of Forecasting, Elsevier, vol. 35(4), pages 1636-1657.
    20. Faust, Jon & Wright, Jonathan H., 2013. "Forecasting Inflation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 2-56, Elsevier.

    More about this item

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C32 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models; Diffusion Processes; State Space Models
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:japmet:v:36:y:2021:i:4:p:369-392. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.interscience.wiley.com/jpages/0883-7252/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.