IDEAS home Printed from https://ideas.repec.org/a/jae/japmet/v25y2010i6p1028-1062.html
   My bibliography  Save this article

Forecast encompassing tests and probability forecasts

Author

Listed:
  • Michael P. Clements

    (Department of Economics, University of Warwick, Coventry, UK)

  • David I. Harvey

    (School of Economics, University of Nottingham, UK)

Abstract

We consider tests of forecast encompassing for probability forecasts, for both quadratic and logarithmic scoring rules. We propose test statistics for the null of forecast encompassing, present the limiting distributions of the test statistics, and investigate the impact of estimating the forecasting models' parameters on these distributions. The small-sample performance is investigated, in terms of small numbers of forecasts and model estimation sample sizes. We show the usefulness of the tests for the evaluation of recession probability forecasts from logit models with different leading indicators as explanatory variables, and for evaluating survey-based probability forecasts. Copyright © 2009 John Wiley & Sons, Ltd.

Suggested Citation

  • Michael P. Clements & David I. Harvey, 2010. "Forecast encompassing tests and probability forecasts," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(6), pages 1028-1062.
    • Handle: RePEc:jae:japmet:v:25:y:2010:i:6:p:1028-1062
    DOI: 10.1002/jae.1097
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1002/jae.1097
    File Function: Link to full text; subscription required
    Download Restriction: no
    File URL: http://qed.econ.queensu.ca:80/jae/2010-v25.6/
    File Function: Supporting data files and programs
    Download Restriction: no
    File URL: https://libkey.io/10.1002/jae.1097?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Hamilton, James D., 1996. "This is what happened to the oil price-macroeconomy relationship," Journal of Monetary Economics, Elsevier, vol. 38(2), pages 215-220, October.
    2. Giacomini, Raffaella & Komunjer, Ivana, 2005. "Evaluation and Combination of Conditional Quantile Forecasts," Journal of Business & Economic Statistics, American Statistical Association, vol. 23, pages 416-431, October.
    3. Evan F. Koenig & Sheila Dolmas & Jeremy Piger, 2003. "The Use and Abuse of Real-Time Data in Economic Forecasting," The Review of Economics and Statistics, MIT Press, vol. 85(3), pages 618-628, August.
    4. Alan A. Carruth & Mark A. Hooker & Andrew J. Oswald, 1998. "Unemployment Equilibria And Input Prices: Theory And Evidence From The United States," The Review of Economics and Statistics, MIT Press, vol. 80(4), pages 621-628, November.
    5. Robert T. Clemen & Robert L. Winkler, 1999. "Combining Probability Distributions From Experts in Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 19(2), pages 187-203, April.
    6. Clark, Todd E. & McCracken, Michael W., 2001. "Tests of equal forecast accuracy and encompassing for nested models," Journal of Econometrics, Elsevier, vol. 105(1), pages 85-110, November.
    7. Fair, Ray C & Shiller, Robert J, 1990. "Comparing Information in Forecasts from Econometric Models," American Economic Review, American Economic Association, vol. 80(3), pages 375-389, June.
    8. West, Kenneth D, 2001. "Tests for Forecast Encompassing When Forecasts Depend on Estimated Regression Parameters," Journal of Business & Economic Statistics, American Statistical Association, vol. 19(1), pages 29-33, January.
    9. Stark, Tom & Croushore, Dean, 2002. "Forecasting with a real-time data set for macroeconomists," Journal of Macroeconomics, Elsevier, vol. 24(4), pages 507-531, December.
    10. Hooker, Mark A., 1996. "What happened to the oil price-macroeconomy relationship?," Journal of Monetary Economics, Elsevier, vol. 38(2), pages 195-213, October.
    11. Arturo Estrella & Frederic S. Mishkin, 1998. "Predicting U.S. Recessions: Financial Variables As Leading Indicators," The Review of Economics and Statistics, MIT Press, vol. 80(1), pages 45-61, February.
    12. Chris Birchenhall & Denise Osborn & Marianne Sensier, 2001. "Predicting UK Business Cycle Regimes," Scottish Journal of Political Economy, Scottish Economic Society, vol. 48(2), pages 179-195, May.
    13. Croushore, Dean & Stark, Tom, 2001. "A real-time data set for macroeconomists," Journal of Econometrics, Elsevier, vol. 105(1), pages 111-130, November.
    14. Hamilton, James Douglas & Kim, Dong Heon, 2000. "A Re-examination of the Predictability of Economic Activity Using the Yield Spread," University of California at San Diego, Economics Working Paper Series qt69v8p1m9, Department of Economics, UC San Diego.
    15. Harvey, David I & Leybourne, Stephen J & Newbold, Paul, 1998. "Tests for Forecast Encompassing," Journal of Business & Economic Statistics, American Statistical Association, vol. 16(2), pages 254-259, April.
    16. West, Kenneth D, 1996. "Asymptotic Inference about Predictive Ability," Econometrica, Econometric Society, vol. 64(5), pages 1067-1084, September.
    17. Clements, M.P. & Hendry, D., 1992. "On the Limitations of Comparing Mean Square Forecast Errors," Economics Series Working Papers 99138, University of Oxford, Department of Economics.
    18. Hamilton, James D., 2003. "What is an oil shock?," Journal of Econometrics, Elsevier, vol. 113(2), pages 363-398, April.
    19. Kamstra, Mark & Kennedy, Peter, 1998. "Combining qualitative forecasts using logit," International Journal of Forecasting, Elsevier, vol. 14(1), pages 83-93, March.
    20. Clemen, Robert T., 1989. "Combining forecasts: A review and annotated bibliography," International Journal of Forecasting, Elsevier, vol. 5(4), pages 559-583.
    21. Anderson, Heather M. & Vahid, Farshid, 2001. "Predicting The Probability Of A Recession With Nonlinear Autoregressive Leading-Indicator Models," Macroeconomic Dynamics, Cambridge University Press, vol. 5(4), pages 482-505, September.
    22. Francis X. Diebold & Jose A. Lopez, 1995. "Forecast evaluation and combination," Research Paper 9525, Federal Reserve Bank of New York.
    23. R. Winkler & Javier Muñoz & José Cervera & José Bernardo & Gail Blattenberger & Joseph Kadane & Dennis Lindley & Allan Murphy & Robert Oliver & David Ríos-Insua, 1996. "Scoring rules and the evaluation of probabilities," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 5(1), pages 1-60, June.
    24. West, Kenneth D., 2006. "Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 3, pages 99-134, Elsevier.
    25. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    26. Yock Y. Chong & David F. Hendry, 1986. "Econometric Evaluation of Linear Macro-Economic Models," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 53(4), pages 671-690.
    27. Hamilton, James D & Kim, Dong Heon, 2002. "A Reexamination of the Predictability of Economic Activity Using the Yield Spread," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 34(2), pages 340-360, May.
    28. Hans-Martin Krolzig & Michael P. Clements, 2002. "Can oil shocks explain asymmetries in the US Business Cycle?," Empirical Economics, Springer, vol. 27(2), pages 185-204.
    29. Mork, Knut Anton, 1989. "Oil and Macroeconomy When Prices Go Up and Down: An Extension of Hamilton's Results," Journal of Political Economy, University of Chicago Press, vol. 97(3), pages 740-744, June.
    30. Hooker, Mark A., 1996. "This is what happened to the oil price-macroeconomy relationship: Reply," Journal of Monetary Economics, Elsevier, vol. 38(2), pages 221-222, October.
    31. Hamilton, James D, 1983. "Oil and the Macroeconomy since World War II," Journal of Political Economy, University of Chicago Press, vol. 91(2), pages 228-248, April.
    32. West, Kenneth D & McCracken, Michael W, 1998. "Regression-Based Tests of Predictive Ability," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 39(4), pages 817-840, November.
    33. Diebold, Francis X & Rudebusch, Glenn D, 1989. "Scoring the Leading Indicators," The Journal of Business, University of Chicago Press, vol. 62(3), pages 369-391, July.
    34. Faust, Jon & Rogers, John H & Wright, Jonathan H, 2005. "News and Noise in G-7 GDP Announcements," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 37(3), pages 403-419, June.
    35. Faust, Jon & Rogers, John H. & H. Wright, Jonathan, 2003. "Exchange rate forecasting: the errors we've really made," Journal of International Economics, Elsevier, vol. 60(1), pages 35-59, May.
    36. Kiseok Lee & Shawn Ni & Ronald A. Ratti, 1995. "Oil Shocks and the Macroeconomy: The Role of Price Variability," The Energy Journal, International Association for Energy Economics, vol. 0(Number 4), pages 39-56.
    37. G. Elliott & C. Granger & A. Timmermann (ed.), 2006. "Handbook of Economic Forecasting," Handbook of Economic Forecasting, Elsevier, edition 1, volume 1, number 1.
    38. Nelson, Charles R, 1972. "The Prediction Performance of the FRB-MIT-PENN Model of the U.S. Economy," American Economic Review, American Economic Association, vol. 62(5), pages 902-917, December.
    39. Granger, C.W.J. & Pesaran, M. H., 1999. "Economic and Statistical Measures of Forecast Accuracy," Cambridge Working Papers in Economics 9910, Faculty of Economics, University of Cambridge.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Clements, Michael P., 2010. "Explanations of the inconsistencies in survey respondents' forecasts," European Economic Review, Elsevier, vol. 54(4), pages 536-549, May.
    2. Rodrigues, Bruno Dore & Stevenson, Maxwell J., 2013. "Takeover prediction using forecast combinations," International Journal of Forecasting, Elsevier, vol. 29(4), pages 628-641.
    3. Lahiri, Kajal & Peng, Huaming & Zhao, Yongchen, 2015. "Testing the value of probability forecasts for calibrated combining," International Journal of Forecasting, Elsevier, vol. 31(1), pages 113-129.
    4. Lahiri, Kajal & Wang, J. George, 2013. "Evaluating probability forecasts for GDP declines using alternative methodologies," International Journal of Forecasting, Elsevier, vol. 29(1), pages 175-190.
    5. Oliver Hülsewig & Johannes Mayr & Stéphane Sorbe, 2007. "Assessing the Forecast Properties of the CESifo World Economic Climate Indicator: Evidence for the Euro Area," ifo Working Paper Series 46, ifo Institute - Leibniz Institute for Economic Research at the University of Munich.
    6. Michael P. Clements, 2011. "An Empirical Investigation of the Effects of Rounding on the SPF Probabilities of Decline and Output Growth Histograms," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 43(1), pages 207-220, February.
    7. Dimitriadis, Timo & Schnaitmann, Julie, 2021. "Forecast encompassing tests for the expected shortfall," International Journal of Forecasting, Elsevier, vol. 37(2), pages 604-621.
    8. Michał Rubaszek & Paweł Skrzypczyński & Grzegorz Koloch, 2010. "Forecasting the Polish Zloty with Non-Linear Models," Central European Journal of Economic Modelling and Econometrics, Central European Journal of Economic Modelling and Econometrics, vol. 2(2), pages 151-167, March.
    9. Timmermann, Allan, 2006. "Forecast Combinations," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 4, pages 135-196, Elsevier.
    10. Dimitriadis, Timo & Liu, Xiaochun & Schnaitmann, Julie, 2020. "Encompassing tests for value at risk and expected shortfall multi-step forecasts based on inference on the boundary," Hohenheim Discussion Papers in Business, Economics and Social Sciences 11-2020, University of Hohenheim, Faculty of Business, Economics and Social Sciences.
    11. Rendon-Sanchez, Juan F. & de Menezes, Lilian M., 2019. "Structural combination of seasonal exponential smoothing forecasts applied to load forecasting," European Journal of Operational Research, Elsevier, vol. 275(3), pages 916-924.
    12. Cepni, Oguzhan & Clements, Michael P., 2024. "How local is the local inflation factor? Evidence from emerging European countries," International Journal of Forecasting, Elsevier, vol. 40(1), pages 160-183.
    13. Turgut Kisinbay & Chikako Baba, 2011. "Predicting Recessions: A New Approach for Identifying Leading Indicators and Forecast Combinations," IMF Working Papers 2011/235, International Monetary Fund.
    14. Clements, Michael P., 2016. "Long-run restrictions and survey forecasts of output, consumption and investment," International Journal of Forecasting, Elsevier, vol. 32(3), pages 614-628.
    15. Clements, Michael P., 2008. "Consensus and uncertainty: Using forecast probabilities of output declines," International Journal of Forecasting, Elsevier, vol. 24(1), pages 76-86.
    16. Timo Dimitriadis & Julie Schnaitmann, 2019. "Forecast Encompassing Tests for the Expected Shortfall," Papers 1908.04569, arXiv.org, revised Aug 2020.
    17. David G. McMillan & Mark E. Wohar, 2010. "Stock return predictability and dividend-price ratio: a nonlinear approach," International Journal of Finance & Economics, John Wiley & Sons, Ltd., vol. 15(4), pages 351-365.
    18. Tsyplakov, Alexander, 2014. "Theoretical guidelines for a partially informed forecast examiner," MPRA Paper 55017, University Library of Munich, Germany.
    19. Guizzardi, Andrea & Stacchini, Annalisa, 2015. "Real-time forecasting regional tourism with business sentiment surveys," Tourism Management, Elsevier, vol. 47(C), pages 213-223.
    20. Xiao, Liye & Wang, Jianzhou & Hou, Ru & Wu, Jie, 2015. "A combined model based on data pre-analysis and weight coefficients optimization for electrical load forecasting," Energy, Elsevier, vol. 82(C), pages 524-549.
    21. Stahl, Dale O., 2018. "Assessing the forecast performance of models of choice," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 73(C), pages 86-92.
    22. Michael P. Clements, 2020. "Are Some Forecasters’ Probability Assessments of Macro Variables Better Than Those of Others?," Econometrics, MDPI, vol. 8(2), pages 1-16, May.
    23. Lu Wang & Shan Li & Chao Liang, 2024. "Exploring the impact of oil security attention on oil volatility: A new perspective," International Finance, Wiley Blackwell, vol. 27(1), pages 61-80, April.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Clements, Michael P. & Harvey, David I., 2011. "Combining probability forecasts," International Journal of Forecasting, Elsevier, vol. 27(2), pages 208-223.
    2. Clements, Michael P. & Galvao, Ana Beatriz, 2006. "Macroeconomic Forecasting with Mixed Frequency Data: Forecasting US output growth and inflation," Economic Research Papers 269743, University of Warwick - Department of Economics.
    3. Jaime Casassus & Freddy Higuera, 2011. "Stock Return Predictability and Oil Prices," Documentos de Trabajo 406, Instituto de Economia. Pontificia Universidad Católica de Chile..
    4. Raffaella Giacomini & Barbara Rossi, 2013. "Forecasting in macroeconomics," Chapters, in: Nigar Hashimzade & Michael A. Thornton (ed.), Handbook of Research Methods and Applications in Empirical Macroeconomics, chapter 17, pages 381-408, Edward Elgar Publishing.
    5. Clements, Michael P. & Franses, Philip Hans & Swanson, Norman R., 2004. "Forecasting economic and financial time-series with non-linear models," International Journal of Forecasting, Elsevier, vol. 20(2), pages 169-183.
    6. Malik, Farooq & Nasereddin, Mahdi, 2006. "Forecasting output using oil prices: A cascaded artificial neural network approach," Journal of Economics and Business, Elsevier, vol. 58(2), pages 168-180.
    7. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    8. Rossi, Barbara, 2013. "Advances in Forecasting under Instability," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1203-1324, Elsevier.
    9. Lizardo, Radhamés A. & Mollick, André V., 2010. "Oil price fluctuations and U.S. dollar exchange rates," Energy Economics, Elsevier, vol. 32(2), pages 399-408, March.
    10. Clark, Todd & McCracken, Michael, 2013. "Advances in Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1107-1201, Elsevier.
    11. Rebeca Jiménez-Rodríguez, 2004. "Oil Price Shocks: Testing for Non-linearity," CSEF Working Papers 115, Centre for Studies in Economics and Finance (CSEF), University of Naples, Italy.
    12. Michael P. Clements & Ana Beatriz Galvão, 2007. "Macroeconomic Forecasting with Mixed Frequency Data: Forecasting US Output Growth," Working Papers 616, Queen Mary University of London, School of Economics and Finance.
    13. Engemann, Kristie M. & Kliesen, Kevin L. & Owyang, Michael T., 2011. "Do Oil Shocks Drive Business Cycles? Some U.S. And International Evidence," Macroeconomic Dynamics, Cambridge University Press, vol. 15(S3), pages 498-517, November.
    14. Clark, Todd E. & West, Kenneth D., 2007. "Approximately normal tests for equal predictive accuracy in nested models," Journal of Econometrics, Elsevier, vol. 138(1), pages 291-311, May.
    15. Lang, Korbinian & Auer, Benjamin R., 2020. "The economic and financial properties of crude oil: A review," The North American Journal of Economics and Finance, Elsevier, vol. 52(C).
    16. Ravazzolo Francesco & Rothman Philip, 2016. "Oil-price density forecasts of US GDP," Studies in Nonlinear Dynamics & Econometrics, De Gruyter, vol. 20(4), pages 441-453, September.
    17. Domenico Ferraro & Kenneth S. Rogoff & Barbara Rossi, 2011. "Can oil prices forecast exchange rates?," Working Papers 11-34, Federal Reserve Bank of Philadelphia.
    18. Ana Gómez-Loscos & Mar𨀠 Dolores Gadea & Antonio Montañ鳠, 2012. "Economic growth, inflation and oil shocks: are the 1970s coming back?," Applied Economics, Taylor & Francis Journals, vol. 44(35), pages 4575-4589, December.
    19. Clark, Todd E. & McCracken, Michael W., 2001. "Tests of equal forecast accuracy and encompassing for nested models," Journal of Econometrics, Elsevier, vol. 105(1), pages 85-110, November.
    20. Matteo Manera & Alessandro Cologni, 2006. "The Asymmetric Effects of Oil Shocks on Output Growth: A Markov-Switching Analysis for the G-7 Countries," Working Papers 2006.29, Fondazione Eni Enrico Mattei.

    More about this item

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C15 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Statistical Simulation Methods: General
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods

    Lists

    This item is featured on the following reading lists, Wikipedia, or ReplicationWiki pages:
    1. Forecast encompassing tests and probability forecasts (Journal of Applied Econometrics 2010) in ReplicationWiki

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:jae:japmet:v:25:y:2010:i:6:p:1028-1062. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley-Blackwell Digital Licensing or Christopher F. Baum (email available below). General contact details of provider: http://www.interscience.wiley.com/jpages/0883-7252/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.