IDEAS home Printed from https://ideas.repec.org/p/fip/fedlwp/2009-051.html
   My bibliography  Save this paper

In-sample tests of predictive ability: a new approach

Author

Abstract

This paper presents analytical, Monte Carlo, and empirical evidence linking in-sample tests of predictive content and out-of-sample forecast accuracy. Our approach focuses on the negative effect that finite-sample estimation error has on forecast accuracy despite the presence of significant population-level predictive content. Specifically, we derive simple-to-use in-sample tests that test not only whether a particular variable has predictive content but also whether this content is estimated precisely enough to improve forecast accuracy. Our tests are asymptotically non-central chi-square or non-central normal. We provide a convenient bootstrap method for computing the relevant critical values. In the Monte Carlo and empirical analysis, we compare the effectiveness of our testing procedure with more common testing procedures.

Suggested Citation

  • Todd E. Clark & Michael W. McCracken, 2009. "In-sample tests of predictive ability: a new approach," Working Papers 2009-051, Federal Reserve Bank of St. Louis.
  • Handle: RePEc:fip:fedlwp:2009-051
    as

    Download full text from publisher

    File URL: http://research.stlouisfed.org/wp/2009/2009-051.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Ivo Welch & Amit Goyal, 2008. "A Comprehensive Look at The Empirical Performance of Equity Premium Prediction," The Review of Financial Studies, Society for Financial Studies, vol. 21(4), pages 1455-1508, July.
    2. Andrew Ang & Geert Bekaert, 2007. "Stock Return Predictability: Is it There?," The Review of Financial Studies, Society for Financial Studies, vol. 20(3), pages 651-707.
    3. Torous, Walter & Valkanov, Rossen, 2000. "Boundaries of Predictability: Noisy Predictive Regressions," University of California at Los Angeles, Anderson Graduate School of Management qt33p7672z, Anderson Graduate School of Management, UCLA.
    4. Clark, Todd E. & McCracken, Michael W., 2015. "Nested forecast model comparisons: A new approach to testing equal accuracy," Journal of Econometrics, Elsevier, vol. 186(1), pages 160-177.
    5. Kirby, Chris, 1997. "Measuring the Predictable Variation in Stock and Bond Returns," The Review of Financial Studies, Society for Financial Studies, vol. 10(3), pages 579-630.
    6. Hodrick, Robert J, 1992. "Dividend Yields and Expected Stock Returns: Alternative Procedures for Inference and Measurement," The Review of Financial Studies, Society for Financial Studies, vol. 5(3), pages 357-386.
    7. Inoue, Atsushi & Kilian, Lutz, 2006. "On the selection of forecasting models," Journal of Econometrics, Elsevier, vol. 130(2), pages 273-306, February.
    8. White, Halbert, 1980. "A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity," Econometrica, Econometric Society, vol. 48(4), pages 817-838, May.
    9. Silvia Goncalves & Lutz Kilian, 2007. "Asymptotic and Bootstrap Inference for AR(∞) Processes with Conditional Heteroskedasticity," Econometric Reviews, Taylor & Francis Journals, vol. 26(6), pages 609-641.
    10. Rapach, David E. & Wohar, Mark E., 2006. "In-sample vs. out-of-sample tests of stock return predictability in the context of data mining," Journal of Empirical Finance, Elsevier, vol. 13(2), pages 231-247, March.
    11. Hansen, Bruce E, 1996. "Erratum: The Likelihood Ratio Test under Nonstandard Conditions: Testing the Markov Switching Model of GNP," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 11(2), pages 195-198, March-Apr.
    12. Newey, Whitney & West, Kenneth, 2014. "A simple, positive semi-definite, heteroscedasticity and autocorrelation consistent covariance matrix," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 33(1), pages 125-132.
    13. John Y. Campbell & Samuel B. Thompson, 2008. "Predicting Excess Stock Returns Out of Sample: Can Anything Beat the Historical Average?," The Review of Financial Studies, Society for Financial Studies, vol. 21(4), pages 1509-1531, July.
    14. de Jong, Robert M., 1997. "Central Limit Theorems for Dependent Heterogeneous Random Variables," Econometric Theory, Cambridge University Press, vol. 13(3), pages 353-367, June.
    15. Halbert White, 2000. "A Reality Check for Data Snooping," Econometrica, Econometric Society, vol. 68(5), pages 1097-1126, September.
    16. Lutz Kilian & Atsushi Inoue, 2004. "Bagging Time Series Models," Econometric Society 2004 North American Summer Meetings 110, Econometric Society.
    17. Kilian, Lutz, 1999. "Exchange Rates and Monetary Fundamentals: What Do We Learn from Long-Horizon Regressions?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 14(5), pages 491-510, Sept.-Oct.
    18. Clark, Todd E. & McCracken, Michael W., 2005. "The power of tests of predictive ability in the presence of structural breaks," Journal of Econometrics, Elsevier, vol. 124(1), pages 1-31, January.
    19. Nelson, Charles R & Kim, Myung J, 1993. "Predictable Stock Returns: The Role of Small Sample Bias," Journal of Finance, American Finance Association, vol. 48(2), pages 641-661, June.
    20. Pesaran, M Hashem & Timmermann, Allan, 1995. "Predictability of Stock Returns: Robustness and Economic Significance," Journal of Finance, American Finance Association, vol. 50(4), pages 1201-1228, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Burcu Erik & Marco J. Lombardi & Dubravko Mihaljek & Hyun Song Shin, 2020. "The Dollar, Bank Leverage, and Real Economic Activity: An Evolving Relationship," AEA Papers and Proceedings, American Economic Association, vol. 110, pages 529-534, May.
    2. Hillebrand, Eric & Lukas, Manuel & Wei, Wei, 2021. "Bagging weak predictors," International Journal of Forecasting, Elsevier, vol. 37(1), pages 237-254.
    3. Calhoun, Gray, 2014. "Out-Of-Sample Comparisons of Overfit Models," Staff General Research Papers Archive 32462, Iowa State University, Department of Economics.
    4. Chauvet, Marcelle & Senyuz, Zeynep & Yoldas, Emre, 2015. "What does financial volatility tell us about macroeconomic fluctuations?," Journal of Economic Dynamics and Control, Elsevier, vol. 52(C), pages 340-360.
    5. Tom Boot & Andreas Pick, 2017. "A near optimal test for structural breaks when forecasting under square error loss," Tinbergen Institute Discussion Papers 17-039/III, Tinbergen Institute.
    6. Hsiu-Hsin Ko, 2016. "Exchange Rate Predictability in Finite Samples," The Japanese Economic Review, Springer, vol. 67(3), pages 361-378, September.
    7. Hossein Hassani & Emmanuel Sirimal Silva, 2015. "A Kolmogorov-Smirnov Based Test for Comparing the Predictive Accuracy of Two Sets of Forecasts," Econometrics, MDPI, vol. 3(3), pages 1-20, August.
    8. Juan Antolin-Diaz & Thomas Drechsel & Ivan Petrella, 2017. "Tracking the Slowdown in Long-Run GDP Growth," The Review of Economics and Statistics, MIT Press, vol. 99(2), pages 343-356, May.
    9. Lutz Kilian & Robert J. Vigfusson, 2013. "Do Oil Prices Help Forecast U.S. Real GDP? The Role of Nonlinearities and Asymmetries," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 31(1), pages 78-93, January.
    10. Petrella, Ivan & Drechsel, Thomas & Antolin-Diaz, Juan, 2014. "Following the Trend: Tracking GDP when Long-Run Growth is Uncertain," CEPR Discussion Papers 10272, C.E.P.R. Discussion Papers.
    11. Hsiu-Hsin Ko, 2016. "Exchange Rate Predictability in Finite Samples," The Japanese Economic Review, Japanese Economic Association, vol. 67(3), pages 361-378, September.
    12. Pincheira-Brown, Pablo & Selaive, Jorge & Nolazco, Jose Luis, 2019. "Forecasting inflation in Latin America with core measures," International Journal of Forecasting, Elsevier, vol. 35(3), pages 1060-1071.
    13. Su, Hao & Ying, Chengwei & Zhu, Xiaoneng, 2022. "Disaster risk matters in the bond market," Finance Research Letters, Elsevier, vol. 47(PA).
    14. Boot, Tom & Pick, Andreas, 2020. "Does modeling a structural break improve forecast accuracy?," Journal of Econometrics, Elsevier, vol. 215(1), pages 35-59.
    15. Mohan Subbiah & Frank J Fabozzi, 2016. "Equity style allocation: A nonparametric approach," Journal of Asset Management, Palgrave Macmillan, vol. 17(3), pages 141-164, May.
    16. Burcu Erik & Marco Jacopo Lombardi & Dubravko Mihaljek & Hyun Song Shin, 2019. "Financial conditions and purchasing managers' indices: exploring the links," BIS Quarterly Review, Bank for International Settlements, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Clark, Todd E. & McCracken, Michael W., 2015. "Nested forecast model comparisons: A new approach to testing equal accuracy," Journal of Econometrics, Elsevier, vol. 186(1), pages 160-177.
    2. Chen, Long, 2009. "On the reversal of return and dividend growth predictability: A tale of two periods," Journal of Financial Economics, Elsevier, vol. 92(1), pages 128-151, April.
    3. Clark, Todd & McCracken, Michael, 2013. "Advances in Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1107-1201, Elsevier.
    4. Maio, Paulo & Santa-Clara, Pedro, 2012. "Multifactor models and their consistency with the ICAPM," Journal of Financial Economics, Elsevier, vol. 106(3), pages 586-613.
    5. Rapach, David & Zhou, Guofu, 2013. "Forecasting Stock Returns," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 328-383, Elsevier.
    6. Boudoukh, Jacob & Israel, Ronen & Richardson, Matthew, 2022. "Biases in long-horizon predictive regressions," Journal of Financial Economics, Elsevier, vol. 145(3), pages 937-969.
    7. Atanasov, Victoria, 2018. "World output gap and global stock returns," Journal of Empirical Finance, Elsevier, vol. 48(C), pages 181-197.
    8. Favero, Carlo A. & Gozluklu, Arie E. & Tamoni, Andrea, 2011. "Demographic Trends, the Dividend-Price Ratio, and the Predictability of Long-Run Stock Market Returns," Journal of Financial and Quantitative Analysis, Cambridge University Press, vol. 46(5), pages 1493-1520, October.
    9. Rapach, David E. & Wohar, Mark E., 2006. "In-sample vs. out-of-sample tests of stock return predictability in the context of data mining," Journal of Empirical Finance, Elsevier, vol. 13(2), pages 231-247, March.
    10. Hai Lin & Daniel Quill & Henk Berkman, 2016. "Information diffusion and the predictability of New Zealand stock market returns," Accounting and Finance, Accounting and Finance Association of Australia and New Zealand, vol. 56(3), pages 749-785, September.
    11. Jurdi, Doureige J., 2022. "Predicting the Australian equity risk premium," Pacific-Basin Finance Journal, Elsevier, vol. 71(C).
    12. Todd E. Clark & Michael W. McCracken, 2010. "Reality checks and nested forecast model comparisons," Working Papers 2010-032, Federal Reserve Bank of St. Louis.
    13. Neil Kellard & John Nankervis & Fotis Papadimitriou, 2007. "Predicting the UK Equity Premium with Dividend Ratios: An Out-Of-Sample Recursive Residuals Graphical Approach," Money Macro and Finance (MMF) Research Group Conference 2006 129, Money Macro and Finance Research Group.
    14. Rapach, David E. & Wohar, Mark E. & Rangvid, Jesper, 2005. "Macro variables and international stock return predictability," International Journal of Forecasting, Elsevier, vol. 21(1), pages 137-166.
    15. Stephan Jank, 2015. "Changes in the Composition of Publicly Traded Firms: Implications for the Dividend-Price Ratio and Return Predictability," Management Science, INFORMS, vol. 61(6), pages 1362-1377, June.
    16. Galvão, Ana Beatriz, 2013. "Changes in predictive ability with mixed frequency data," International Journal of Forecasting, Elsevier, vol. 29(3), pages 395-410.
    17. Rossi, Barbara, 2013. "Advances in Forecasting under Instability," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1203-1324, Elsevier.
    18. repec:grz:wpaper:2012-02 is not listed on IDEAS
    19. Dladla, Pholile & Malikane, Christopher, 2019. "Stock return predictability: Evidence from a structural model," International Review of Economics & Finance, Elsevier, vol. 59(C), pages 412-424.
    20. Ivo Welch & Amit Goyal, 2008. "A Comprehensive Look at The Empirical Performance of Equity Premium Prediction," The Review of Financial Studies, Society for Financial Studies, vol. 21(4), pages 1455-1508, July.
    21. Dichtl, Hubert & Drobetz, Wolfgang & Neuhierl, Andreas & Wendt, Viktoria-Sophie, 2021. "Data snooping in equity premium prediction," International Journal of Forecasting, Elsevier, vol. 37(1), pages 72-94.

    More about this item

    Keywords

    Economic forecasting;

    JEL classification:

    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods
    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection
    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:fip:fedlwp:2009-051. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Anna Oates (email available below). General contact details of provider: https://edirc.repec.org/data/frbslus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.