IDEAS home Printed from https://ideas.repec.org/p/yor/yorken/15-15.html
   My bibliography  Save this paper

Comparing predictive accuracy in small samples

Author

Listed:
  • Laura Coroneo
  • Fabrizio Iacone

Abstract

We consider fixed-b and fixed-m asymptotics for the Diebold and Mariano (1995) test of predictive accuracy. We show that this approach allows to obtain predictive accuracy tests that are correctly sized even in small samples. We apply the alternative asympotics for the Diebold and Mariano (1995) test to evaluate the predictive accuracy of the Survey of Professional Forecasters (SPF) against a simple random walk. Our results show that the predictive ability of the SPF was partially spurious, especially in the last decade.

Suggested Citation

  • Laura Coroneo & Fabrizio Iacone, 2015. "Comparing predictive accuracy in small samples," Discussion Papers 15/15, Department of Economics, University of York.
    • Handle: RePEc:yor:yorken:15/15
    as

    Download full text from publisher

    File URL: https://www.york.ac.uk/media/economics/documents/discussionpapers/2015/1515.pdf
    File Function: Main text
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Yixiao Sun, 2013. "A heteroskedasticity and autocorrelation robust F test using an orthonormal series variance estimator," Econometrics Journal, Royal Economic Society, vol. 16(1), pages 1-26, February.
    2. Clark, Todd & McCracken, Michael, 2013. "Advances in Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1107-1201, Elsevier.
    3. Kiefer, Nicholas M. & Vogelsang, Timothy J., 2005. "A New Asymptotic Theory For Heteroskedasticity-Autocorrelation Robust Tests," Econometric Theory, Cambridge University Press, vol. 21(6), pages 1130-1164, December.
    4. Whitney K. Newey & Kenneth D. West, 1994. "Automatic Lag Selection in Covariance Matrix Estimation," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 61(4), pages 631-653.
    5. Javier Hualde & Fabrizio Iacone, 2015. "Autocorrelation robust inference using the Daniell kernel with fixed bandwidth," Discussion Papers 15/14, Department of Economics, University of York.
    6. Li, Jia & Patton, Andrew J., 2018. "Asymptotic inference about predictive accuracy using high frequency data," Journal of Econometrics, Elsevier, vol. 203(2), pages 223-240.
    7. Clark, Todd E. & McCracken, Michael W., 2001. "Tests of equal forecast accuracy and encompassing for nested models," Journal of Econometrics, Elsevier, vol. 105(1), pages 85-110, November.
    8. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    9. Francis X. Diebold, 2015. "Comparing Predictive Accuracy, Twenty Years Later: A Personal Perspective on the Use and Abuse of Diebold-Mariano Tests," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 33(1), pages 1-1, January.
    10. Kokoszka, Piotr & Mikosch, Thomas, 2000. "The periodogram at the Fourier frequencies," Stochastic Processes and their Applications, Elsevier, vol. 86(1), pages 49-79, March.
    11. Raffaella Giacomini & Barbara Rossi, 2010. "Forecast comparisons in unstable environments," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(4), pages 595-620.
    12. Harvey, David & Leybourne, Stephen & Newbold, Paul, 1997. "Testing the equality of prediction mean squared errors," International Journal of Forecasting, Elsevier, vol. 13(2), pages 281-291, June.
    13. G. Elliott & C. Granger & A. Timmermann (ed.), 2013. "Handbook of Economic Forecasting," Handbook of Economic Forecasting, Elsevier, edition 1, volume 2, number 2.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Hideyuki Takamizawa, 2018. "A term structure model of interest rates with quadratic volatility," Quantitative Finance, Taylor & Francis Journals, vol. 18(7), pages 1173-1198, July.
    2. Döhrn, Roland, 2019. "Comparing forecast accuracy in small samples," Ruhr Economic Papers 833, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
    3. Raïsa Basselier & David Antonio Liedo & Geert Langenus, 2018. "Nowcasting Real Economic Activity in the Euro Area: Assessing the Impact of Qualitative Surveys," Journal of Business Cycle Research, Springer;Centre for International Research on Economic Tendency Surveys (CIRET), vol. 14(1), pages 1-46, April.
    4. Harvey, David I. & Leybourne, Stephen J. & Whitehouse, Emily J., 2017. "Forecast evaluation tests and negative long-run variance estimates in small samples," International Journal of Forecasting, Elsevier, vol. 33(4), pages 833-847.
    5. Michael W. McCracken & Serena Ng, 2021. "FRED-QD: A Quarterly Database for Macroeconomic Research," Review, Federal Reserve Bank of St. Louis, vol. 103(1), pages 1-44, January.
    6. Rubaszek Michal & Karolak Zuzanna & Kwas Marek & Uddin Gazi Salah, 2020. "The role of the threshold effect for the dynamics of futures and spot prices of energy commodities," Studies in Nonlinear Dynamics & Econometrics, De Gruyter, vol. 24(5), pages 1-20, December.
    7. Jack Fosten & Daniel Gutknecht, 2021. "Horizon confidence sets," Empirical Economics, Springer, vol. 61(2), pages 667-692, August.
    8. Alberto Caruso & Laura Coroneo, 2019. "Predicting interest rates in real-time," Discussion Papers 19/18, Department of Economics, University of York.
    9. Rubaszek, Michał & Karolak, Zuzanna & Kwas, Marek, 2020. "Mean-reversion, non-linearities and the dynamics of industrial metal prices. A forecasting perspective," Resources Policy, Elsevier, vol. 65(C).
    10. Norman R. Swanson & Weiqi Xiong & Xiye Yang, 2020. "Predicting interest rates using shrinkage methods, real‐time diffusion indexes, and model combinations," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 35(5), pages 587-613, August.
    11. Marek Kwas & Michał Rubaszek, 2021. "Forecasting Commodity Prices: Looking for a Benchmark," Forecasting, MDPI, vol. 3(2), pages 1-13, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Laura Coroneo & Fabrizio Iacone, 2020. "Comparing predictive accuracy in small samples using fixed‐smoothing asymptotics," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 35(4), pages 391-409, June.
    2. Håvard Hungnes, 2020. "Equal predictability test for multi-step-ahead system forecasts invariant to linear transformations," Discussion Papers 931, Statistics Norway, Research Department.
    3. Sekkel, Rodrigo M., 2015. "Balance sheets of financial intermediaries: Do they forecast economic activity?," International Journal of Forecasting, Elsevier, vol. 31(2), pages 263-275.
    4. Michael W. McCracken, 2019. "Tests of Conditional Predictive Ability: Some Simulation Evidence," Working Papers 2019-11, Federal Reserve Bank of St. Louis.
    5. Jamali, Ibrahim & Yamani, Ehab, 2019. "Out-of-sample exchange rate predictability in emerging markets: Fundamentals versus technical analysis," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 61(C), pages 241-263.
    6. Magdalena Grothe & Aidan Meyler, 2018. "Inflation Forecasts: Are Market-Based and Survey-Based Measures Informative?," International Journal of Financial Research, International Journal of Financial Research, Sciedu Press, vol. 9(1), pages 171-188, January.
    7. Pablo Pincheira Brown & Nicolás Hardy, 2024. "Correlation‐based tests of predictability," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 43(6), pages 1835-1858, September.
    8. Tuhkuri, Joonas, 2016. "Forecasting Unemployment with Google Searches," ETLA Working Papers 35, The Research Institute of the Finnish Economy.
    9. Moramarco, Graziano, 2024. "Financial-cycle ratios and medium-term predictions of GDP: Evidence from the United States," International Journal of Forecasting, Elsevier, vol. 40(2), pages 777-795.
    10. Kontogeorgos, Georgios & Lambrias, Kyriacos, 2019. "An analysis of the Eurosystem/ECB projections," Working Paper Series 2291, European Central Bank.
    11. Fabian Krüger & Todd E. Clark & Francesco Ravazzolo, 2017. "Using Entropic Tilting to Combine BVAR Forecasts With External Nowcasts," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 35(3), pages 470-485, July.
    12. Pablo Pincheira & Nicolás Hardy & Felipe Muñoz, 2021. "“Go Wild for a While!”: A New Test for Forecast Evaluation in Nested Models," Mathematics, MDPI, vol. 9(18), pages 1-28, September.
    13. Hinterlang, Natascha, 2019. "Predicting Monetary Policy Using Artificial Neural Networks," VfS Annual Conference 2019 (Leipzig): 30 Years after the Fall of the Berlin Wall - Democracy and Market Economy 203503, Verein für Socialpolitik / German Economic Association.
    14. Hinterlang, Natascha, 2020. "Predicting monetary policy using artificial neural networks," Discussion Papers 44/2020, Deutsche Bundesbank.
    15. Tunaru, Diana, 2017. "Gaussian estimation and forecasting of the U.K. yield curve with multi-factor continuous-time models," International Review of Financial Analysis, Elsevier, vol. 52(C), pages 119-129.
    16. Michael W. McCracken, 2020. "Tests of Conditional Predictive Ability: Existence, Size, and Power," Working Papers 2020-050, Federal Reserve Bank of St. Louis.
    17. Barbara Rossi, 2013. "Exchange Rate Predictability," Journal of Economic Literature, American Economic Association, vol. 51(4), pages 1063-1119, December.
    18. Matei Demetrescu & Christoph Hanck & Robinson Kruse‐Becher, 2022. "Robust inference under time‐varying volatility: A real‐time evaluation of professional forecasters," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(5), pages 1010-1030, August.
    19. Ahmed, Shamim & Liu, Xiaoquan & Valente, Giorgio, 2016. "Can currency-based risk factors help forecast exchange rates?," International Journal of Forecasting, Elsevier, vol. 32(1), pages 75-97.
    20. Harvey, David I. & Leybourne, Stephen J. & Whitehouse, Emily J., 2017. "Forecast evaluation tests and negative long-run variance estimates in small samples," International Journal of Forecasting, Elsevier, vol. 33(4), pages 833-847.

    More about this item

    Keywords

    Diebold and Mariano test; long run variance estimation; fixed-b and fixed-m asymptotic theory; SPF.;
    All these keywords.

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C32 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models; Diffusion Processes; State Space Models
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods
    • E17 - Macroeconomics and Monetary Economics - - General Aggregative Models - - - Forecasting and Simulation: Models and Applications

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:yor:yorken:15/15. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Paul Hodgson (email available below). General contact details of provider: https://edirc.repec.org/data/deyoruk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.