IDEAS home Printed from https://ideas.repec.org/p/zbw/rwirep/833.html
   My bibliography  Save this paper

Comparing forecast accuracy in small samples

Author

Listed:
  • Döhrn, Roland

Abstract

The Diebold-Mariano-Test has become a common tool to compare the accuracy of macroeconomic forecasts. Since these are typically model-free forecasts, distribution free tests might be a good alternative to the Diebold-Mariano-Test. This paper suggests a permutation test. Stochastic simulations show that permutation tests outperform the Diebold-Mariano-Test. Furthermore, a test statistic based on absolute errors seems to be more sensitive to differences in forecast accuracy than a statistic based on squared errors.

Suggested Citation

  • Döhrn, Roland, 2019. "Comparing forecast accuracy in small samples," Ruhr Economic Papers 833, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
    • Handle: RePEc:zbw:rwirep:833
    DOI: 10.4419/86788966
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/209589/1/168581185X.pdf
    Download Restriction: no
    File URL: https://libkey.io/10.4419/86788966?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Giampiero M. Gallo & Clive W.J. Granger & Yongil Jeon, 2002. "Copycats and Common Swings: The Impact of the Use of Forecasts in Information Sets," IMF Staff Papers, Palgrave Macmillan, vol. 49(1), pages 1-2.
    2. Harvey, David & Leybourne, Stephen & Newbold, Paul, 1997. "Testing the equality of prediction mean squared errors," International Journal of Forecasting, Elsevier, vol. 13(2), pages 281-291, June.
    3. Laura Coroneo & Fabrizio Iacone, 2015. "Comparing predictive accuracy in small samples," Discussion Papers 15/15, Department of Economics, University of York.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Harvey, David I. & Leybourne, Stephen J. & Whitehouse, Emily J., 2017. "Forecast evaluation tests and negative long-run variance estimates in small samples," International Journal of Forecasting, Elsevier, vol. 33(4), pages 833-847.
    2. Raïsa Basselier & David Antonio Liedo & Geert Langenus, 2018. "Nowcasting Real Economic Activity in the Euro Area: Assessing the Impact of Qualitative Surveys," Journal of Business Cycle Research, Springer;Centre for International Research on Economic Tendency Surveys (CIRET), vol. 14(1), pages 1-46, April.
    3. Kelly Burns & Imad Moosa, 2017. "Demystifying the Meese–Rogoff puzzle: structural breaks or measures of forecasting accuracy?," Applied Economics, Taylor & Francis Journals, vol. 49(48), pages 4897-4910, October.
    4. Chang, Chia-Lin & de Bruijn, Bert & Franses, Philip Hans & McAleer, Michael, 2013. "Analyzing fixed-event forecast revisions," International Journal of Forecasting, Elsevier, vol. 29(4), pages 622-627.
    5. Carlo Altavilla & Paul De Grauwe, 2010. "Forecasting and combining competing models of exchange rate determination," Applied Economics, Taylor & Francis Journals, vol. 42(27), pages 3455-3480.
    6. Alberto Fuertes & Simón Sosvilla-Rivero, 2019. "“Forecasting emerging market currencies: Are inflation expectations useful?”," IREA Working Papers 201918, University of Barcelona, Research Institute of Applied Economics, revised Oct 2019.
    7. Gkillas, Konstantinos & Gupta, Rangan & Pierdzioch, Christian, 2020. "Forecasting realized oil-price volatility: The role of financial stress and asymmetric loss," Journal of International Money and Finance, Elsevier, vol. 104(C).
    8. Xiaojie Xu, 2017. "The rolling causal structure between the Chinese stock index and futures," Financial Markets and Portfolio Management, Springer;Swiss Society for Financial Market Research, vol. 31(4), pages 491-509, November.
    9. Dal Bianco, Marcos & Camacho, Maximo & Perez Quiros, Gabriel, 2012. "Short-run forecasting of the euro-dollar exchange rate with economic fundamentals," Journal of International Money and Finance, Elsevier, vol. 31(2), pages 377-396.
    10. Daniel Buncic, 2012. "Understanding forecast failure of ESTAR models of real exchange rates," Empirical Economics, Springer, vol. 43(1), pages 399-426, August.
    11. Gary Koop & Dimitris Korobilis, 2019. "Forecasting with High‐Dimensional Panel VARs," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 81(5), pages 937-959, October.
    12. Scott Brave & R. Andrew Butters & Alejandro Justiniano, 2016. "Forecasting Economic Activity with Mixed Frequency Bayesian VARs," Working Paper Series WP-2016-5, Federal Reserve Bank of Chicago.
    13. Máximo Camacho & Rafael Doménech, 2012. "MICA-BBVA: a factor model of economic and financial indicators for short-term GDP forecasting," SERIEs: Journal of the Spanish Economic Association, Springer;Spanish Economic Association, vol. 3(4), pages 475-497, December.
    14. Keen Meng Choy & Hwee Kwan Chow, 2004. "Forecasting the Global Electronics Cycle with Leading Indicators: A VAR Approach," Econometric Society 2004 Australasian Meetings 223, Econometric Society.
    15. Ard H.J. den Reijer, 2005. "Forecasting Dutch GDP using Large Scale Factor Models," DNB Working Papers 028, Netherlands Central Bank, Research Department.
    16. Andrea Carriero & Todd E. Clark & Massimiliano Marcellino, 2015. "Realtime nowcasting with a Bayesian mixed frequency model with stochastic volatility," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 178(4), pages 837-862, October.
    17. Goodness C. Aye & Stephen M. Miller & Rangan Gupta & Mehmet Balcilar, 2016. "Forecasting US real private residential fixed investment using a large number of predictors," Empirical Economics, Springer, vol. 51(4), pages 1557-1580, December.
    18. Chuliá, Helena & Garrón, Ignacio & Uribe, Jorge M., 2024. "Daily growth at risk: Financial or real drivers? The answer is not always the same," International Journal of Forecasting, Elsevier, vol. 40(2), pages 762-776.
    19. Mehmet Balcilar & Rangan Gupta & Stephen M. Miller, 2015. "The out-of-sample forecasting performance of nonlinear models of regional housing prices in the US," Applied Economics, Taylor & Francis Journals, vol. 47(22), pages 2259-2277, May.
    20. Aastveit, Knut Are & Trovik, Tørres, 2014. "Estimating the output gap in real time: A factor model approach," The Quarterly Review of Economics and Finance, Elsevier, vol. 54(2), pages 180-193.

    More about this item

    Keywords

    macroeconomic forecast; forecast accuracy; Diebold-Mariano test; permutation test;
    All these keywords.

    JEL classification:

    • C14 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Semiparametric and Nonparametric Methods: General
    • C15 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Statistical Simulation Methods: General
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:rwirep:833. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/rwiesde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.