IDEAS home Printed from https://ideas.repec.org/a/eee/ecmode/v140y2024ics0264999324002050.html
   My bibliography  Save this article

Multibenchmark reality checks

Author

Listed:
  • Arbués, Ignacio
  • Matilla-García, Mariano

Abstract

Empirical economic modelers often have to choose between two classes of models, with each class containing multiple models. In many cases, this decision is based on the predictive ability of the considered models. This entails that multiple testing and/or p-hacking pose known risks. This study presents a new statistical approach for comparing all model in a single test, serving as a multi-benchmark reality check test. The behavior of the test is studied asymptotically and in small finite samples. We show how the new approach works by analyzing whether one family of linear bivariate models outperforms a univariate family in predicting commodity prices. This paper raises new questions for future research. From an empirical viewpoint, we present several open questions in economic modeling that can be tested with multi-benchmark tests. Meanwhile, from a theoretical viewpoint, further studies can investigate whether a more general method for approximating or simulating the test distribution can be developed.

Suggested Citation

  • Arbués, Ignacio & Matilla-García, Mariano, 2024. "Multibenchmark reality checks," Economic Modelling, Elsevier, vol. 140(C).
  • Handle: RePEc:eee:ecmode:v:140:y:2024:i:c:s0264999324002050
    DOI: 10.1016/j.econmod.2024.106848
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0264999324002050
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.econmod.2024.106848?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Raffaella Giacomini & Halbert White, 2006. "Tests of Conditional Predictive Ability," Econometrica, Econometric Society, vol. 74(6), pages 1545-1578, November.
    2. Rossi, Barbara, 2005. "Optimal Tests For Nested Model Selection With Underlying Parameter Instability," Econometric Theory, Cambridge University Press, vol. 21(5), pages 962-990, October.
    3. Charles Engel & Kenneth D. West, 2005. "Exchange Rates and Fundamentals," Journal of Political Economy, University of Chicago Press, vol. 113(3), pages 485-517, June.
    4. Davidson, Russell & Flachaire, Emmanuel, 2008. "The wild bootstrap, tamed at last," Journal of Econometrics, Elsevier, vol. 146(1), pages 162-169, September.
    5. Richard Meese & Kenneth Rogoff, 1983. "The Out-of-Sample Failure of Empirical Exchange Rate Models: Sampling Error or Misspecification?," NBER Chapters, in: Exchange Rates and International Macroeconomics, pages 67-112, National Bureau of Economic Research, Inc.
    6. Clark, Todd E. & McCracken, Michael W., 2001. "Tests of equal forecast accuracy and encompassing for nested models," Journal of Econometrics, Elsevier, vol. 105(1), pages 85-110, November.
    7. McCracken, Michael W., 2004. "Parameter estimation and tests of equal forecast accuracy between non-nested models," International Journal of Forecasting, Elsevier, vol. 20(3), pages 503-514.
    8. West, Kenneth D, 1996. "Asymptotic Inference about Predictive Ability," Econometrica, Econometric Society, vol. 64(5), pages 1067-1084, September.
    9. Hansen, Peter Reinhard, 2005. "A Test for Superior Predictive Ability," Journal of Business & Economic Statistics, American Statistical Association, vol. 23, pages 365-380, October.
    10. Yu-Chin Chen & Kenneth S. Rogoff & Barbara Rossi, 2010. "Can Exchange Rates Forecast Commodity Prices?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 125(3), pages 1145-1194.
    11. Todd Clark & Michael McCracken, 2012. "Reality Checks and Comparisons of Nested Predictive Models," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 30(1), pages 53-66.
    12. Todd Clark & Michael McCracken, 2005. "Evaluating Direct Multistep Forecasts," Econometric Reviews, Taylor & Francis Journals, vol. 24(4), pages 369-404.
    13. Vuong, Quang H, 1989. "Likelihood Ratio Tests for Model Selection and Non-nested Hypotheses," Econometrica, Econometric Society, vol. 57(2), pages 307-333, March.
    14. Hansen, Bruce E., 1992. "Convergence to Stochastic Integrals for Dependent Heterogeneous Processes," Econometric Theory, Cambridge University Press, vol. 8(4), pages 489-500, December.
    15. Abel Brodeur & Nikolai Cook & Anthony Heyes, 2020. "Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics," American Economic Review, American Economic Association, vol. 110(11), pages 3634-3660, November.
    16. Swanson, Norman R. & White, Halbert, 1997. "Forecasting economic time series using flexible versus fixed specification and linear versus nonlinear econometric models," International Journal of Forecasting, Elsevier, vol. 13(4), pages 439-461, December.
    17. Ding, Shusheng & Zhang, Yongmin, 2020. "Cross market predictions for commodity prices," Economic Modelling, Elsevier, vol. 91(C), pages 455-462.
    18. Cheung, Yin-Wong & Chinn, Menzie D. & Pascual, Antonio Garcia, 2005. "Empirical exchange rate models of the nineties: Are any fit to survive?," Journal of International Money and Finance, Elsevier, vol. 24(7), pages 1150-1175, November.
    19. Kirstin Hubrich & Kenneth D. West, 2010. "Forecast evaluation of small nested model sets," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(4), pages 574-594.
    20. Todd E. Clark & Michael W. Mccracken, 2014. "Tests Of Equal Forecast Accuracy For Overlapping Models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 29(3), pages 415-430, April.
    21. Atsushi Inoue & Lutz Kilian, 2005. "In-Sample or Out-of-Sample Tests of Predictability: Which One Should We Use?," Econometric Reviews, Taylor & Francis Journals, vol. 23(4), pages 371-402.
    22. Meese, Richard A. & Rogoff, Kenneth, 1983. "Empirical exchange rate models of the seventies : Do they fit out of sample?," Journal of International Economics, Elsevier, vol. 14(1-2), pages 3-24, February.
    23. Matilla-García, Mariano & Marín, Manuel Ruiz & Dore, Mohammed I., 2014. "A permutation entropy based test for causality: The volume–stock price relation," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 398(C), pages 280-288.
    24. Joseph, Anto & Sisodia, Garima & Tiwari, Aviral Kumar, 2014. "A frequency domain causality investigation between futures and spot prices of Indian commodity markets," Economic Modelling, Elsevier, vol. 40(C), pages 250-258.
    25. Ca’ Zorzi, Michele & Rubaszek, Michał, 2023. "How many fundamentals should we include in the behavioral equilibrium exchange rate model?," Economic Modelling, Elsevier, vol. 118(C).
    26. Timmermann, Allan & Granger, Clive W. J., 2004. "Efficient market hypothesis and forecasting," International Journal of Forecasting, Elsevier, vol. 20(1), pages 15-27.
    27. Kagraoka, Yusho, 2016. "Common dynamic factors in driving commodity prices: Implications of a generalized dynamic factor model," Economic Modelling, Elsevier, vol. 52(PB), pages 609-617.
    28. Harvey, A C & Todd, P H J, 1983. "Forecasting Economic Time Series with Structural and Box-Jenkins Models: A Case Study," Journal of Business & Economic Statistics, American Statistical Association, vol. 1(4), pages 299-307, October.
    29. Stock, James H & Watson, Mark W, 2002. "Macroeconomic Forecasting Using Diffusion Indexes," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(2), pages 147-162, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Clark, Todd & McCracken, Michael, 2013. "Advances in Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1107-1201, Elsevier.
    2. Todd E. Clark & Michael W. McCracken, 2010. "Reality checks and nested forecast model comparisons," Working Papers 2010-032, Federal Reserve Bank of St. Louis.
    3. Barbara Rossi & Atsushi Inoue, 2012. "Out-of-Sample Forecast Tests Robust to the Choice of Window Size," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 30(3), pages 432-453, April.
    4. Granziera, Eleonora & Hubrich, Kirstin & Moon, Hyungsik Roger, 2014. "A predictability test for a small number of nested models," Journal of Econometrics, Elsevier, vol. 182(1), pages 174-185.
    5. Ahmed, Shamim & Liu, Xiaoquan & Valente, Giorgio, 2016. "Can currency-based risk factors help forecast exchange rates?," International Journal of Forecasting, Elsevier, vol. 32(1), pages 75-97.
    6. Rossi, Barbara, 2013. "Advances in Forecasting under Instability," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1203-1324, Elsevier.
    7. Molodtsova, Tanya & Papell, David H., 2009. "Out-of-sample exchange rate predictability with Taylor rule fundamentals," Journal of International Economics, Elsevier, vol. 77(2), pages 167-180, April.
    8. Todd E. Clark & Michael W. McCracken, 2010. "Testing for unconditional predictive ability," Working Papers 2010-031, Federal Reserve Bank of St. Louis.
    9. Corradi, Valentina & Fosten, Jack & Gutknecht, Daniel, 2024. "Predictive ability tests with possibly overlapping models," Journal of Econometrics, Elsevier, vol. 241(1).
    10. Kenneth S. Rogoff & Vania Stavrakeva, 2008. "The Continuing Puzzle of Short Horizon Exchange Rate Forecasting," NBER Working Papers 14071, National Bureau of Economic Research, Inc.
    11. West, Kenneth D., 2006. "Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 3, pages 99-134, Elsevier.
    12. Todd E. Clark & Michael W. Mccracken, 2014. "Tests Of Equal Forecast Accuracy For Overlapping Models," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 29(3), pages 415-430, April.
    13. McCracken, Michael W., 2007. "Asymptotics for out of sample tests of Granger causality," Journal of Econometrics, Elsevier, vol. 140(2), pages 719-752, October.
    14. Pincheira-Brown, Pablo & Neumann, Federico, 2020. "Can we beat the Random Walk? The case of survey-based exchange rate forecasts in Chile," Finance Research Letters, Elsevier, vol. 37(C).
    15. Raffaella Giacomini & Barbara Rossi, 2013. "Forecasting in macroeconomics," Chapters, in: Nigar Hashimzade & Michael A. Thornton (ed.), Handbook of Research Methods and Applications in Empirical Macroeconomics, chapter 17, pages 381-408, Edward Elgar Publishing.
    16. Busetti, Fabio & Marcucci, Juri, 2013. "Comparing forecast accuracy: A Monte Carlo investigation," International Journal of Forecasting, Elsevier, vol. 29(1), pages 13-27.
    17. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    18. Nicolás Magner & Nicolás Hardy, 2022. "Cryptocurrency Forecasting: More Evidence of the Meese-Rogoff Puzzle," Mathematics, MDPI, vol. 10(13), pages 1-27, July.
    19. Rapach, David & Zhou, Guofu, 2013. "Forecasting Stock Returns," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 328-383, Elsevier.
    20. Feng, Wenjun & Zhang, Zhengjun, 2023. "Currency exchange rate predictability: The new power of Bitcoin prices," Journal of International Money and Finance, Elsevier, vol. 132(C).

    More about this item

    Keywords

    Granger causality; Out-of-sample forecast; Model selection; Bootstrap;
    All these keywords.

    JEL classification:

    • F31 - International Economics - - International Finance - - - Foreign Exchange
    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ecmode:v:140:y:2024:i:c:s0264999324002050. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/inca/30411 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.