IDEAS home Printed from https://ideas.repec.org/p/imf/imfwpa/2018-260.html
   My bibliography  Save this paper

Overfitting in Judgment-based Economic Forecasts: The Case of IMF Growth Projections

Author

Listed:
  • Klaus-Peter Hellwig

Abstract

I regress real GDP growth rates on the IMF’s growth forecasts and find that IMF forecasts behave similarly to those generated by overfitted models, placing too much weight on observable predictors and underestimating the forces of mean reversion. I identify several such variables that explain forecasts well but are not predictors of actual growth. I show that, at long horizons, IMF forecasts are little better than a forecasting rule that uses no information other than the historical global sample average growth rate (i.e., a constant). Given the large noise component in forecasts, particularly at longer horizons, the paper calls into question the usefulness of judgment-based medium and long-run forecasts for policy analysis, including for debt sustainability assessments, and points to statistical methods to improve forecast accuracy by taking into account the risk of overfitting.

Suggested Citation

  • Klaus-Peter Hellwig, 2018. "Overfitting in Judgment-based Economic Forecasts: The Case of IMF Growth Projections," IMF Working Papers 2018/260, International Monetary Fund.
  • Handle: RePEc:imf:imfwpa:2018/260
    as

    Download full text from publisher

    File URL: http://www.imf.org/external/pubs/cat/longres.aspx?sk=46390
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Mr. Andrew J Tiffin, 2016. "Seeing in the Dark: A Machine-Learning Approach to Nowcasting in Lebanon," IMF Working Papers 2016/056, International Monetary Fund.
    2. Blume, Lawrence E. & Easley, David, 1982. "Learning to be rational," Journal of Economic Theory, Elsevier, vol. 26(2), pages 340-351, April.
    3. Pedro Bordalo & Nicola Gennaioli & Yueran Ma & Andrei Shleifer, 2020. "Overreaction in Macroeconomic Expectations," American Economic Review, American Economic Association, vol. 110(9), pages 2748-2782, September.
    4. Valerie Cerra & Sweta Chaman Saxena, 2008. "Growth Dynamics: The Myth of Economic Recovery," American Economic Review, American Economic Association, vol. 98(1), pages 439-457, March.
    5. Dani Rodrik, 2008. "The Real Exchange Rate and Economic Growth," Brookings Papers on Economic Activity, Economic Studies Program, The Brookings Institution, vol. 39(2 (Fall)), pages 365-439.
    6. Jin-Kyu Jung & Manasa Patnam & Anna Ter-Martirosyan, 2018. "An Algorithmic Crystal Ball: Forecasts-based on Machine Learning," IMF Working Papers 2018/230, International Monetary Fund.
    7. Robert E. Lucas, 2011. "What Economists Do," Journal of Applied Economics, Taylor & Francis Journals, vol. 14(1), pages 1-4, May.
    8. Paul Beaudry & Tim Willems, 2022. "On the Macroeconomic Consequences of Over-Optimism," American Economic Journal: Macroeconomics, American Economic Association, vol. 14(1), pages 38-59, January.
    9. Olivier J. Blanchard & Daniel Leigh, 2013. "Growth Forecast Errors and Fiscal Multipliers," American Economic Review, American Economic Association, vol. 103(3), pages 117-120, May.
    10. Mark Aguiar & Gita Gopinath, 2007. "Emerging Market Business Cycles: The Cycle Is the Trend," Journal of Political Economy, University of Chicago Press, vol. 115(1), pages 69-102.
    11. Allan Timmermann, 2007. "An Evaluation of the World Economic Outlook Forecasts," IMF Staff Papers, Palgrave Macmillan, vol. 54(1), pages 1-33, May.
    12. Eicher, Theo S. & Kuenzel, David J. & Papageorgiou, Chris & Christofides, Charis, 2019. "Forecasts in times of crises," International Journal of Forecasting, Elsevier, vol. 35(3), pages 1143-1159.
    13. Giang Ho & Paolo Mauro, 2016. "Growth—Now and Forever?," IMF Economic Review, Palgrave Macmillan;International Monetary Fund, vol. 64(3), pages 526-547, August.
    14. Sylwia Nowak & Pratiti Chatterjee, 2016. "Forecast Errors and Uncertainty Shocks," IMF Working Papers 2016/228, International Monetary Fund.
    15. Frankel, Jeffrey A. & Schreger, Jesse, 2016. "Bias in Official Fiscal Forecasts: Can Private Forecasts Help?," Working Paper Series 16-021, Harvard University, John F. Kennedy School of Government.
    16. Jacob A. Mincer & Victor Zarnowitz, 1969. "The Evaluation of Economic Forecasts," NBER Chapters, in: Economic Forecasts and Expectations: Analysis of Forecasting Behavior and Performance, pages 3-46, National Bureau of Economic Research, Inc.
    17. Gultekin Isiklar & Kajal Lahiri & Prakash Loungani, 2006. "How quickly do forecasters incorporate news? Evidence from cross‐country surveys," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 21(6), pages 703-725, September.
    18. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    19. Lant Pritchett & Lawrence H. Summers, 2013. "Asia-phoria meet regression to the mean," Proceedings, Federal Reserve Bank of San Francisco, issue Nov, pages 1-35.
    20. Jacob A. Mincer, 1969. "Economic Forecasts and Expectations: Analysis of Forecasting Behavior and Performance," NBER Books, National Bureau of Economic Research, Inc, number minc69-1.
    21. Jalles, João Tovar & Karibzhanov, Iskander & Loungani, Prakash, 2015. "Cross-country evidence on the quality of private sector fiscal forecasts," Journal of Macroeconomics, Elsevier, vol. 45(C), pages 186-201.
    22. Sendhil Mullainathan & Jann Spiess, 2017. "Machine Learning: An Applied Econometric Approach," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 87-106, Spring.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. María Paula Bonel & Daniel J. Aromí, 2021. "Assessing GDP forecasts from autoregressive models: the impact of model complexity and training dataset," Asociación Argentina de Economía Política: Working Papers 4440, Asociación Argentina de Economía Política.
    2. Philippe Goulet Coulombe, 2020. "To Bag is to Prune," Papers 2008.07063, arXiv.org, revised Sep 2024.
    3. Philippe Goulet Coulombe, 2021. "To Bag is to Prune," Working Papers 21-03, Chair in macroeconomics and forecasting, University of Quebec in Montreal's School of Management, revised Jun 2021.
    4. Karminsky, A. & Dyachkova, N., 2020. "Empirical study of the relationship between credit cycles and changes in credit ratings," Journal of the New Economic Association, New Economic Association, vol. 48(4), pages 138-160.
    5. Jorge Fornero & Andrés Gatty, 2020. "Back testing fan charts of activity and inflation: the Chilean case," Working Papers Central Bank of Chile 881, Central Bank of Chile.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Eicher, Theo S. & Rollinson, Yuan Gao, 2023. "The accuracy of IMF crises nowcasts," International Journal of Forecasting, Elsevier, vol. 39(1), pages 431-449.
    2. Yan Carrière-Swallow & José Marzluf, 2023. "Macrofinancial Causes of Optimism in Growth Forecasts," IMF Economic Review, Palgrave Macmillan;International Monetary Fund, vol. 71(2), pages 509-537, June.
    3. Tsuchiya, Yoichi, 2023. "Assessing the World Bank’s growth forecasts," Economic Analysis and Policy, Elsevier, vol. 77(C), pages 64-84.
    4. J. Daniel Aromí, 2018. "GDP growth forecasts and information flows: Is there evidence of overreactions?," International Finance, Wiley Blackwell, vol. 21(2), pages 122-139, June.
    5. Eicher, Theo S. & Kawai, Reina, 2023. "IMF trade forecasts for crisis countries: Bias, inefficiency, and their origins," International Journal of Forecasting, Elsevier, vol. 39(4), pages 1615-1639.
    6. Carabotta, Laura & Paluzie, Elisenda & Ramos, Raul, 2017. "Does fiscal responsibility matter? Evidence from public and private forecasters in Italy," International Journal of Forecasting, Elsevier, vol. 33(3), pages 694-706.
    7. Yoichi Tsuchiya, 2021. "Thirty‐year assessment of Asian Development Bank's forecasts," Asian-Pacific Economic Literature, The Crawford School, The Australian National University, vol. 35(2), pages 18-40, November.
    8. Morikawa, Masayuki, 2022. "Uncertainty in long-term macroeconomic forecasts: Ex post evaluation of forecasts by economics researchers," The Quarterly Review of Economics and Finance, Elsevier, vol. 85(C), pages 8-15.
    9. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    10. Emilian Dobrescu, 2014. "Attempting to Quantify the Accuracy of Complex Macroeconomic Forecasts," Journal for Economic Forecasting, Institute for Economic Forecasting, vol. 0(4), pages 5-21, December.
    11. Eicher, Theo S. & Kawai, Reina, 2024. "Systemic bias of IMF reserve and debt forecasts for program countries," International Journal of Forecasting, Elsevier, vol. 40(3), pages 985-1001.
    12. Aromí, J. Daniel, 2019. "Medium term growth forecasts: Experts vs. simple models," International Journal of Forecasting, Elsevier, vol. 35(3), pages 1085-1099.
    13. Giovannelli, Alessandro & Pericoli, Filippo Maria, 2020. "Are GDP forecasts optimal? Evidence on European countries," International Journal of Forecasting, Elsevier, vol. 36(3), pages 963-973.
    14. Ericsson, Neil R., 2016. "Eliciting GDP forecasts from the FOMC’s minutes around the financial crisis," International Journal of Forecasting, Elsevier, vol. 32(2), pages 571-583.
    15. Barbara Rossi, 2013. "Exchange Rate Predictability," Journal of Economic Literature, American Economic Association, vol. 51(4), pages 1063-1119, December.
    16. Leif Anders Thorsrud, 2016. "Nowcasting using news topics Big Data versus big bank," Working Papers No 6/2016, Centre for Applied Macro- and Petroleum economics (CAMP), BI Norwegian Business School.
    17. Knüppel, Malte & Schultefrankenfeld, Guido, 2019. "Assessing the uncertainty in central banks’ inflation outlooks," International Journal of Forecasting, Elsevier, vol. 35(4), pages 1748-1769.
    18. Rossi, Barbara & Sekhposyan, Tatevik, 2011. "Understanding models' forecasting performance," Journal of Econometrics, Elsevier, vol. 164(1), pages 158-172, September.
    19. Joan Paredes & Javier J. Pérez & Gabriel Perez Quiros, 2023. "Fiscal targets. A guide to forecasters?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(4), pages 472-492, June.
    20. Santos, Douglas G. & Candido, Osvaldo & Tófoli, Paula V., 2022. "Forecasting risk measures using intraday and overnight information," The North American Journal of Economics and Finance, Elsevier, vol. 60(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:imf:imfwpa:2018/260. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Akshay Modi (email available below). General contact details of provider: https://edirc.repec.org/data/imfffus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.