IDEAS home Printed from https://ideas.repec.org/p/gwc/wpaper/2010-002.html
   My bibliography  Save this paper

Perspectives on Evaluating Macroeconomic Forecasts

Author

Listed:
  • Ullrich Heilemann

    (Univeristy of Leipzig)

  • Herman Stekler

    (Department of Economics The George Washington University)

Abstract

Over the past 50 or so years, I have been concerned with the quality of economic forecasts and have written both about the procedures for evaluating these predictions and the results that were obtained from these evaluations. In this paper I provide some perspectives on the issues involved in judging the quality of these forecasts. These include the reasons for evaluating forecasts, the questions that have been asked in these evaluations, the statistical tools that have been used, and the generally accepted results. (I do also present some new material that has not yet been published.) I do this in two parts: first focusing on short-run GDP and inflation predictions and then turning to labor market forecasts.

Suggested Citation

  • Ullrich Heilemann & Herman Stekler, 2010. "Perspectives on Evaluating Macroeconomic Forecasts," Working Papers 2010-002, The George Washington University, Department of Economics, H. O. Stekler Research Program on Forecasting.
    • Handle: RePEc:gwc:wpaper:2010-002
    as

    Download full text from publisher

    File URL: https://www2.gwu.edu/~forcpgm/2010-002.pdf
    File Function: First version, 2010
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Jacob A. Mincer & Victor Zarnowitz, 1969. "The Evaluation of Economic Forecasts," NBER Chapters, in: Economic Forecasts and Expectations: Analysis of Forecasting Behavior and Performance, pages 3-46, National Bureau of Economic Research, Inc.
    2. Stekler, H. O., 1991. "Macroeconomic forecast evaluation techniques," International Journal of Forecasting, Elsevier, vol. 7(3), pages 375-384, November.
    3. Henriksson, Roy D & Merton, Robert C, 1981. "On Market Timing and Investment Performance. II. Statistical Procedures for Evaluating Forecasting Skills," The Journal of Business, University of Chicago Press, vol. 54(4), pages 513-533, October.
    4. Fildes, Robert & Stekler, Herman, 2002. "The state of macroeconomic forecasting," Journal of Macroeconomics, Elsevier, vol. 24(4), pages 435-468, December.
    5. Batchelor, Roy, 2007. "Bias in macroeconomic forecasts," International Journal of Forecasting, Elsevier, vol. 23(2), pages 189-203.
    6. Athanasios Orphanides, 2001. "Monetary Policy Rules Based on Real-Time Data," American Economic Review, American Economic Association, vol. 91(4), pages 964-985, September.
    7. Fred Joutz & Michael P. Clements & Herman O. Stekler, 2007. "An evaluation of the forecasts of the federal reserve: a pooled approach," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 22(1), pages 121-136.
    8. Golan, Amos & Perloff, Jeffrey M., 2002. "Superior Forecasts of the U.S. Unemployment Rate Using a Nonparametric Method," Department of Agricultural & Resource Economics, UC Berkeley, Working Paper Series qt2bw559zk, Department of Agricultural & Resource Economics, UC Berkeley.
    9. West, Kenneth D., 2006. "Forecast Evaluation," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 3, pages 99-134, Elsevier.
    10. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    11. Laura Brown & Saeed Moshiri, 2004. "Unemployment variation over the business cycles: a comparison of forecasting models," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 23(7), pages 497-511.
    12. Graham Elliott & Allan Timmermann, 2016. "Economic Forecasting," Economics Books, Princeton University Press, edition 1, number 10740.
    13. Baghestani, Hamid, 2008. "Federal Reserve versus private information: Who is the best unemployment rate predictor," Journal of Policy Modeling, Elsevier, vol. 30(1), pages 101-110.
    14. Massimiliano Marcellino, "undated". "Instability and non-linearity in the EMU," Working Papers 211, IGIER (Innocenzo Gasparini Institute for Economic Research), Bocconi University.
    15. Amos Golan & Jeffrey M. Perloff, 2004. "Superior Forecasts of the U.S. Unemployment Rate Using a Nonparametric Method," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 433-438, February.
    16. Gopal Naik & Raymond M. Leuthold, 1986. "A Note on Qualitative Forecast Evaluation," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 68(3), pages 721-726.
    17. Lawrence, Michael & Goodwin, Paul & O'Connor, Marcus & Onkal, Dilek, 2006. "Judgmental forecasting: A review of progress over the last 25 years," International Journal of Forecasting, Elsevier, vol. 22(3), pages 493-518.
    18. Jacob A. Mincer, 1969. "Economic Forecasts and Expectations: Analysis of Forecasting Behavior and Performance," NBER Books, National Bureau of Economic Research, Inc, number minc69-1.
    19. Allen, P. Geoffrey & Morzuch, Bernard J., 2006. "Twenty-five years of progress, problems, and conflicting evidence in econometric forecasting. What about the next 25 years?," International Journal of Forecasting, Elsevier, vol. 22(3), pages 475-492.
    20. Herman O. Stekler, 2008. "What Do We Know About G-7 Macro Forecasts?," Working Papers 2008-009, The George Washington University, Department of Economics, H. O. Stekler Research Program on Forecasting.
    21. Christian Pierdzioch & Jan-Christoph Rulke & Georg Stadtmann, 2011. "Do professional economists' forecasts reflect Okun's law? Some evidence for the G7 countries," Applied Economics, Taylor & Francis Journals, vol. 43(11), pages 1365-1373.
    22. Öller, Lars-Erik & Barot, Bharat, 1999. "Comparing the Accuracy of European GDP Forecasts," Working Papers 64, National Institute of Economic Research.
    23. Diebold, Francis X & Rudebusch, Glenn D, 1989. "Scoring the Leading Indicators," The Journal of Business, University of Chicago Press, vol. 62(3), pages 369-391, July.
    24. Skalin, Joakim & Teräsvirta, Timo, 2002. "Modeling Asymmetries And Moving Equilibria In Unemployment Rates," Macroeconomic Dynamics, Cambridge University Press, vol. 6(2), pages 202-241, April.
    25. Michael Woodford, 2001. "The Taylor Rule and Optimal Monetary Policy," American Economic Review, American Economic Association, vol. 91(2), pages 232-237, May.
    26. Norman R. Swanson & Halbert White, 1997. "A Model Selection Approach To Real-Time Macroeconomic Forecasting Using Linear Models And Artificial Neural Networks," The Review of Economics and Statistics, MIT Press, vol. 79(4), pages 540-550, November.
    27. Heilemann, Ullrich & Stekler, H. O., 2003. "Has the accuracy of German macroeconomic forecasts improved?," Technical Reports 2003,31, Technische Universität Dortmund, Sonderforschungsbereich 475: Komplexitätsreduktion in multivariaten Datenstrukturen.
    28. Isiklar, Gultekin & Lahiri, Kajal, 2007. "How far ahead can we forecast? Evidence from cross-country surveys," International Journal of Forecasting, Elsevier, vol. 23(2), pages 167-187.
    29. Pesaran, M. Hashem & Timmermann, Allan, 2004. "How costly is it to ignore breaks when forecasting the direction of a time series?," International Journal of Forecasting, Elsevier, vol. 20(3), pages 411-425.
    30. Oller, Lars-Erik & Teterukovsky, Alex, 2007. "Quantifying the quality of macroeconomic variables," International Journal of Forecasting, Elsevier, vol. 23(2), pages 205-217.
    31. Christina D. Romer & David H. Romer, 2008. "The FOMC versus the Staff: Where Can Monetary Policymakers Add Value?," American Economic Review, American Economic Association, vol. 98(2), pages 230-235, May.
    32. Milas, Costas & Rothman, Philip, 2008. "Out-of-sample forecasting of unemployment rates with pooled STVECM forecasts," International Journal of Forecasting, Elsevier, vol. 24(1), pages 101-121.
    33. Ray Fair, 2009. "Analyzing Macroeconomic Forecastability," Yale School of Management Working Papers amz2443, Yale School of Management, revised 01 Oct 2009.
    34. Jörg Döpke & Ulrich Fritsche, 2006. "Growth and inflation forecasts for Germany a panel-based assessment of accuracy and efficiency," Empirical Economics, Springer, vol. 31(3), pages 777-798, September.
    35. Christopher D. Carroll, 2003. "Macroeconomic Expectations of Households and Professional Forecasters," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 118(1), pages 269-298.
    36. Schnader, M H & Stekler, H O, 1990. "Evaluating Predictions of Change," The Journal of Business, University of Chicago Press, vol. 63(1), pages 99-107, January.
    37. Newey, Whitney & West, Kenneth, 2014. "A simple, positive semi-definite, heteroscedasticity and autocorrelation consistent covariance matrix," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 33(1), pages 125-132.
    38. Lukas Vogel, 2007. "How do the OECD Growth Projections for the G7 Economies Perform?: A Post-Mortem," OECD Economics Department Working Papers 573, OECD Publishing.
    39. Clements, Michael P & Krolzig, Hans-Martin, 2003. "Business Cycle Asymmetries: Characterization and Testing Based on Markov-Switching Autoregressions," Journal of Business & Economic Statistics, American Statistical Association, vol. 21(1), pages 196-211, January.
    40. Harvey, David & Leybourne, Stephen & Newbold, Paul, 1997. "Testing the equality of prediction mean squared errors," International Journal of Forecasting, Elsevier, vol. 13(2), pages 281-291, June.
    41. Smith, Stanley K. & Sincich, Terry, 1992. "Evaluating the forecast accuracy and bias of alternative population projections for states," International Journal of Forecasting, Elsevier, vol. 8(3), pages 495-508, November.
    42. Merton, Robert C, 1981. "On Market Timing and Investment Performance. I. An Equilibrium Theory of Value for Market Forecasts," The Journal of Business, University of Chicago Press, vol. 54(3), pages 363-406, July.
    43. repec:gwc:wpaper:2008-9 is not listed on IDEAS
    44. Michael S. Kaylen & Jon A. Brandt, 1988. "A Note on Qualitative Forecast Evaluation: Comment," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 70(2), pages 415-416.
    45. Francis X. Diebold & Glenn D. Rudebusch, 1989. "Forecasting output with the composite leading index: an ex ante analysis," Finance and Economics Discussion Series 90, Board of Governors of the Federal Reserve System (U.S.).
    46. Neftci, Salih N, 1984. "Are Economic Time Series Asymmetric over the Business Cycle?," Journal of Political Economy, University of Chicago Press, vol. 92(2), pages 307-328, April.
    47. Heilemann, Ullrich, 2002. "Increasing the transparency of macroeconometric forecasts: a report from the trenches," International Journal of Forecasting, Elsevier, vol. 18(1), pages 85-105.
    48. Leitch, Gordon & Tanner, J Ernest, 1991. "Economic Forecast Evaluation: Profits versus the Conventional Error Measures," American Economic Review, American Economic Association, vol. 81(3), pages 580-590, June.
    49. Batchelor, R A, 1990. "All Forecasters Are Equal," Journal of Business & Economic Statistics, American Statistical Association, vol. 8(1), pages 143-144, January.
    50. Taylor, John B., 1993. "Discretion versus policy rules in practice," Carnegie-Rochester Conference Series on Public Policy, Elsevier, vol. 39(1), pages 195-214, December.
    51. Philip Rothman, 1998. "Forecasting Asymmetric Unemployment Rates," The Review of Economics and Statistics, MIT Press, vol. 80(1), pages 164-168, February.
    52. D. A. Peel & A. E. H. Speight, 2000. "Threshold nonlinearities in unemployment rates: further evidence for the UK and G3 economies," Applied Economics, Taylor & Francis Journals, vol. 32(6), pages 705-715.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Herman O. Stekler, 2008. "What Do We Know About G-7 Macro Forecasts?," Working Papers 2008-009, The George Washington University, Department of Economics, H. O. Stekler Research Program on Forecasting.
    2. Bespalova, Olga, 2018. "Forecast Evaluation in Macroeconomics and International Finance. Ph.D. thesis, George Washington University, Washington, DC, USA," MPRA Paper 117706, University Library of Munich, Germany.
    3. Milas, Costas & Rothman, Philip, 2008. "Out-of-sample forecasting of unemployment rates with pooled STVECM forecasts," International Journal of Forecasting, Elsevier, vol. 24(1), pages 101-121.
    4. Constantin Burgi, 2016. "What Do We Lose When We Average Expectations?," Working Papers 2016-013, The George Washington University, Department of Economics, H. O. Stekler Research Program on Forecasting.
    5. Baghestani, Hamid, 2011. "Federal Reserve and private forecasts of growth in investment," Journal of Economics and Business, Elsevier, vol. 63(4), pages 290-305, July.
    6. Heilemann, Ullrich & Stekler, H. O., 2003. "Has the accuracy of German macroeconomic forecasts improved?," Technical Reports 2003,31, Technische Universität Dortmund, Sonderforschungsbereich 475: Komplexitätsreduktion in multivariaten Datenstrukturen.
    7. repec:spo:wpecon:info:hdl:2441/f4rshpf3v1umfa09lat09b1bg is not listed on IDEAS
    8. Heilemann Ullrich & Stekler Herman O., 2013. "Has The Accuracy of Macroeconomic Forecasts for Germany Improved?," German Economic Review, De Gruyter, vol. 14(2), pages 235-253, May.
    9. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    10. Kappler, Marcus, 2007. "Projecting the Medium-Term: Outcomes and Errors for GDP Growth," ZEW Discussion Papers 07-068, ZEW - Leibniz Centre for European Economic Research.
    11. Tara Sinclair & H. O. Stekler & L. Kitzinger, 2010. "Directional forecasts of GDP and inflation: a joint evaluation with an application to Federal Reserve predictions," Applied Economics, Taylor & Francis Journals, vol. 42(18), pages 2289-2297.
    12. repec:hal:wpspec:info:hdl:2441/f4rshpf3v1umfa09lat09b1bg is not listed on IDEAS
    13. Karsten Müller, 2022. "German forecasters’ narratives: How informative are German business cycle forecast reports?," Empirical Economics, Springer, vol. 62(5), pages 2373-2415, May.
    14. repec:hal:spmain:info:hdl:2441/f4rshpf3v1umfa09lat09b1bg is not listed on IDEAS
    15. Paul Hubert, 2010. "Monetary policy, imperfect information and the expectations channel [Politique monétaire,information imparfaite et canal des anticipations]," SciencePo Working papers Main tel-04095385, HAL.
    16. repec:spo:wpmain:info:hdl:2441/f4rshpf3v1umfa09lat09b1bg is not listed on IDEAS
    17. Terasvirta, Timo, 2006. "Forecasting economic variables with nonlinear models," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 1, chapter 8, pages 413-457, Elsevier.
    18. McCracken,M.W. & West,K.D., 2001. "Inference about predictive ability," Working papers 14, Wisconsin Madison - Social Systems.
    19. Döpke, Jörg & Fritsche, Ulrich & Müller, Karsten, 2019. "Has macroeconomic forecasting changed after the Great Recession? Panel-based evidence on forecast accuracy and forecaster behavior from Germany," Journal of Macroeconomics, Elsevier, vol. 62(C).
    20. D'Amuri, Francesco & Marcucci, Juri, 2009. "‘Google it!’ Forecasting the US unemployment rate with a Google job search index," ISER Working Paper Series 2009-32, Institute for Social and Economic Research.
    21. Rossi, Barbara, 2013. "Advances in Forecasting under Instability," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1203-1324, Elsevier.
    22. Elena Olmedo, 2014. "Forecasting Spanish Unemployment Using Near Neighbour and Neural Net Techniques," Computational Economics, Springer;Society for Computational Economics, vol. 43(2), pages 183-197, February.
    23. Dovern, Jonas & Jannsen, Nils, 2017. "Systematic errors in growth expectations over the business cycle," International Journal of Forecasting, Elsevier, vol. 33(4), pages 760-769.
    24. Barbara Rossi, 2019. "Forecasting in the presence of instabilities: How do we know whether models predict well and how to improve them," Economics Working Papers 1711, Department of Economics and Business, Universitat Pompeu Fabra, revised Jul 2021.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gwc:wpaper:2010-002. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: GW Economics Department (email available below). General contact details of provider: https://edirc.repec.org/data/pfgwuus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.