IDEAS home Printed from https://ideas.repec.org/a/fip/fedaer/y1998iq4p4-20nv.83no.4.html
   My bibliography  Save this article

Data vintages and measuring forecast model performance

Author

Listed:
  • John C. Robertson
  • Ellis W. Tallman

Abstract

The data on economic variables are usually estimates, and these estimates may be revised many times after their initial publication. Most historical forecast evaluation exercises use the \"latest available\" or most recently revised vintage of historical data when constructing the forecasts-that is, they use estimates that may well have been unavailable to a forecaster in real time. Evaluations using such data could thus give a misleading picture of the forecast performance that can be expected in real-time situations. This fact is particularly relevant if a forecasting model's performance is to be compared with that of published real-time forecasts. One practical question is whether actually using the data set available to a forecaster in real time would lead to inferences that are substantially different from those made using the latest available vintage of data. A related question is whether it matters which vintage of data the forecasts are evaluated against. ; The authors argue that the choice of data vintage can have both a quantitative and a qualitative influence on forecast and model comparisons, at least over short horizons. This influence is illustrated by examining the performance of the composite index of leading indicators as a forecaster of alternative measures of real output. However, more research is required in order to determine whether the results generalize to forecasts of other series that are subject to revision, such as the various money aggregate measures.

Suggested Citation

  • John C. Robertson & Ellis W. Tallman, 1998. "Data vintages and measuring forecast model performance," Economic Review, Federal Reserve Bank of Atlanta, vol. 83(Q 4), pages 4-20.
    • Handle: RePEc:fip:fedaer:y:1998:i:q4:p:4-20:n:v.83no.4
    as

    Download full text from publisher

    File URL: https://www.frbatlanta.org/-/media/documents/research/publications/economic-review/1998/vol83no4_robertson-tallman.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Makridakis, Spyros & Chatfield, Chris & Hibon, Michele & Lawrence, Michael & Mills, Terence & Ord, Keith & Simmons, LeRoy F., 1993. "The M2-competition: A real-time judgmentally based forecasting study," International Journal of Forecasting, Elsevier, vol. 9(1), pages 5-22, April.
    2. Norman R. Swanson & Halbert White, 1997. "A Model Selection Approach To Real-Time Macroeconomic Forecasting Using Linear Models And Artificial Neural Networks," The Review of Economics and Statistics, MIT Press, vol. 79(4), pages 540-550, November.
    3. Athanasios Orphanides, 2001. "Monetary Policy Rules Based on Real-Time Data," American Economic Review, American Economic Association, vol. 91(4), pages 964-985, September.
    4. Swanson Norman, 1996. "Forecasting Using First-Available Versus Fully Revised Economic Time-Series Data," Studies in Nonlinear Dynamics & Econometrics, De Gruyter, vol. 1(1), pages 1-20, April.
    5. Hamilton, James D & Perez-Quiros, Gabriel, 1996. "What Do the Leading Indicators Lead?," The Journal of Business, University of Chicago Press, vol. 69(1), pages 27-49, January.
    6. Fair, Ray C & Shiller, Robert J, 1990. "Comparing Information in Forecasts from Econometric Models," American Economic Review, American Economic Association, vol. 80(3), pages 375-389, June.
    7. Stephen K. McNees, 1988. "How accurate are macroeconomic forecasts?," New England Economic Review, Federal Reserve Bank of Boston, issue Jul, pages 15-36.
    8. James H. Stock & Mark W. Watson, 1998. "A Comparison of Linear and Nonlinear Univariate Models for Forecasting Macroeconomic Time Series," NBER Working Papers 6607, National Bureau of Economic Research, Inc.
    9. Francis X. Diebold & Glenn D. Rudebusch, 1989. "Forecasting output with the composite leading index: an ex ante analysis," Finance and Economics Discussion Series 90, Board of Governors of the Federal Reserve System (U.S.).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Christoffersen, Peter & Ghysels, Eric & Swanson, Norman R., 2002. "Let's get "real" about using economic data," Journal of Empirical Finance, Elsevier, vol. 9(3), pages 343-360, August.
    2. Evan F. Koenig & Sheila Dolmas & Jeremy Piger, 2003. "The Use and Abuse of Real-Time Data in Economic Forecasting," The Review of Economics and Statistics, MIT Press, vol. 85(3), pages 618-628, August.
    3. Croushore, Dean & Stark, Tom, 2001. "A real-time data set for macroeconomists," Journal of Econometrics, Elsevier, vol. 105(1), pages 111-130, November.
    4. Eric Ghysels & Norman R. Swanson & Myles Callan, 2002. "Monetary Policy Rules with Model and Data Uncertainty," Southern Economic Journal, John Wiley & Sons, vol. 69(2), pages 239-265, October.
    5. Dean Croushore & Tom Stark, 2000. "A real-time data set for macroeconomists: does data vintage matter for forecasting?," Working Papers 00-6, Federal Reserve Bank of Philadelphia.
    6. Stark, Tom & Croushore, Dean, 2002. "Forecasting with a real-time data set for macroeconomists," Journal of Macroeconomics, Elsevier, vol. 24(4), pages 507-531, December.
    7. Domenico Giannone & Jérôme Henry & Magdalena Lalik & Michele Modugno, 2012. "An Area-Wide Real-Time Database for the Euro Area," The Review of Economics and Statistics, MIT Press, vol. 94(4), pages 1000-1013, November.
    8. Dean Croushore, 2011. "Frontiers of Real-Time Data Analysis," Journal of Economic Literature, American Economic Association, vol. 49(1), pages 72-100, March.
    9. Ullrich Heilemann & Herman Stekler, 2010. "Perspectives on Evaluating Macroeconomic Forecasts," Working Papers 2010-002, The George Washington University, Department of Economics, H. O. Stekler Research Program on Forecasting.
    10. Dean Croushore & Tom Stark, 2002. "Is macroeconomic research robust to alternative data sets?," Working Papers 02-3, Federal Reserve Bank of Philadelphia.
    11. van Dijk, D.J.C. & Franses, Ph.H.B.F. & Ravazzolo, F., 2007. "Evaluating real-time forecasts in real-time," Econometric Institute Research Papers EI 2007-33, Erasmus University Rotterdam, Erasmus School of Economics (ESE), Econometric Institute.
    12. Croushore, Dean & Evans, Charles L., 2006. "Data revisions and the identification of monetary policy shocks," Journal of Monetary Economics, Elsevier, vol. 53(6), pages 1135-1160, September.
    13. Bedri Kamil Onur Taş, 2016. "Does the Federal Reserve have Private Information about its Future Actions?," Economica, London School of Economics and Political Science, vol. 83(331), pages 498-517, July.
    14. Robert H. McGuckin & Ataman Ozyildirim, 2004. "Real-Time Tests of the Leading Economic Index: Do Changes in the Index Composition Matter?," Journal of Business Cycle Measurement and Analysis, OECD Publishing, Centre for International Research on Economic Tendency Surveys, vol. 2004(2), pages 171-191.
    15. Corradi, Valentina & Fernandez, Andres & Swanson, Norman R., 2009. "Information in the Revision Process of Real-Time Datasets," Journal of Business & Economic Statistics, American Statistical Association, vol. 27(4), pages 455-467.
    16. Clements, Michael P. & Beatriz Galvão, Ana, 2010. "First announcements and real economic activity," European Economic Review, Elsevier, vol. 54(6), pages 803-817, August.
    17. Parigi, Giuseppe & Golinelli, Roberto, 2005. "Short-Run Italian GDP Forecasting and Real-Time Data," CEPR Discussion Papers 5302, C.E.P.R. Discussion Papers.
    18. Rodríguez-Vargas, Adolfo, 2020. "Forecasting Costa Rican inflation with machine learning methods," Latin American Journal of Central Banking (previously Monetaria), Elsevier, vol. 1(1).
    19. repec:lan:wpaper:470 is not listed on IDEAS
    20. Vito Polito & Yunyi Zhang, 2021. "Tackling Large Outliers in Macroeconomic Data with Vector Artificial Neural Network Autoregression," CESifo Working Paper Series 9395, CESifo.
    21. Massimiliano Marcellino, "undated". "Forecast pooling for short time series of macroeconomic variables," Working Papers 212, IGIER (Innocenzo Gasparini Institute for Economic Research), Bocconi University.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:fip:fedaer:y:1998:i:q4:p:4-20:n:v.83no.4. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Meredith Rector (email available below). General contact details of provider: https://edirc.repec.org/data/frbatus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.