IDEAS home Printed from https://ideas.repec.org/p/hhs/rbnkwp/0328.html
   My bibliography  Save this paper

Adjusting for Information Content when Comparing Forecast Performance

Author

Listed:
  • Andersson, Michael K.

    (Finansinspektionen)

  • Aranki, Ted

    (Monetary Policy Department, Central Bank of Sweden)

  • Reslow, André

    (Monetary Policy Department, Central Bank of Sweden)

Abstract

Cross institutional forecast evaluations may be severely distorted by the fact that forecasts are made at different points in time, and thus with different amount of information. This paper proposes a method to account for these differences. The method computes the timing effect and the forecaster's ability simultaneously. Monte Carlo simulation demonstrate that evaluations that do not adjust for the differences in information content may be misleading. In addition, the method is applied on a real-world data set of 10 Swedish forecasters for the period 1999-2015. The results show that the ranking of the forecasters is affected by the proposed adjustment.

Suggested Citation

  • Andersson, Michael K. & Aranki, Ted & Reslow, André, 2016. "Adjusting for Information Content when Comparing Forecast Performance," Working Paper Series 328, Sveriges Riksbank (Central Bank of Sweden).
    • Handle: RePEc:hhs:rbnkwp:0328
    as

    Download full text from publisher

    File URL: http://www.riksbank.se/Documents/Rapporter/Working_papers/2016/rap_wp328_160915.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Allan Timmermann, 2007. "An Evaluation of the World Economic Outlook Forecasts," IMF Staff Papers, Palgrave Macmillan, vol. 54(1), pages 1-33, May.
    2. White, Halbert, 1980. "A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity," Econometrica, Econometric Society, vol. 48(4), pages 817-838, May.
    3. Laura Gonzalez Cabanillas & Alessio Terzi, 2012. "The accuracy of the European Commission's forecasts re-examined," European Economy - Economic Papers 2008 - 2015 476, Directorate General Economic and Financial Affairs (DG ECFIN), European Commission.
    4. Lukas Vogel, 2007. "How do the OECD Growth Projections for the G7 Economies Perform?: A Post-Mortem," OECD Economics Department Working Papers 573, OECD Publishing.
    5. Gneiting, Tilmann, 2011. "Making and Evaluating Point Forecasts," Journal of the American Statistical Association, American Statistical Association, vol. 106(494), pages 746-762.
    6. Davies, Antony, 2006. "A framework for decomposing shocks and measuring volatilities derived from multi-dimensional panel data of survey forecasts," International Journal of Forecasting, Elsevier, vol. 22(2), pages 373-393.
    7. Andrew Bauer & Robert A. Eisenbeis & Daniel F. Waggoner & Tao Zha, 2003. "Forecast evaluation with cross-sectional data: The Blue Chip Surveys," Economic Review, Federal Reserve Bank of Atlanta, vol. 88(Q2), pages 17-31.
    8. Davies, Anthony & Lahiri, Kajal, 1995. "A new framework for analyzing survey forecasts using three-dimensional panel data," Journal of Econometrics, Elsevier, vol. 68(1), pages 205-227, July.
    9. Boero, Gianna & Smith, Jeremy & Wallis, Kenneth F., 2008. "Evaluating a three-dimensional panel of point forecasts: The Bank of England Survey of External Forecasters," International Journal of Forecasting, Elsevier, vol. 24(3), pages 354-367.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Cipullo, Davide & Reslow, André, 2019. "Biased Forecasts to Affect Voting Decisions? The Brexit Case," Working Paper Series 364, Sveriges Riksbank (Central Bank of Sweden).
    2. Cipullo, Davide & Reslow, André, 2022. "Electoral cycles in macroeconomic forecasts," Journal of Economic Behavior & Organization, Elsevier, vol. 202(C), pages 307-340.
    3. Reslow, André, 2019. "Inefficient Use of Competitors'Forecasts?," Working Paper Series 380, Sveriges Riksbank (Central Bank of Sweden).
    4. Kontogeorgos, Georgios & Lambrias, Kyriacos, 2019. "An analysis of the Eurosystem/ECB projections," Working Paper Series 2291, European Central Bank.
    5. Ard Reijer & Andreas Johansson, 2019. "Nowcasting Swedish GDP with a large and unbalanced data set," Empirical Economics, Springer, vol. 57(4), pages 1351-1373, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tsuchiya, Yoichi, 2023. "Assessing the World Bank’s growth forecasts," Economic Analysis and Policy, Elsevier, vol. 77(C), pages 64-84.
    2. Klein, Tony, 2022. "Agree to disagree? Predictions of U.S. nonfarm payroll changes between 2008 and 2020 and the impact of the COVID19 labor shock," Journal of Economic Behavior & Organization, Elsevier, vol. 194(C), pages 264-286.
    3. Dovern, Jonas & Weisser, Johannes, 2011. "Accuracy, unbiasedness and efficiency of professional macroeconomic forecasts: An empirical comparison for the G7," International Journal of Forecasting, Elsevier, vol. 27(2), pages 452-465.
    4. Mihaela SIMIONESCU, 2015. "The Evaluation of Global Accuracy of Romanian Inflation Rate Predictions Using Mahalanobis Distance," Management Dynamics in the Knowledge Economy, College of Management, National University of Political Studies and Public Administration, vol. 3(1), pages 133-149, March.
    5. Behrens, Christoph, 2020. "German trade forecasts since 1970: An evaluation using the panel dimension," Working Papers 26, German Research Foundation's Priority Programme 1859 "Experience and Expectation. Historical Foundations of Economic Behaviour", Humboldt University Berlin.
    6. Bourmpoula, Evangelia. & Wieser, Christina., 2014. "An accuracy assessment of the global employment trends' unemployment rate forecasts," Global Employment Trends Reports 994888903402676, International Labour Office, Economic and Labour Market Analysis Department.
    7. William C. Horrace & Kurt E. Schnier, 2010. "Fixed-Effect Estimation of Highly Mobile Production Technologies," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 92(5), pages 1432-1445.
    8. Thomas Jobert & Lionel Persyn, 2012. "Quelques constats sur les prévisions conjoncturelles de la croissance française," Revue d'économie politique, Dalloz, vol. 122(6), pages 833-849.
    9. Masahiro Ashiya, 2009. "Strategic bias and professional affiliations of macroeconomic forecasters," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 28(2), pages 120-130.
    10. Carlos Fonseca Marinheiro, 2010. "Fiscal sustainability and the accuracy of macroeconomic forecasts: do supranational forecasts rather than government forecasts make a difference?," GEMF Working Papers 2010-07, GEMF, Faculty of Economics, University of Coimbra.
    11. Ager, P. & Kappler, M. & Osterloh, S., 2009. "The accuracy and efficiency of the Consensus Forecasts: A further application and extension of the pooled approach," International Journal of Forecasting, Elsevier, vol. 25(1), pages 167-181.
    12. repec:ilo:ilowps:488890 is not listed on IDEAS
    13. Boero, Gianna & Smith, Jeremy & Wallis, Kenneth F., 2008. "Evaluating a three-dimensional panel of point forecasts: The Bank of England Survey of External Forecasters," International Journal of Forecasting, Elsevier, vol. 24(3), pages 354-367.
    14. Yoichi Tsuchiya, 2021. "Thirty‐year assessment of Asian Development Bank's forecasts," Asian-Pacific Economic Literature, The Crawford School, The Australian National University, vol. 35(2), pages 18-40, November.
    15. Rosen Valchev & Antony Davies, 2009. "Transparency, Performance, and Agency Budgets: A Rational Expectations Modeling Approach," Working Papers 2009-004, The George Washington University, Department of Economics, H. O. Stekler Research Program on Forecasting.
    16. Drechsel, Katja & Giesen, Sebastian & Lindner, Axel, 2014. "Outperforming IMF Forecasts by the Use of Leading Indicators," VfS Annual Conference 2014 (Hamburg): Evidence-based Economic Policy 100393, Verein für Socialpolitik / German Economic Association.
    17. Timmermann, Allan & Qu, Ritong & Zhu, Yinchu, 2019. "Do Any Economists Have Superior Forecasting Skills?," CEPR Discussion Papers 14112, C.E.P.R. Discussion Papers.
    18. Baghestani, Hamid, 2006. "An evaluation of the professional forecasts of U.S. long-term interest rates," Review of Financial Economics, Elsevier, vol. 15(2), pages 177-191.
    19. Klein, Tony, 2021. "Agree to Disagree? Predictions of U.S. Nonfarm Payroll Changes between 2008 and 2020 and the Impact of the COVID19 Labor Shock," QBS Working Paper Series 2021/07, Queen's University Belfast, Queen's Business School.
    20. Ildeberta Abreu, 2011. "International organisations’ vs. private analysts’ forecasts: an evaluation," Working Papers w201120, Banco de Portugal, Economics and Research Department.

    More about this item

    Keywords

    Forecast error; Forecast comparison; Publication time; Evaluation; Error component model; Panel data;
    All these keywords.

    JEL classification:

    • C23 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Models with Panel Data; Spatio-temporal Models
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods
    • E37 - Macroeconomics and Monetary Economics - - Prices, Business Fluctuations, and Cycles - - - Forecasting and Simulation: Models and Applications

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:hhs:rbnkwp:0328. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Lena Löfgren (email available below). General contact details of provider: https://edirc.repec.org/data/rbgovse.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.