IDEAS home Printed from https://ideas.repec.org/p/ucm/doicae/1107.html
   My bibliography  Save this paper

A statistical test for forecast evaluation under a discrete loss function

Author

Abstract

We propose a new approach to evaluating the usefulness of a set of forecasts, based on the use of a discrete loss function de ned on the space of data and forecasts. Existing procedures for such an evaluation either do not allow for formal testing, or use tests statistics based just on the frequency distribution of (data, forecasts) -pairs. They can easily lead to misleading conclusions in some reasonable situations, because of the way they formalize the underlying null hypothesis that "the set of forecasts is not useful". Even though the ambiguity of the underlying null hypothesis precludes us from performing a standard analysis of the size and power of the tests, we get results suggesting that the proposed DISC test performs better than its competitors.

Suggested Citation

  • Francisco J. Eransus & Alfonso Novales Cinca, 2011. "A statistical test for forecast evaluation under a discrete loss function," Documentos de Trabajo del ICAE 2011-07, Universidad Complutense de Madrid, Facultad de Ciencias Económicas y Empresariales, Instituto Complutense de Análisis Económico.
    • Handle: RePEc:ucm:doicae:1107
    as

    Download full text from publisher

    File URL: https://eprints.ucm.es/id/eprint/12521/1/1107.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Henriksson, Roy D & Merton, Robert C, 1981. "On Market Timing and Investment Performance. II. Statistical Procedures for Evaluating Forecasting Skills," The Journal of Business, University of Chicago Press, vol. 54(4), pages 513-533, October.
    2. Ash, J. C. K. & Smyth, D. J. & Heravi, S. M., 1998. "Are OECD forecasts rational and useful?: a directional analysis," International Journal of Forecasting, Elsevier, vol. 14(3), pages 381-391, September.
    3. Greer, Mark, 2003. "Directional accuracy tests of long-term interest rate forecasts," International Journal of Forecasting, Elsevier, vol. 19(2), pages 291-298.
    4. Pesaran, M Hashem & Timmermann, Allan, 1992. "A Simple Nonparametric Test of Predictive Performance," Journal of Business & Economic Statistics, American Statistical Association, vol. 10(4), pages 561-565, October.
    5. Joutz, Fred & Stekler, H. O., 2000. "An evaluation of the predictions of the Federal Reserve," International Journal of Forecasting, Elsevier, vol. 16(1), pages 17-38.
    6. Merton, Robert C, 1981. "On Market Timing and Investment Performance. I. An Equilibrium Theory of Value for Market Forecasts," The Journal of Business, University of Chicago Press, vol. 54(3), pages 363-406, July.
    7. Mills, Terence C. & Pepper, Gordon T., 1999. "Assessing the forecasters: an analysis of the forecasting records of the Treasury, the London Business School and the National Institute," International Journal of Forecasting, Elsevier, vol. 15(3), pages 247-257, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. repec:hum:wpaper:sfb649dp2008-073 is not listed on IDEAS
    2. Blaskowitz, Oliver J. & Herwartz, Helmut, 2008. "Testing directional forecast value in the presence of serial correlation," SFB 649 Discussion Papers 2008-073, Humboldt University Berlin, Collaborative Research Center 649: Economic Risk.
    3. Tsuchiya, Yoichi, 2014. "Purchasing and supply managers provide early clues on the direction of the US economy: An application of a new market-timing test," International Review of Economics & Finance, Elsevier, vol. 29(C), pages 599-618.
    4. Tsuchiya, Yoichi, 2016. "Directional analysis of fiscal sustainability: Revisiting Domar's debt sustainability condition," International Review of Economics & Finance, Elsevier, vol. 41(C), pages 189-201.
    5. Hamid Baghestani, 2011. "A directional analysis of Federal Reserve predictions of growth in unit labor costs and productivity," International Review of Applied Economics, Taylor & Francis Journals, vol. 25(3), pages 303-311.
    6. Y. Tsuchiya, 2014. "A directional evaluation of corporate executives' exchange rate forecasts," Applied Economics, Taylor & Francis Journals, vol. 46(1), pages 95-101, January.
    7. Greer, Mark, 2003. "Directional accuracy tests of long-term interest rate forecasts," International Journal of Forecasting, Elsevier, vol. 19(2), pages 291-298.
    8. Tim Meyer, 2019. "On the Directional Accuracy of United States Housing Starts Forecasts: Evidence from Survey Data," The Journal of Real Estate Finance and Economics, Springer, vol. 58(3), pages 457-488, April.
    9. Mihaela Simionescu, 2014. "Directional accuracy for inflation and unemployment rate predictions in Romania," International Journal of Business and Economic Sciences Applied Research (IJBESAR), Democritus University of Thrace (DUTH), Kavala Campus, Greece, vol. 7(2), pages 129-138, September.
    10. Tsuchiya, Yoichi, 2013. "Do corporate executives have accurate predictions for the economy? A directional analysis," Economic Modelling, Elsevier, vol. 30(C), pages 167-174.
    11. Tsuchiya, Yoichi, 2016. "Do production managers predict turning points? A directional analysis," Economic Modelling, Elsevier, vol. 58(C), pages 1-8.
    12. Baghestani, Hamid, 2011. "Federal Reserve and private forecasts of growth in investment," Journal of Economics and Business, Elsevier, vol. 63(4), pages 290-305, July.
    13. Blaskowitz, Oliver & Herwartz, Helmut, 2014. "Testing the value of directional forecasts in the presence of serial correlation," International Journal of Forecasting, Elsevier, vol. 30(1), pages 30-42.
    14. Hamid Baghestani, 2010. "Evaluating Blue Chip forecasts of the trade-weighted dollar exchange rate," Applied Financial Economics, Taylor & Francis Journals, vol. 20(24), pages 1879-1889.
    15. Joshy Easaw & Saeed Heravi, 2009. "Are household subjective forecasts of personal finances accurate and useful? A directional analysis of the British Household Panel Survey," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 28(8), pages 667-680.
    16. Stekler, H. O. & Petrei, G., 2003. "Diagnostics for evaluating the value and rationality of economic forecasts," International Journal of Forecasting, Elsevier, vol. 19(4), pages 735-742.
    17. Yoichi Tsuchiya, 2012. "Is the Purchasing Managers' Index useful for assessing the economy's strength? A directional analysis," Economics Bulletin, AccessEcon, vol. 32(2), pages 1302-1311.
    18. Gultekin Isiklar & Kajal Lahiri & Prakash Loungani, 2006. "How quickly do forecasters incorporate news? Evidence from cross‐country surveys," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 21(6), pages 703-725, September.
    19. Jordi Pons, 2001. "The rationality of price forecasts: a directional analysis," Applied Financial Economics, Taylor & Francis Journals, vol. 11(3), pages 287-290.
    20. Masahiro Ashiya, 2006. "Are 16-month-ahead forecasts useful? A directional analysis of Japanese GDP forecasts," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 25(3), pages 201-207.
    21. IIZUKA Nobuo, 2013. "Predicting Business Cycle Phases by Professional Forecasters－ Are They Useful ?," ESRI Discussion paper series 305, Economic and Social Research Institute (ESRI).

    More about this item

    Keywords

    Forecasting Evaluation; Loss Function.;

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection
    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ucm:doicae:1107. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Águeda González Abad (email available below). General contact details of provider: https://edirc.repec.org/data/feucmes.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.