IDEAS home Printed from https://ideas.repec.org/p/msh/ebswps/2005-13.html
   My bibliography  Save this paper

Another Look at Measures of Forecast Accuracy

Author

Listed:
  • Rob J. Hyndman
  • Anne B. Koehler

Abstract

We discuss and compare measures of accuracy of univariate time series forecasts. The methods used in the M-competition and the M3-competition, and many of the measures recommended by previous authors on this topic, are found to be inadequate, and many of them are degenerate in commonly occurring situations. Instead, we propose that the mean absolute scaled error become the standard measure for comparing forecast accuracy across multiple time series.

Suggested Citation

  • Rob J. Hyndman & Anne B. Koehler, 2005. "Another Look at Measures of Forecast Accuracy," Monash Econometrics and Business Statistics Working Papers 13/05, Monash University, Department of Econometrics and Business Statistics.
    • Handle: RePEc:msh:ebswps:2005-13
    as

    Download full text from publisher

    File URL: http://www.buseco.monash.edu.au/ebs/pubs/wpapers/2005/wp13-05.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Hyndman, Rob J. & Koehler, Anne B. & Snyder, Ralph D. & Grose, Simone, 2002. "A state space framework for automatic forecasting using exponential smoothing methods," International Journal of Forecasting, Elsevier, vol. 18(3), pages 439-454.
    2. Everette S. Gardner, 1990. "Evaluating Forecast Performance in an Inventory Control System," Management Science, INFORMS, vol. 36(4), pages 490-499, April.
    3. Goodwin, Paul & Lawton, Richard, 1999. "On the asymmetry of the symmetric MAPE," International Journal of Forecasting, Elsevier, vol. 15(4), pages 405-408, October.
    4. Chatfield, Chris, 1988. "Apples, oranges and mean square error," International Journal of Forecasting, Elsevier, vol. 4(4), pages 515-518.
    5. David Swanson & Jeff Tayman & Charles Barr, 2000. "A note on the measurement of accuracy for subnational demographic estimates," Demography, Springer;Population Association of America (PAA), vol. 37(2), pages 193-201, May.
    6. Thompson, Patrick A., 1990. "An MSE statistic for comparing forecast accuracy across series," International Journal of Forecasting, Elsevier, vol. 6(2), pages 219-227, July.
    7. Hyndman, Rob J. & Billah, Baki, 2003. "Unmasking the Theta method," International Journal of Forecasting, Elsevier, vol. 19(2), pages 287-290.
    8. Granger, C.W.J. & Pesaran, M. H., 1999. "Economic and Statistical Measures of Forecast Accuracy," Cambridge Working Papers in Economics 9910, Faculty of Economics, University of Cambridge.
    9. Makridakis, Spyros, 1993. "Accuracy measures: theoretical and practical concerns," International Journal of Forecasting, Elsevier, vol. 9(4), pages 527-529, December.
    10. Assimakopoulos, V. & Nikolopoulos, K., 2000. "The theta model: a decomposition approach to forecasting," International Journal of Forecasting, Elsevier, vol. 16(4), pages 521-530.
    11. Billah, Baki & King, Maxwell L. & Snyder, Ralph D. & Koehler, Anne B., 2006. "Exponential smoothing model selection for forecasting," International Journal of Forecasting, Elsevier, vol. 22(2), pages 239-247.
    12. Makridakis, Spyros & Hibon, Michele, 2000. "The M3-Competition: results, conclusions and implications," International Journal of Forecasting, Elsevier, vol. 16(4), pages 451-476.
    13. Clements, M.P. & Hendry, D., 1992. "On the Limitations of Comparing Mean Square Forecast Errors," Economics Series Working Papers 99138, University of Oxford, Department of Economics.
    14. Lawrence, Michael & O'Connor, Marcus, 2005. "Judgmental forecasting in the presence of loss functions," International Journal of Forecasting, Elsevier, vol. 21(1), pages 3-14.
    15. Chatfield, Chris, 1992. "A commentary on error measures," International Journal of Forecasting, Elsevier, vol. 8(1), pages 100-102, June.
    16. Armstrong, J. Scott & Collopy, Fred, 1992. "Error measures for generalizing about forecasting methods: Empirical comparisons," International Journal of Forecasting, Elsevier, vol. 8(1), pages 69-80, June.
    17. Koning, Alex J. & Franses, Philip Hans & Hibon, Michele & Stekler, H.O., 2005. "The M3 competition: Statistical tests of the results," International Journal of Forecasting, Elsevier, vol. 21(3), pages 397-409.
    18. Fildes, Robert, 1992. "The evaluation of extrapolative forecasting methods," International Journal of Forecasting, Elsevier, vol. 8(1), pages 81-98, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. De Gooijer, Jan G. & Hyndman, Rob J., 2006. "25 years of time series forecasting," International Journal of Forecasting, Elsevier, vol. 22(3), pages 443-473.
    2. Jan G. De Gooijer & Rob J. Hyndman, 2005. "25 Years of IIF Time Series Forecasting: A Selective Review," Monash Econometrics and Business Statistics Working Papers 12/05, Monash University, Department of Econometrics and Business Statistics.
    3. Makridakis, Spyros & Spiliotis, Evangelos & Assimakopoulos, Vassilios, 2020. "The M4 Competition: 100,000 time series and 61 forecasting methods," International Journal of Forecasting, Elsevier, vol. 36(1), pages 54-74.
    4. Spiliotis, Evangelos & Nikolopoulos, Konstantinos & Assimakopoulos, Vassilios, 2019. "Tales from tails: On the empirical distributions of forecasting errors and their implication to risk," International Journal of Forecasting, Elsevier, vol. 35(2), pages 687-698.
    5. Theodosiou, Marina, 2011. "Forecasting monthly and quarterly time series using STL decomposition," International Journal of Forecasting, Elsevier, vol. 27(4), pages 1178-1195, October.
    6. Kim, Sungil & Kim, Heeyoung, 2016. "A new metric of absolute percentage error for intermittent demand forecasts," International Journal of Forecasting, Elsevier, vol. 32(3), pages 669-679.
    7. Crone, Sven F. & Hibon, Michèle & Nikolopoulos, Konstantinos, 2011. "Advances in forecasting with neural networks? Empirical evidence from the NN3 competition on time series prediction," International Journal of Forecasting, Elsevier, vol. 27(3), pages 635-660, July.
    8. Jennifer L. Castle & Jurgen A. Doornik & David F. Hendry, 2021. "Forecasting Principles from Experience with Forecasting Competitions," Forecasting, MDPI, vol. 3(1), pages 1-28, February.
    9. Yelland, Phillip M., 2010. "Bayesian forecasting of parts demand," International Journal of Forecasting, Elsevier, vol. 26(2), pages 374-396, April.
    10. Fiorucci, Jose A. & Pellegrini, Tiago R. & Louzada, Francisco & Petropoulos, Fotios & Koehler, Anne B., 2016. "Models for optimising the theta method and their relationship to state space models," International Journal of Forecasting, Elsevier, vol. 32(4), pages 1151-1161.
    11. Gardner, Everette Jr., 2006. "Exponential smoothing: The state of the art--Part II," International Journal of Forecasting, Elsevier, vol. 22(4), pages 637-666.
    12. Dean W. Wichern & Benito E. Flores, 2005. "Evaluating forecasts: a look at aggregate bias and accuracy measures," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 24(6), pages 433-451.
    13. Semenoglou, Artemios-Anargyros & Spiliotis, Evangelos & Makridakis, Spyros & Assimakopoulos, Vassilios, 2021. "Investigating the accuracy of cross-learning time series forecasting methods," International Journal of Forecasting, Elsevier, vol. 37(3), pages 1072-1084.
    14. Kang, Yanfei & Spiliotis, Evangelos & Petropoulos, Fotios & Athiniotis, Nikolaos & Li, Feng & Assimakopoulos, Vassilios, 2021. "Déjà vu: A data-centric forecasting approach through time series cross-similarity," Journal of Business Research, Elsevier, vol. 132(C), pages 719-731.
    15. Makridakis, Spyros & Spiliotis, Evangelos & Assimakopoulos, Vassilios, 2022. "M5 accuracy competition: Results, findings, and conclusions," International Journal of Forecasting, Elsevier, vol. 38(4), pages 1346-1364.
    16. Petropoulos, Fotios & Makridakis, Spyros & Assimakopoulos, Vassilios & Nikolopoulos, Konstantinos, 2014. "‘Horses for Courses’ in demand forecasting," European Journal of Operational Research, Elsevier, vol. 237(1), pages 152-163.
    17. Fotios Petropoulos & Enno Siemsen, 2023. "Forecast Selection and Representativeness," Management Science, INFORMS, vol. 69(5), pages 2672-2690, May.
    18. Sbrana, Giacomo & Silvestrini, Andrea, 2022. "Random coefficient state-space model: Estimation and performance in M3–M4 competitions," International Journal of Forecasting, Elsevier, vol. 38(1), pages 352-366.
    19. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    20. Fildes, Robert & Petropoulos, Fotios, 2015. "Simple versus complex selection rules for forecasting many time series," Journal of Business Research, Elsevier, vol. 68(8), pages 1692-1701.

    More about this item

    Keywords

    Forecast accuracy; Forecast evaluation; Forecast error measures; M-competition; Mean absolute scaled error.;
    All these keywords.

    JEL classification:

    • C53 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Forecasting and Prediction Models; Simulation Methods
    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection
    • C22 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Time-Series Models; Dynamic Quantile Regressions; Dynamic Treatment Effect Models; Diffusion Processes

    NEP fields

    This paper has been announced in the following NEP Reports:

    Lists

    This item is featured on the following reading lists, Wikipedia, or ReplicationWiki pages:
    1. Technology Assessment

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:msh:ebswps:2005-13. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Professor Xibin Zhang (email available below). General contact details of provider: https://edirc.repec.org/data/dxmonau.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.