IDEAS home Printed from https://ideas.repec.org/a/inm/oropre/v65y2017i1p200-211.html
   My bibliography  Save this article

Percentage and Relative Error Measures in Forecast Evaluation

Author

Listed:
  • Victor Richmond R. Jose

    (McDonough School of Business, Georgetown University, Washington, DC 20057)

Abstract

Properties of two large families of scale-free forecast accuracy measures that include popular measures such as mean absolute percentage error, relative error, and squared percentage error, are examined in this paper. We describe the optimal reports when forecasts are evaluated using these measures. We also provide analytic expressions for the optimal Bayes’ act associated with these measures under a general power transformation for several well-known probability distributions. We then show that using measures from these two families may inadvertently provide incentives for either pessimism or optimism among forecasters, i.e., rewarding underforecasts or overforecasts relative to some reference measure of central tendency. As an illustration of these concepts, we examine the use of these measures for model selection in a forecast aggregation example using stock price forecasts derived from the Thomson Reuters Institutional Brokers’ Estimate System. This example illustrates how aggregation methods that always yield lower estimates relative to the mean or median generally exhibit better scores using percentage error-based measures, while those that yield higher estimates compared to the mean or median will effectively rank higher when relative error-based measures are used.

Suggested Citation

  • Victor Richmond R. Jose, 2017. "Percentage and Relative Error Measures in Forecast Evaluation," Operations Research, INFORMS, vol. 65(1), pages 200-211, February.
  • Handle: RePEc:inm:oropre:v:65:y:2017:i:1:p:200-211
    DOI: 10.1287/opre.2016.1550
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/opre.2016.1550
    Download Restriction: no

    File URL: https://libkey.io/10.1287/opre.2016.1550?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Thomson, William, 1979. "Eliciting production possibilities from a well-informed manager," Journal of Economic Theory, Elsevier, vol. 20(3), pages 360-380, June.
    2. Kriti Jain & Kanchan Mukherjee & J. Neil Bearden & Anil Gaba, 2013. "Unpacking the Future: A Nudge Toward Wider Subjective Confidence Intervals," Management Science, INFORMS, vol. 59(9), pages 1970-1987, September.
    3. De Bondt, Werner F M & Thaler, Richard H, 1990. "Do Security Analysts Overreact?," American Economic Review, American Economic Association, vol. 80(2), pages 52-57, May.
    4. Koenker,Roger, 2005. "Quantile Regression," Cambridge Books, Cambridge University Press, number 9780521608275.
    5. Hyndman, Rob J. & Koehler, Anne B., 2006. "Another look at measures of forecast accuracy," International Journal of Forecasting, Elsevier, vol. 22(4), pages 679-688.
    6. Chatfield, Chris, 1988. "Apples, oranges and mean square error," International Journal of Forecasting, Elsevier, vol. 4(4), pages 515-518.
    7. Robert Fildes & Paul Goodwin, 2007. "Against Your Better Judgment? How Organizations Can Improve Their Use of Management Judgment in Forecasting," Interfaces, INFORMS, vol. 37(6), pages 570-576, December.
    8. Park, Heungsun & Stefanski, L. A., 1998. "Relative-error prediction," Statistics & Probability Letters, Elsevier, vol. 40(3), pages 227-236, October.
    9. Clemon, Robert T & Winkler, Robert L, 1986. "Combining Economic Forecasts," Journal of Business & Economic Statistics, American Statistical Association, vol. 4(1), pages 39-46, January.
    10. Zellner, Arnold, 1986. "A tale of forecasting 1001 series : The Bayesian knight strikes again," International Journal of Forecasting, Elsevier, vol. 2(4), pages 491-494.
    11. Clemen, Robert T., 1989. "Combining forecasts: A review and annotated bibliography," International Journal of Forecasting, Elsevier, vol. 5(4), pages 559-583.
    12. Donna F. Davis & John T. Mentzer & Teresa M. Mccarthy & Susan L. Golicic, 2006. "The evolution of sales forecasting management: a 20-year longitudinal study of forecasting practices," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 25(5), pages 303-324.
    13. Chen, Kani & Guo, Shaojun & Lin, Yuanyuan & Ying, Zhiliang, 2010. "Least Absolute Relative Error Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 105(491), pages 1104-1112.
    14. Patton, Andrew J., 2011. "Volatility forecast comparison using imperfect volatility proxies," Journal of Econometrics, Elsevier, vol. 160(1), pages 246-256, January.
    15. Victor Richmond R. Jose & Yael Grushka-Cockayne & Kenneth C. Lichtendahl, 2014. "Trimmed Opinion Pools and the Crowd's Calibration Problem," Management Science, INFORMS, vol. 60(2), pages 463-475, February.
    16. Victor Richmond R. Jose & Robert L. Winkler, 2009. "Evaluating Quantile Assessments," Operations Research, INFORMS, vol. 57(5), pages 1287-1297, October.
    17. Rob J. Hyndman, 2006. "Another Look at Forecast Accuracy Metrics for Intermittent Demand," Foresight: The International Journal of Applied Forecasting, International Institute of Forecasters, issue 4, pages 43-46, June.
    18. Davydenko, Andrey & Fildes, Robert, 2013. "Measuring forecasting accuracy: The case of judgmental adjustments to SKU-level demand forecasts," International Journal of Forecasting, Elsevier, vol. 29(3), pages 510-522.
    19. Koenker,Roger, 2005. "Quantile Regression," Cambridge Books, Cambridge University Press, number 9780521845731.
    20. John C. Easterwood & Stacey R. Nutt, 1999. "Inefficiency in Analysts' Earnings Forecasts: Systematic Misreaction or Systematic Optimism?," Journal of Finance, American Finance Association, vol. 54(5), pages 1777-1797, October.
    21. Jose, Victor Richmond R. & Winkler, Robert L., 2008. "Simple robust averages of forecasts: Some empirical results," International Journal of Forecasting, Elsevier, vol. 24(1), pages 163-169.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Victor Richmond R. Jose, 2017. "Percentage and Relative Error Measures in Forecast Evaluation," Operations Research, INFORMS, vol. 65(1), pages 200-211, February.
    2. Gneiting, Tilmann, 2011. "Making and Evaluating Point Forecasts," Journal of the American Statistical Association, American Statistical Association, vol. 106(494), pages 746-762.
    3. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    4. Davydenko, Andrey & Fildes, Robert, 2013. "Measuring forecasting accuracy: The case of judgmental adjustments to SKU-level demand forecasts," International Journal of Forecasting, Elsevier, vol. 29(3), pages 510-522.
    5. Perera, H. Niles & Hurley, Jason & Fahimnia, Behnam & Reisi, Mohsen, 2019. "The human factor in supply chain forecasting: A systematic review," European Journal of Operational Research, Elsevier, vol. 274(2), pages 574-600.
    6. Bruzda, Joanna, 2019. "Quantile smoothing in supply chain and logistics forecasting," International Journal of Production Economics, Elsevier, vol. 208(C), pages 122-139.
    7. Wang, Xiaoqian & Hyndman, Rob J. & Li, Feng & Kang, Yanfei, 2023. "Forecast combinations: An over 50-year review," International Journal of Forecasting, Elsevier, vol. 39(4), pages 1518-1547.
    8. Makridakis, Spyros & Spiliotis, Evangelos & Assimakopoulos, Vassilios, 2020. "The M4 Competition: 100,000 time series and 61 forecasting methods," International Journal of Forecasting, Elsevier, vol. 36(1), pages 54-74.
    9. Fildes, Robert & Ma, Shaohui & Kolassa, Stephan, 2022. "Retail forecasting: Research and practice," International Journal of Forecasting, Elsevier, vol. 38(4), pages 1283-1318.
    10. Weron, Rafał, 2014. "Electricity price forecasting: A review of the state-of-the-art with a look into the future," International Journal of Forecasting, Elsevier, vol. 30(4), pages 1030-1081.
    11. Anil Gaba & Ilia Tsetlin & Robert L. Winkler, 2017. "Combining Interval Forecasts," Decision Analysis, INFORMS, vol. 14(1), pages 1-20, March.
    12. Gneiting, Tilmann, 2011. "Quantiles as optimal point forecasts," International Journal of Forecasting, Elsevier, vol. 27(2), pages 197-207, April.
    13. Natalia Nolde & Johanna F. Ziegel, 2016. "Elicitability and backtesting: Perspectives for banking regulation," Papers 1608.05498, arXiv.org, revised Feb 2017.
    14. Gneiting, Tilmann, 2011. "Quantiles as optimal point forecasts," International Journal of Forecasting, Elsevier, vol. 27(2), pages 197-207.
    15. Petropoulos, Fotios & Fildes, Robert & Goodwin, Paul, 2016. "Do ‘big losses’ in judgmental adjustments to statistical forecasts affect experts’ behaviour?," European Journal of Operational Research, Elsevier, vol. 249(3), pages 842-852.
    16. Robert L. Winkler & Yael Grushka-Cockayne & Kenneth C. Lichtendahl Jr. & Victor Richmond R. Jose, 2019. "Probability Forecasts and Their Combination: A Research Perspective," Decision Analysis, INFORMS, vol. 16(4), pages 239-260, December.
    17. Armstrong, J. Scott & Green, Kesten C. & Graefe, Andreas, 2015. "Golden rule of forecasting: Be conservative," Journal of Business Research, Elsevier, vol. 68(8), pages 1717-1731.
    18. Kourentzes, Nikolaos & Petropoulos, Fotios & Trapero, Juan R., 2014. "Improving forecasting by estimating time series structural components across multiple frequencies," International Journal of Forecasting, Elsevier, vol. 30(2), pages 291-302.
    19. Leitner, Johannes & Leopold-Wildburger, Ulrike, 2011. "Experiments on forecasting behavior with several sources of information - A review of the literature," European Journal of Operational Research, Elsevier, vol. 213(3), pages 459-469, September.
    20. Li, Li & Kang, Yanfei & Li, Feng, 2023. "Bayesian forecast combination using time-varying features," International Journal of Forecasting, Elsevier, vol. 39(3), pages 1287-1302.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:oropre:v:65:y:2017:i:1:p:200-211. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.