IDEAS home Printed from https://ideas.repec.org/a/eee/intfor/v39y2023i3p1448-1459.html
   My bibliography  Save this article

Evaluating probabilistic forecasts of extremes using continuous ranked probability score distributions

Author

Listed:
  • Taillardat, Maxime
  • Fougères, Anne-Laure
  • Naveau, Philippe
  • de Fondeville, Raphaël

Abstract

Verifying probabilistic forecasts for extreme events is a highly active research area because popular media and public opinions are naturally focused on extreme events, and biased conclusions are readily made. In this context, classical verification methods tailored for extreme events, such as thresholded and weighted scoring rules, have undesirable properties that cannot be mitigated, and the well-known continuous ranked probability score (CRPS) is no exception.

Suggested Citation

  • Taillardat, Maxime & Fougères, Anne-Laure & Naveau, Philippe & de Fondeville, Raphaël, 2023. "Evaluating probabilistic forecasts of extremes using continuous ranked probability score distributions," International Journal of Forecasting, Elsevier, vol. 39(3), pages 1448-1459.
    • Handle: RePEc:eee:intfor:v:39:y:2023:i:3:p:1448-1459
    DOI: 10.1016/j.ijforecast.2022.07.003
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0169207022001017
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ijforecast.2022.07.003?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Werner Ehm & Tilmann Gneiting & Alexander Jordan & Fabian Krüger, 2016. "Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 78(3), pages 505-562, June.
    2. John W. Galbraith & Simon van Norden, 2012. "Assessing gross domestic product and inflation probability forecasts derived from Bank of England fan charts," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 175(3), pages 713-727, July.
    3. Petra Friederichs & Thordis L. Thorarinsdottir, 2012. "Forecast verification for extreme value distributions with an application to probabilistic peak wind prediction," Environmetrics, John Wiley & Sons, Ltd., vol. 23(7), pages 579-594, November.
    4. Francis X. Diebold & Todd A. Gunther & Anthony S. Tay, "undated". "Evaluating Density Forecasts," CARESS Working Papres 97-18, University of Pennsylvania Center for Analytic Research and Economics in the Social Sciences.
    5. Hong, Tao & Pinson, Pierre & Fan, Shu & Zareipour, Hamidreza & Troccoli, Alberto & Hyndman, Rob J., 2016. "Probabilistic energy forecasting: Global Energy Forecasting Competition 2014 and beyond," International Journal of Forecasting, Elsevier, vol. 32(3), pages 896-913.
    6. Tilmann Gneiting & Roopesh Ranjan, 2011. "Comparing Density Forecasts Using Threshold- and Quantile-Weighted Scoring Rules," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 29(3), pages 411-422, July.
    7. R. Winkler & Javier Muñoz & José Cervera & José Bernardo & Gail Blattenberger & Joseph Kadane & Dennis Lindley & Allan Murphy & Robert Oliver & David Ríos-Insua, 1996. "Scoring rules and the evaluation of probabilities," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 5(1), pages 1-60, June.
    8. Leonard Smith & Emma Suckling & Erica Thompson & Trevor Maynard & Hailiang Du, 2015. "Towards improving the framework for probabilistic forecast evaluation," Climatic Change, Springer, vol. 132(1), pages 31-45, September.
    9. Tsyplakov, Alexander, 2011. "Evaluating density forecasts: a comment," MPRA Paper 31184, University Library of Munich, Germany.
    10. Alexander Henzi & Gian-Reto Kleger & Matthias P Hilty & Pedro D Wendel Garcia & Johanna F Ziegel & on behalf of RISC-19-ICU Investigators for Switzerland, 2021. "Probabilistic analysis of COVID-19 patients’ individual length of stay in Swiss intensive care units," PLOS ONE, Public Library of Science, vol. 16(2), pages 1-14, February.
    11. Diks, Cees & Panchenko, Valentyn & van Dijk, Dick, 2011. "Likelihood-based scoring rules for comparing density forecasts in tails," Journal of Econometrics, Elsevier, vol. 163(2), pages 215-230, August.
    12. Tilmann Gneiting & Fadoua Balabdaoui & Adrian E. Raftery, 2007. "Probabilistic forecasts, calibration and sharpness," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 69(2), pages 243-268, April.
    13. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    14. Ghosh, Souvik & Resnick, Sidney, 2010. "A discussion on mean excess plots," Stochastic Processes and their Applications, Elsevier, vol. 120(8), pages 1492-1517, August.
    15. Gneiting, Tilmann & Ranjan, Roopesh, 2011. "Comparing Density Forecasts Using Threshold- and Quantile-Weighted Scoring Rules," Journal of Business & Economic Statistics, American Statistical Association, vol. 29(3), pages 411-422.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Allen, Sam & Koh, Jonathan & Segers, Johan & Ziegel, Johanna, 2024. "Tail calibration of probabilistic forecasts," LIDAM Discussion Papers ISBA 2024018, Université catholique de Louvain, Institute of Statistics, Biostatistics and Actuarial Sciences (ISBA).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Marc-Oliver Pohle, 2020. "The Murphy Decomposition and the Calibration-Resolution Principle: A New Perspective on Forecast Evaluation," Papers 2005.01835, arXiv.org.
    2. Ruben Loaiza‐Maya & Gael M. Martin & David T. Frazier, 2021. "Focused Bayesian prediction," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 36(5), pages 517-543, August.
    3. Jonas R. Brehmer & Tilmann Gneiting, 2020. "Properization: constructing proper scoring rules via Bayes acts," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 72(3), pages 659-673, June.
    4. Allen, Sam & Koh, Jonathan & Segers, Johan & Ziegel, Johanna, 2024. "Tail calibration of probabilistic forecasts," LIDAM Discussion Papers ISBA 2024018, Université catholique de Louvain, Institute of Statistics, Biostatistics and Actuarial Sciences (ISBA).
    5. C. Alexander & M. Coulon & Y. Han & X. Meng, 2024. "Evaluating the discrimination ability of proper multi-variate scoring rules," Annals of Operations Research, Springer, vol. 334(1), pages 857-883, March.
    6. Luisa Bisaglia & Matteo Grigoletto, 2021. "A new time-varying model for forecasting long-memory series," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 30(1), pages 139-155, March.
    7. Hajo Holzmann & Matthias Eulert, 2014. "The role of the information set for forecasting - with applications to risk management," Papers 1404.7653, arXiv.org.
    8. Elena Andreou & Andros Kourtellos, 2018. "Scoring rules for simple forecasting models: The case of Cyprus GDP and its sectors," Cyprus Economic Policy Review, University of Cyprus, Economics Research Centre, vol. 12(1), pages 59-73, June.
    9. Yael Grushka-Cockayne & Kenneth C. Lichtendahl Jr. & Victor Richmond R. Jose & Robert L. Winkler, 2017. "Quantile Evaluation, Sensitivity to Bracketing, and Sharing Business Payoffs," Operations Research, INFORMS, vol. 65(3), pages 712-728, June.
    10. Alexander Henzi & Johanna F. Ziegel & Tilmann Gneiting, 2021. "Isotonic distributional regression," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 83(5), pages 963-993, November.
    11. Gensler, André & Sick, Bernhard & Vogt, Stephan, 2018. "A review of uncertainty representations and metaverification of uncertainty assessment techniques for renewable energies," Renewable and Sustainable Energy Reviews, Elsevier, vol. 96(C), pages 352-379.
    12. Xiaochun Meng & James W. Taylor & Souhaib Ben Taieb & Siran Li, 2020. "Scores for Multivariate Distributions and Level Sets," Papers 2002.09578, arXiv.org, revised Jun 2023.
    13. Tobias Fissler & Hajo Holzmann, 2022. "Measurability of functionals and of ideal point forecasts," Papers 2203.08635, arXiv.org.
    14. Onno Kleen, 2024. "Scaling and measurement error sensitivity of scoring rules for distribution forecasts," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 39(5), pages 833-849, August.
    15. Luisa Bisaglia & Matteo Grigoletto, 2018. "A new time-varying model for forecasting long-memory series," Papers 1812.07295, arXiv.org.
    16. Roberto Casarin & Stefano Grassi & Francesco Ravazzolo & Herman K. van Dijk, 2020. "A Bayesian Dynamic Compositional Model for Large Density Combinations in Finance," Working Paper series 20-27, Rimini Centre for Economic Analysis.
    17. Chris McDonald & Craig Thamotheram & Shaun P. Vahey & Elizabeth C. Wakerly, 2016. "Assessing the economic value of probabilistic forecasts in the presence of an inflation target," Reserve Bank of New Zealand Discussion Paper Series DP2016/10, Reserve Bank of New Zealand.
    18. Emilio Zanetti Chini, 2018. "Forecasters’ utility and forecast coherence," CREATES Research Papers 2018-23, Department of Economics and Business Economics, Aarhus University.
    19. Sándor Baran & Patrícia Szokol & Marianna Szabó, 2021. "Truncated generalized extreme value distribution‐based ensemble model output statistics model for calibration of wind speed ensemble forecasts," Environmetrics, John Wiley & Sons, Ltd., vol. 32(6), September.
    20. repec:wrk:wrkemf:09 is not listed on IDEAS
    21. Matteo Malavasi & Gareth W. Peters & Stefan Treuck & Pavel V. Shevchenko & Jiwook Jang & Georgy Sofronov, 2024. "Cyber Risk Taxonomies: Statistical Analysis of Cybersecurity Risk Classifications," Papers 2410.05297, arXiv.org.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:intfor:v:39:y:2023:i:3:p:1448-1459. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/ijforecast .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.