IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2104.10673.html
   My bibliography  Save this paper

Backtesting Systemic Risk Forecasts using Multi-Objective Elicitability

Author

Listed:
  • Tobias Fissler
  • Yannick Hoga

Abstract

Systemic risk measures such as CoVaR, CoES and MES are widely-used in finance, macroeconomics and by regulatory bodies. Despite their importance, we show that they fail to be elicitable and identifiable. This renders forecast comparison and validation, commonly summarised as `backtesting', impossible. The novel notion of \emph{multi-objective elicitability} solves this problem. Specifically, we propose Diebold--Mariano type tests utilising two-dimensional scores equipped with the lexicographic order. We illustrate the test decisions by an easy-to-apply traffic-light approach. We apply our traffic-light approach to DAX~30 and S\&P~500 returns, and infer some recommendations for regulators.

Suggested Citation

  • Tobias Fissler & Yannick Hoga, 2021. "Backtesting Systemic Risk Forecasts using Multi-Objective Elicitability," Papers 2104.10673, arXiv.org, revised Feb 2022.
    • Handle: RePEc:arx:papers:2104.10673
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2104.10673
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Natalia Nolde & Johanna F. Ziegel, 2016. "Elicitability and backtesting: Perspectives for banking regulation," Papers 1608.05498, arXiv.org, revised Feb 2017.
    2. Gneiting, Tilmann, 2011. "Making and Evaluating Point Forecasts," Journal of the American Statistical Association, American Statistical Association, vol. 106(494), pages 746-762.
    3. Raffaella Giacomini & Halbert White, 2006. "Tests of Conditional Predictive Ability," Econometrica, Econometric Society, vol. 74(6), pages 1545-1578, November.
    4. Denisa Banulescu-Radu & Christophe Hurlin & Jérémy Leymarie & Olivier Scaillet, 2021. "Backtesting Marginal Expected Shortfall and Related Systemic Risk Measures," Management Science, INFORMS, vol. 67(9), pages 5730-5754, September.
    5. Yannick Hoga, 2019. "Confidence Intervals for Conditional Tail Risk Measures in ARMA–GARCH Models," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 37(4), pages 613-624, October.
    6. Viral V. Acharya & Lasse H. Pedersen & Thomas Philippon & Matthew Richardson, 2017. "Measuring Systemic Risk," The Review of Financial Studies, Society for Financial Studies, vol. 30(1), pages 2-47.
    7. Werner Ehm & Tilmann Gneiting & Alexander Jordan & Fabian Krüger, 2016. "Of quantiles and expectiles: consistent scoring functions, Choquet representations and forecast rankings," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 78(3), pages 505-562, June.
    8. Davidson, James, 1994. "Stochastic Limit Theory: An Introduction for Econometricians," OUP Catalogue, Oxford University Press, number 9780198774037.
    9. Rafael Frongillo & Ian A. Kash, 2015. "Elicitation Complexity of Statistical Properties," Papers 1506.07212, arXiv.org, revised Aug 2020.
    10. West, Kenneth D, 1996. "Asymptotic Inference about Predictive Ability," Econometrica, Econometric Society, vol. 64(5), pages 1067-1084, September.
    11. Christian Brownlees & Robert F. Engle, 2017. "SRISK: A Conditional Capital Shortfall Measure of Systemic Risk," The Review of Financial Studies, Society for Financial Studies, vol. 30(1), pages 48-79.
    12. Alexander J. McNeil & Rüdiger Frey & Paul Embrechts, 2015. "Quantitative Risk Management: Concepts, Techniques and Tools Revised edition," Economics Books, Princeton University Press, edition 2, number 10496.
    13. Ang, Andrew & Bekaert, Geert & Wei, Min, 2007. "Do macro variables, asset markets, or surveys forecast inflation better?," Journal of Monetary Economics, Elsevier, vol. 54(4), pages 1163-1212, May.
    14. Raffaella Giacomini & Barbara Rossi, 2010. "Forecast comparisons in unstable environments," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(4), pages 595-620.
    15. E. Kromer & L. Overbeck & K. Zilch, 2016. "Systemic risk measures on general measurable spaces," Mathematical Methods of Operations Research, Springer;Gesellschaft für Operations Research (GOR);Nederlands Genootschap voor Besliskunde (NGB), vol. 84(2), pages 323-357, October.
    16. David Ardia & Kris Boudt & Leopoldo Catania, 2016. "Generalized Autoregressive Score Models in R: The GAS Package," Papers 1609.02354, arXiv.org.
    17. Diebold, Francis X & Mariano, Roberto S, 2002. "Comparing Predictive Accuracy," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(1), pages 134-144, January.
    18. Robert M. De Jong & James Davidson, 2000. "Consistency of Kernel Estimators of Heteroscedastic and Autocorrelated Covariance Matrices," Econometrica, Econometric Society, vol. 68(2), pages 407-424, March.
    19. Philippe Artzner & Freddy Delbaen & Jean‐Marc Eber & David Heath, 1999. "Coherent Measures of Risk," Mathematical Finance, Wiley Blackwell, vol. 9(3), pages 203-228, July.
    20. Glosten, Lawrence R & Jagannathan, Ravi & Runkle, David E, 1993. "On the Relation between the Expected Value and the Volatility of the Nominal Excess Return on Stocks," Journal of Finance, American Finance Association, vol. 48(5), pages 1779-1801, December.
    21. Frederic S. Mishkin, 2011. "Over the Cliff: From the Subprime to the Global Financial Crisis," Journal of Economic Perspectives, American Economic Association, vol. 25(1), pages 49-70, Winter.
    22. Patton, Andrew J. & Ziegel, Johanna F. & Chen, Rui, 2019. "Dynamic semiparametric models for expected shortfall (and Value-at-Risk)," Journal of Econometrics, Elsevier, vol. 211(2), pages 388-413.
    23. Mas-Colell, Andreu & Whinston, Michael D. & Green, Jerry R., 1995. "Microeconomic Theory," OUP Catalogue, Oxford University Press, number 9780195102680.
    24. Yannick Hoga, 2017. "Testing for changes in (extreme) VaR," Econometrics Journal, Royal Economic Society, vol. 20(1), pages 23-51, February.
    25. Laura Coroneo & Fabrizio Iacone, 2020. "Comparing predictive accuracy in small samples using fixed‐smoothing asymptotics," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 35(4), pages 391-409, June.
    26. Tobias Eckernkemper, 2018. "Modeling Systemic Risk: Time-Varying Tail Dependence When Forecasting Marginal Expected Shortfall," Journal of Financial Econometrics, Oxford University Press, vol. 16(1), pages 63-117.
    27. De Lira Salvatierra, Irving & Patton, Andrew J., 2015. "Dynamic copula models and high frequency data," Journal of Empirical Finance, Elsevier, vol. 30(C), pages 120-135.
    28. Gneiting, Tilmann, 2011. "Quantiles as optimal point forecasts," International Journal of Forecasting, Elsevier, vol. 27(2), pages 197-207, April.
    29. Timo Dimitriadis & iaochun Liu & Julie Schnaitmann, 2023. "Encompassing Tests for Value at Risk and Expected Shortfall Multistep Forecasts Based on Inference on the Boundary," Journal of Financial Econometrics, Oxford University Press, vol. 21(2), pages 412-444.
    30. Mauro Bernardi & Leopoldo Catania, 2019. "Switching generalized autoregressive score copula models with application to systemic risk," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 34(1), pages 43-65, January.
    31. Gneiting, Tilmann, 2011. "Quantiles as optimal point forecasts," International Journal of Forecasting, Elsevier, vol. 27(2), pages 197-207.
    32. Andrew J. Patton, 2020. "Comparing Possibly Misspecified Forecasts," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 38(4), pages 796-809, October.
    33. Girardi, Giulio & Tolga Ergün, A., 2013. "Systemic risk measurement: Multivariate GARCH estimation of CoVaR," Journal of Banking & Finance, Elsevier, vol. 37(8), pages 3169-3180.
    34. Drew Creal & Siem Jan Koopman & André Lucas, 2013. "Generalized Autoregressive Score Models With Applications," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 28(5), pages 777-795, August.
    35. Natalia Nolde & Jinyuan Zhang, 2020. "Conditional Extremes in Asymmetric Financial Markets," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 38(1), pages 201-213, January.
    36. Hoffmann, Hannes & Meyer-Brandis, Thilo & Svindland, Gregor, 2016. "Risk-consistent conditional systemic risk measures," Stochastic Processes and their Applications, Elsevier, vol. 126(7), pages 2014-2037.
    37. Fabio Bellini & Valeria Bignozzi, 2015. "On elicitable risk measures," Quantitative Finance, Taylor & Francis Journals, vol. 15(5), pages 725-733, May.
    38. Luca Trapin, 2018. "Can Volatility Models Explain Extreme Events?," Journal of Financial Econometrics, Oxford University Press, vol. 16(2), pages 297-315.
    39. Francesca Biagini & Jean‐Pierre Fouque & Marco Frittelli & Thilo Meyer‐Brandis, 2019. "A unified approach to systemic risk measures via acceptance sets," Mathematical Finance, Wiley Blackwell, vol. 29(1), pages 329-367, January.
    40. James W. Taylor, 2019. "Forecasting Value at Risk and Expected Shortfall Using a Semiparametric Approach Based on the Asymmetric Laplace Distribution," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 37(1), pages 121-133, January.
    41. Susanne Emmer & Marie Kratz & Dirk Tasche, 2013. "What is the best risk measure in practice? A comparison of standard measures," Papers 1312.1645, arXiv.org, revised Apr 2015.
    42. Marc-Oliver Pohle, 2020. "The Murphy Decomposition and the Calibration-Resolution Principle: A New Perspective on Forecast Evaluation," Papers 2005.01835, arXiv.org.
    43. Philip R. Lane, 2012. "The European Sovereign Debt Crisis," Journal of Economic Perspectives, American Economic Association, vol. 26(3), pages 49-68, Summer.
    44. Matthias Ehrgott, 2005. "Multicriteria Optimization," Springer Books, Springer, edition 0, number 978-3-540-27659-3, March.
    45. Hannes Hoffmann & Thilo Meyer-Brandis & Gregor Svindland, 2016. "Risk-Consistent Conditional Systemic Risk Measures," Papers 1609.07897, arXiv.org.
    46. Hajo Holzmann & Matthias Eulert, 2014. "The role of the information set for forecasting - with applications to risk management," Papers 1404.7653, arXiv.org.
    47. Chen Chen & Garud Iyengar & Ciamac C. Moallemi, 2013. "An Axiomatic Approach to Systemic Risk," Management Science, INFORMS, vol. 59(6), pages 1373-1388, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Tobias Fissler & Hajo Holzmann, 2022. "Measurability of functionals and of ideal point forecasts," Papers 2203.08635, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tobias Fissler & Jana Hlavinová & Birgit Rudloff, 2021. "Elicitability and identifiability of set-valued measures of systemic risk," Finance and Stochastics, Springer, vol. 25(1), pages 133-165, January.
    2. Timo Dimitriadis & iaochun Liu & Julie Schnaitmann, 2023. "Encompassing Tests for Value at Risk and Expected Shortfall Multistep Forecasts Based on Inference on the Boundary," Journal of Financial Econometrics, Oxford University Press, vol. 21(2), pages 412-444.
    3. Timo Dimitriadis & Yannick Hoga, 2022. "Dynamic CoVaR Modeling and Estimation," Papers 2206.14275, arXiv.org, revised Jan 2025.
    4. Tobias Fissler & Yannick Hoga, 2024. "How to Compare Copula Forecasts?," Papers 2410.04165, arXiv.org.
    5. Dimitriadis, Timo & Schnaitmann, Julie, 2021. "Forecast encompassing tests for the expected shortfall," International Journal of Forecasting, Elsevier, vol. 37(2), pages 604-621.
    6. Fissler Tobias & Ziegel Johanna F., 2021. "On the elicitability of range value at risk," Statistics & Risk Modeling, De Gruyter, vol. 38(1-2), pages 25-46, January.
    7. Timo Dimitriadis & Tobias Fissler & Johanna Ziegel, 2020. "The Efficiency Gap," Papers 2010.14146, arXiv.org, revised Sep 2022.
    8. Tobias Fissler & Fangda Liu & Ruodu Wang & Linxiao Wei, 2024. "Elicitability and identifiability of tail risk measures," Papers 2404.14136, arXiv.org, revised Jun 2024.
    9. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    10. Owusu Junior, Peterson & Tiwari, Aviral Kumar & Tweneboah, George & Asafo-Adjei, Emmanuel, 2022. "GAS and GARCH based value-at-risk modeling of precious metals," Resources Policy, Elsevier, vol. 75(C).
    11. Timo Dimitriadis & Julie Schnaitmann, 2019. "Forecast Encompassing Tests for the Expected Shortfall," Papers 1908.04569, arXiv.org, revised Aug 2020.
    12. Owusu Junior, Peterson & Alagidede, Imhotep, 2020. "Risks in emerging markets equities: Time-varying versus spatial risk analysis," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 542(C).
    13. Takaaki Koike & Cathy W. S. Chen & Edward M. H. Lin, 2024. "Forecasting and Backtesting Gradient Allocations of Expected Shortfall," Papers 2401.11701, arXiv.org, revised Jun 2024.
    14. Hoga, Yannick, 2021. "The uncertainty in extreme risk forecasts from covariate-augmented volatility models," International Journal of Forecasting, Elsevier, vol. 37(2), pages 675-686.
    15. David Happersberger & Harald Lohre & Ingmar Nolte, 2020. "Estimating portfolio risk for tail risk protection strategies," European Financial Management, European Financial Management Association, vol. 26(4), pages 1107-1146, September.
    16. Simon Fritzsch & Maike Timphus & Gregor Weiss, 2021. "Marginals Versus Copulas: Which Account For More Model Risk In Multivariate Risk Forecasting?," Papers 2109.10946, arXiv.org.
    17. Timo Dimitriadis & Yannick Hoga, 2023. "Regressions under Adverse Conditions," Papers 2311.13327, arXiv.org, revised Feb 2025.
    18. Oh, Dong Hwan & Patton, Andrew J., 2024. "Better the devil you know: Improved forecasts from imperfect models," Journal of Econometrics, Elsevier, vol. 242(1).
    19. Fritzsch, Simon & Timphus, Maike & Weiß, Gregor, 2024. "Marginals versus copulas: Which account for more model risk in multivariate risk forecasting?," Journal of Banking & Finance, Elsevier, vol. 158(C).
    20. Dong Hwan Oh & Andrew J. Patton, 2021. "Better the Devil You Know: Improved Forecasts from Imperfect Models," Finance and Economics Discussion Series 2021-071, Board of Governors of the Federal Reserve System (U.S.).

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2104.10673. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.