IDEAS home Printed from https://ideas.repec.org/a/eee/jbfina/v35y2011i3p698-708.html
   My bibliography  Save this article

Pitfalls and remedies in testing the calibration quality of rating systems

Author

Listed:
  • Aussenegg, Wolfgang
  • Resch, Florian
  • Winkler, Gerhard

Abstract

Testing calibration quality by means of backtesting is an integral part in the validation of credit rating systems. Against this background this paper provides a comprehensive overview of existing testing procedures. We study the procedures' deficiencies theoretically and illustrate their impact empirically. Based on the insights gained therefrom, we develop enhanced hybrid testing procedures which turn out to be superior to the commonly applied methods. We also propose computationally efficient algorithms for our calibration tests. Finally, we are able to demonstrate empirically that our method outperforms existing tests in a scenario analysis using rating data of Moody's.

Suggested Citation

  • Aussenegg, Wolfgang & Resch, Florian & Winkler, Gerhard, 2011. "Pitfalls and remedies in testing the calibration quality of rating systems," Journal of Banking & Finance, Elsevier, vol. 35(3), pages 698-708, March.
    • Handle: RePEc:eee:jbfina:v:35:y:2011:i:3:p:698-708
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0378-4266(10)00445-0
    Download Restriction: Full text for ScienceDirect subscribers only
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Kerkhof, Jeroen & Melenberg, Bertrand & Schumacher, Hans, 2010. "Model risk and capital reserves," Journal of Banking & Finance, Elsevier, vol. 34(1), pages 267-279, January.
    2. Tarashev, Nikola, 2010. "Measuring portfolio credit risk correctly: Why parameter uncertainty matters," Journal of Banking & Finance, Elsevier, vol. 34(9), pages 2065-2076, September.
    3. Marwan Elkhoury, 2007. "Credit Rating Agencies And Their Potential Impact On Developing Countries," UNCTAD Discussion Papers 186, United Nations Conference on Trade and Development.
    4. Khandani, Amir E. & Kim, Adlar J. & Lo, Andrew W., 2010. "Consumer credit-risk models via machine-learning algorithms," Journal of Banking & Finance, Elsevier, vol. 34(11), pages 2767-2787, November.
    5. Krahnen, Jan Pieter & Weber, Martin, 2001. "Generally accepted rating principles: A primer," Journal of Banking & Finance, Elsevier, vol. 25(1), pages 3-23, January.
    6. Rosen, Dan & Saunders, David, 2010. "Risk factor contributions in portfolio credit risk models," Journal of Banking & Finance, Elsevier, vol. 34(2), pages 336-349, February.
    7. Lando, David & Skodeberg, Torben M., 2002. "Analyzing rating transitions and rating drift with continuous observations," Journal of Banking & Finance, Elsevier, vol. 26(2-3), pages 423-444, March.
    8. Medema, Lydian & Koning, Ruud H. & Lensink, Robert, 2009. "A practical approach to validating a PD model," Journal of Banking & Finance, Elsevier, vol. 33(4), pages 701-708, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Wosnitza, Jan Henrik, 2022. "Calibration alternatives to logistic regression and their potential for transferring the dispersion of discriminatory power into uncertainties of probabilities of default," Discussion Papers 04/2022, Deutsche Bundesbank.
    2. António Antunes & Homero Gonçalves & Pedro Prego, 2017. "Firm default probabilities revisited," IFC Bulletins chapters, in: Bank for International Settlements (ed.), Uses of central balance sheet data offices' information, volume 45, Bank for International Settlements.
    3. Sauer, Stephan & Coppens, François & Mayer, Manuel & Millischer, Laurent & Resch, Florian & Schulze, Klaas, 2016. "Advances in multivariate back-testing for credit risk underestimation," Working Paper Series 1885, European Central Bank.
    4. Patrick Kurth & Max Nendel & Jan Streicher, 2024. "A Hypothesis Test for the Long-Term Calibration in Rating Systems with Overlapping Time Windows," Risks, MDPI, vol. 12(8), pages 1-28, August.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Andrés Alonso Robisco & José Manuel Carbó Martínez, 2022. "Measuring the model risk-adjusted performance of machine learning algorithms in credit default prediction," Financial Innovation, Springer;Southwestern University of Finance and Economics, vol. 8(1), pages 1-35, December.
    2. Valerio Vacca, 2011. "An unexpected crisis? Looking at pricing effectiveness of different banks," Temi di discussione (Economic working papers) 814, Bank of Italy, Economic Research and International Relations Area.
    3. François Coppens & Fernando Gonzáles & Gerhard Winkler, 2007. "The performance of credit rating systems in the assessment of collateral used in Eurosystem monetary policy operations," Working Paper Research 118, National Bank of Belgium.
    4. Tunaru, Radu & Zheng, Teng, 2017. "Parameter estimation risk in asset pricing and risk management: A Bayesian approach," International Review of Financial Analysis, Elsevier, vol. 53(C), pages 80-93.
    5. Tarashev, Nikola, 2010. "Measuring portfolio credit risk correctly: Why parameter uncertainty matters," Journal of Banking & Finance, Elsevier, vol. 34(9), pages 2065-2076, September.
    6. Claußen, Arndt & Rösch, Daniel & Schmelzle, Martin, 2019. "Hedging parameter risk," Journal of Banking & Finance, Elsevier, vol. 100(C), pages 111-121.
    7. Fitzpatrick, Trevor & Mues, Christophe, 2016. "An empirical comparison of classification algorithms for mortgage default prediction: evidence from a distressed mortgage market," European Journal of Operational Research, Elsevier, vol. 249(2), pages 427-439.
    8. Radu Tunaru, 2015. "Model Risk in Financial Markets:From Financial Engineering to Risk Management," World Scientific Books, World Scientific Publishing Co. Pte. Ltd., number 9524, August.
    9. Jankowitsch, Rainer & Pichler, Stefan & Schwaiger, Walter S.A., 2007. "Modelling the economic value of credit rating systems," Journal of Banking & Finance, Elsevier, vol. 31(1), pages 181-198, January.
    10. Loffler, Gunter, 2005. "Avoiding the rating bounce: why rating agencies are slow to react to new information," Journal of Economic Behavior & Organization, Elsevier, vol. 56(3), pages 365-381, March.
    11. Marcus C. Christiansen, 2021. "Time-dynamic evaluations under non-monotone information generated by marked point processes," Finance and Stochastics, Springer, vol. 25(3), pages 563-596, July.
    12. Loffler, Gunter, 2004. "An anatomy of rating through the cycle," Journal of Banking & Finance, Elsevier, vol. 28(3), pages 695-720, March.
    13. Valeriane Jokhadze & Wolfgang M. Schmidt, 2020. "Measuring Model Risk In Financial Risk Management And Pricing," International Journal of Theoretical and Applied Finance (IJTAF), World Scientific Publishing Co. Pte. Ltd., vol. 23(02), pages 1-37, April.
    14. Giovanni Butera & Robert Faff, 2006. "An integrated multi-model credit rating system for private firms," Review of Quantitative Finance and Accounting, Springer, vol. 27(3), pages 311-340, November.
    15. Pflug, Georg Ch. & Pichler, Alois & Wozabal, David, 2012. "The 1/N investment strategy is optimal under high model ambiguity," Journal of Banking & Finance, Elsevier, vol. 36(2), pages 410-417.
    16. Nguyen, Thu Ha & Lan, Yihui & Treepongkaruna, Sirimon & Zhong, Rui, 2023. "Credit rating downgrades and stock price crash risk: International evidence," Finance Research Letters, Elsevier, vol. 55(PB).
    17. Wosnitza, Jan Henrik, 2022. "Calibration alternatives to logistic regression and their potential for transferring the dispersion of discriminatory power into uncertainties of probabilities of default," Discussion Papers 04/2022, Deutsche Bundesbank.
    18. Simon Cornée, 2014. "Soft Information and Default Prediction in Cooperative and Social Banks," Journal of Entrepreneurial and Organizational Diversity, European Research Institute on Cooperative and Social Enterprises, vol. 3(1), pages 89-103, June.
    19. Pesaran, M. Hashem & Schuermann, Til & Treutler, Bjorn-Jakob & Weiner, Scott M., 2006. "Macroeconomic Dynamics and Credit Risk: A Global Perspective," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 38(5), pages 1211-1261, August.
    20. Lucian Belascu & Alexandra Horobet & Georgiana Vrinceanu & Consuela Popescu, 2021. "Performance Dissimilarities in European Union Manufacturing: The Effect of Ownership and Technological Intensity," Sustainability, MDPI, vol. 13(18), pages 1-19, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:jbfina:v:35:y:2011:i:3:p:698-708. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/jbf .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.