IDEAS home Printed from https://ideas.repec.org/a/inm/ormnsc/v57y2011i3p487-505.html
   My bibliography  Save this article

A New Goodness-of-Fit Test for Event Forecasting and Its Application to Credit Defaults

Author

Listed:
  • Andreas Blöchlinger

    (Zürcher Kantonalbank, 8010 Zurich, Switzerland)

  • Markus Leippold

    (University of Zurich, 8032 Zurich, Switzerland)

Abstract

We develop a new goodness-of-fit test for validating the performance of probability forecasts. Our test statistic is particularly powerful under sparseness and dependence in the observed data. To build our test statistic, we start from a formal definition of calibrated forecasts, which we operationalize by introducing two components. The first component tests the level of the estimated probabilities; the second validates the shape, measuring the differentiation between high and low probability events. After constructing test statistics for both level and shape, we provide a global goodness-of-fit statistic, which is asymptotically \chi 2 distributed. In a simulation exercise, we find that our approach is correctly sized and more powerful than alternative statistics. In particular, our shape statistic is significantly more powerful than the Kolmogorov-Smirnov test. Under independence, our global test has significantly greater power than the popular Hosmer-Lemeshow's \chi 2 test. Moreover, even under dependence, our global test remains correctly sized and consistent. As a timely and important empirical application of our method, we study the validation of a forecasting model for credit default events. This paper was accepted by Wei Xiong, finance.

Suggested Citation

  • Andreas Blöchlinger & Markus Leippold, 2011. "A New Goodness-of-Fit Test for Event Forecasting and Its Application to Credit Defaults," Management Science, INFORMS, vol. 57(3), pages 487-505, March.
  • Handle: RePEc:inm:ormnsc:v:57:y:2011:i:3:p:487-505
    DOI: 10.1287/mnsc.1100.1283
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/mnsc.1100.1283
    Download Restriction: no

    File URL: https://libkey.io/10.1287/mnsc.1100.1283?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Han Bleichrodt & Jose Luis Pinto, 2000. "A Parameter-Free Elicitation of the Probability Weighting Function in Medical Decision Analysis," Management Science, INFORMS, vol. 46(11), pages 1485-1496, November.
    2. G. Noether, 1963. "Note on the kolmogorov statistic in the discrete case," Metrika: International Journal for Theoretical and Applied Statistics, Springer, vol. 7(1), pages 115-116, December.
    3. William Harbaugh & Kate Krause & Lise Vesterlund, 2002. "Risk Attitudes of Children and Adults: Choices Over Small and Large Probability Gains and Losses," Experimental Economics, Springer;Economic Science Association, vol. 5(1), pages 53-84, June.
    4. Gordy, Michael B., 2003. "A risk-factor model foundation for ratings-based bank capital rules," Journal of Financial Intermediation, Elsevier, vol. 12(3), pages 199-232, July.
    5. Orme, Christopher, 1988. "The Calculation of the Information Matrix Test for Binary Data Models," The Manchester School of Economic & Social Studies, University of Manchester, vol. 56(4), pages 370-376, December.
    6. Donald W. K. Andrews, 2005. "Cross-Section Regression with Common Shocks," Econometrica, Econometric Society, vol. 73(5), pages 1551-1585, September.
    7. White, Halbert, 1982. "Maximum Likelihood Estimation of Misspecified Models," Econometrica, Econometric Society, vol. 50(1), pages 1-25, January.
    8. Sanjiv R. Das & Darrell Duffie & Nikunj Kapadia & Leandro Saita, 2007. "Common Failings: How Corporate Defaults Are Correlated," Journal of Finance, American Finance Association, vol. 62(1), pages 93-117, February.
    9. Archer, Kellie J. & Lemeshow, Stanley & Hosmer, David W., 2007. "Goodness-of-fit tests for logistic regression models when data are collected using a complex sampling design," Computational Statistics & Data Analysis, Elsevier, vol. 51(9), pages 4450-4464, May.
    10. Hanson, Samuel & Schuermann, Til, 2006. "Confidence intervals for probabilities of default," Journal of Banking & Finance, Elsevier, vol. 30(8), pages 2281-2301, August.
    11. Cubitt, Robin P & Starmer, Chris & Sugden, Robert, 1998. "Dynamic Choice and the Common Ratio Effect: An Experimental Investigation," Economic Journal, Royal Economic Society, vol. 108(450), pages 1362-1380, September.
    12. Wei Pan, 2002. "Goodness‐of‐fit Tests for GEE with Correlated Binary Data," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 29(1), pages 101-110, March.
    13. J. B. Copas, 1989. "Unweighted Sum of Squares Test for Proportions," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 38(1), pages 71-80, March.
    14. Gordy, Michael B., 2000. "A comparative anatomy of credit risk models," Journal of Banking & Finance, Elsevier, vol. 24(1-2), pages 119-149, January.
    15. Blochlinger, Andreas & Leippold, Markus, 2006. "Economic benefit of powerful credit scoring," Journal of Banking & Finance, Elsevier, vol. 30(3), pages 851-873, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Zewei Lin & Dungang Liu, 2022. "Model diagnostics of discrete data regression: a unifying framework using functional residuals," Papers 2207.04299, arXiv.org.
    2. Andreas Blöchlinger, 2018. "Credit Rating and Pricing: Poles Apart," JRFM, MDPI, vol. 11(2), pages 1-26, May.
    3. Lahiri, Kajal & Wang, J. George, 2013. "Evaluating probability forecasts for GDP declines using alternative methodologies," International Journal of Forecasting, Elsevier, vol. 29(1), pages 175-190.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hanson, Samuel G. & Pesaran, M. Hashem & Schuermann, Til, 2008. "Firm heterogeneity and credit risk diversification," Journal of Empirical Finance, Elsevier, vol. 15(4), pages 583-612, September.
    2. Nicholas M. Kiefer, 2017. "Correlated defaults, temporal correlation, expert information and predictability of default rates," Econometric Reviews, Taylor & Francis Journals, vol. 36(6-9), pages 699-712, October.
    3. Oliver Blümke, 2020. "Estimating the probability of default for no‐default and low‐default portfolios," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 69(1), pages 89-107, January.
    4. Steffi Höse & Stefan Huschens, 2011. "Confidence Intervals for Asset Correlations in the Asymptotic Single Risk Factor Model," Operations Research Proceedings, in: Bo Hu & Karl Morasch & Stefan Pickl & Markus Siegle (ed.), Operations Research Proceedings 2010, pages 111-116, Springer.
    5. Xie, Xian-Jin & Pendergast, Jane & Clarke, William, 2008. "Increasing the power: A practical approach to goodness-of-fit test for logistic regression models with continuous predictors," Computational Statistics & Data Analysis, Elsevier, vol. 52(5), pages 2703-2713, January.
    6. Bandyopadhyay, Arindam, 2010. "Understanding the Effect of Concentration Risk in the Banks’ Credit Portfolio: Indian Cases," MPRA Paper 24822, University Library of Munich, Germany.
    7. Lee, Yongwoong & Rösch, Daniel & Scheule, Harald, 2021. "Systematic credit risk in securitised mortgage portfolios," Journal of Banking & Finance, Elsevier, vol. 122(C).
    8. Davidson, Russell & MacKinnon, James G., 1989. "Testing for Consistency using Artificial Regressions," Econometric Theory, Cambridge University Press, vol. 5(3), pages 363-384, December.
    9. Andersen, Torben G. & Fusari, Nicola & Todorov, Viktor & Varneskov, Rasmus T., 2019. "Unified inference for nonlinear factor models from panels with fixed and large time span," Journal of Econometrics, Elsevier, vol. 212(1), pages 4-25.
    10. Enrico Diecidue & Peter Wakker & Marcel Zeelenberg, 2007. "Eliciting decision weights by adapting de Finetti’s betting-odds method to prospect theory," Journal of Risk and Uncertainty, Springer, vol. 34(3), pages 179-199, June.
    11. Carole Bernard & Ludger Rüschendorf & Steven Vanduffel & Ruodu Wang, 2017. "Risk bounds for factor models," Finance and Stochastics, Springer, vol. 21(3), pages 631-659, July.
    12. Nicholas M. Kiefer, 2011. "Default estimation, correlated defaults, and expert information," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 26(2), pages 173-192, March.
    13. Rosen, Dan & Saunders, David, 2009. "Analytical methods for hedging systematic credit risk with linear factor portfolios," Journal of Economic Dynamics and Control, Elsevier, vol. 33(1), pages 37-52, January.
    14. Hamilton, James D., 1996. "Specification testing in Markov-switching time-series models," Journal of Econometrics, Elsevier, vol. 70(1), pages 127-157, January.
    15. Orme, Chris, 1995. "Simulated conditional moment tests," Economics Letters, Elsevier, vol. 49(3), pages 239-245, September.
    16. Georges Dionne & Geneviève Gauthier & Khemais Hammami & Mathieu Maurice & Jean‐Guy Simonato, 2010. "Default Risk in Corporate Yield Spreads," Financial Management, Financial Management Association International, vol. 39(2), pages 707-731, June.
    17. repec:gnv:wpaper:unige:76321 is not listed on IDEAS
    18. Marc Gürtler & Dirk Heithecker, 2006. "Modellkonsistente Bestimmung des LGD im IRB-Ansatz von Basel II," Schmalenbach Journal of Business Research, Springer, vol. 58(5), pages 554-587, August.
    19. Gagliardini, Patrick & Gourieroux, Christian, 2014. "Efficiency In Large Dynamic Panel Models With Common Factors," Econometric Theory, Cambridge University Press, vol. 30(5), pages 961-1020, October.
    20. Lechner, Michael & Smith, Jeffrey, 2007. "What is the value added by caseworkers?," Labour Economics, Elsevier, vol. 14(2), pages 135-151, April.
    21. Egloff, Daniel & Leippold, Markus & Vanini, Paolo, 2007. "A simple model of credit contagion," Journal of Banking & Finance, Elsevier, vol. 31(8), pages 2475-2492, August.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ormnsc:v:57:y:2011:i:3:p:487-505. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.