Quantifying model selection risk in macroeconomic sensitivity models

Model selection is a critical step in model development and brings with it a significant level of model risk. This analysis considers a real-world example and compares the forecasts and uncertainties of three possibilities: the model selected as best, the best ensemble and the model not selected. A key part of this analysis is propagating the prediction intervals for the individual forecasts using a Monte Carlo simulation all the way through the lifetime probability of default calculations to obtain distributions for the lifetime forecasts. The results highlight two sources of risk: a systematic difference between the expectation value of the forecast and the median of the simulated distribution, and the 95% confidence interval from the distribution. The difference between the forecast and the simulated median should be included as part of the loss reserve. The confidence interval from the distribution should contribute to the model risk component of the economic capital. This paper also illustrates how uncertainties can be computed for Current Expected Credit Losses or International Financial Reporting Standard 9 Stage 2 lifetime loss estimates.