A Comparison of Time-series Bootstrap Methods in Terms of Backtesting Risk Measurement Models of Banks

The banking crises of the last decade and the devastating effect of banks’ bankruptcies on the economy forced regulators to pay more attention to internal risk measurement systems and also to risk measurement models as a part of these systems. Thus, in accordance with the requirements of both the Basel Committee (bcbs128, bcbs152) and the Bank of Russia (483-P and 3624-U), the validation of risk measurement models is an integral part of the Internal Capital Adequacy and Assessment Process (ICAAP) and Internal Ratings-Based (IRB) approach. However, standard backtesting procedures, which are based on the independence of observations, are often inconsistent due to a long forecasting horizon and, accordingly, overlapping observations. [Ruiz, 2014] proposed to correct the statistics’ distributions and take into account the dependence of observations, using a set of risk factor replicas. Meanwhile this is difficult with unknown data generating process. This paper provides an analysis of several bootstrap methods (e.g. block bootstrap and maximum entropy bootstrap). It estimates its ability to simulate risk factor time-series with original properties, and therefore to be used for correction of statistics’ distributions in the absence of independent observations. Analysis of every method is made in terms of preserving the distribution of risk-factor, its time structure and inter-factor correlation. Thus, the use of the maximum entropy bootstrap provides different outcomes depending on whether it is applied to time series in differences or in absolute values. Nevertheless, in both cases, the resulting statistics tends to distort the backtesting procedure both by accepting the incorrect model, and by rejecting the correct one. The use of block bootstrap methods can also lead to inaccuracy, but, generally, it provides more conservative results. Using intersections in the process of creation of simulated series allows reducing this mismatch.