Data-Driven Model Evaluation: A Test for Revealed Performance

When comparing two competing approximate models using a particular loss function, the one having smallest `expected true error' for that loss function is expected to lie closest to the underlying data generating process (DGP) given this loss function and is therefore to be preferred. In this chapter we consider a data-driven method for testing whether or not two competing approximate models are equivalent in terms of their expected true error (i.e., their expected performance on unseen data drawn from the same DGP). The proposed test is quite flexible with regards to the types of models that can be compared (i.e., nested versus non-nested, parametric versus nonparametric) and is applicable in cross-sectional and time-series settings. Moreover, in time-series settings our method overcomes two of the drawbacks associated with dominant approaches, namely, their reliance on only one split of the data and the need to have a sufficiently large `hold-out' sample for these tests to possess adequate power.