Preventing Dataset Shift During Cross-Validation: Is it worth it?

Dataset shift occurs when there is a discrepancy between the distribution of samples used for training and testing a model, which can lead to a reduction in the model’s predictive performance when attempting to generalize model predictions. Dataset shift can occur with respect to the 1) criterion variable (prior probability shift), 2) predictor variables (covariate shift), and 3) the relationship between criterion and predictors (concept shift). The present paper investigated the implications of avoiding dataset shift during k-fold cross-validation. To circumvent the various forms of dataset shift during cross-validation, a range of anticlustering algorithms were employed to ensure equal means, variances, and covariance structure across folds. A comparative analysis was conducted to assess the bias in validation error obtained through anticlustering-based cross-validation and standard cross-validation. The bias in validation error was computed using the true test error based on unseen test data as reference. Utilizing linear regression models, simulated and empirical datasets demonstrated that avoiding prior probability shift in conjunction with covariate shift—but not concept shift—can effectively curtail the bias in the prediction error during cross-validation. The advantage of using anticlustering-based cross-validation over standard cross-validation however diminished when R2 increased. The study underscores the merits of leveraging anticlustering methodologies within the framework of k-fold cross-validation to mitigate adverse effects of prior probability and covariate shift by equating means and variances in predictor and criterion variables across folds.