The Balance Permutation Test: A Machine Learning Replacement for Balance Tables

Balance tests are standard for experiments in numerous fields, with many journals across disciplines recommending or requiring them for publication. This standard persists despite significant evidence of balance tests' inadequacies and the development of better tools for detecting failures of random assignment and covariate imbalance. To date there is still no consensus on how randomization and balance should be checked, and also how these failures and imbalances should be addressed, or if they should be addressed at all. In this article we provide clear guidelines and implement a new statistical test, the "balance permutation test," designed to detect arbitrarily complex randomization failures. Our approach leverages a combination of permutation inference and the predictive power of machine learning to accomplish this task. Additionally, we advocate reporting both simple unadjusted and "doubly robust" treatment effect estimates in all experimental contexts, but particularly in situations where failures are detected. To justify our recommendations and the use of our method, we report the results of two sets of applications. First, we show how the balance permutation test is able to detect complex imbalance in real, simulated, and even fabricated data. Second, using an extensive set of Monte Carlo simulations, we demonstrate the overwhelming advantages of doubly robust treatment effect estimation over existing methods. Finally, we introduce an efficient, easy-to-use R package, MLbalance, that implements the balance permutation test approach. Our hope is that this method helps resolve the longstanding debate over how to detect and adjust for assignment failures in experiments.