A powerful approach to the study of moderate effect modification in observational studies

Effect modification means the magnitude or stability of a treatment effect varies as a function of an observed covariate. Generally, larger and more stable treatment effects are insensitive to larger biases from unmeasured covariates, so a causal conclusion may be considerably firmer if this pattern is noted if it occurs. We propose a new strategy, called the submax‐method, that combines exploratory, and confirmatory efforts to determine whether there is stronger evidence of causality—that is, greater insensitivity to unmeasured confounding—in some subgroups of individuals. It uses the joint distribution of test statistics that split the data in various ways based on certain observed covariates. For L binary covariates, the method splits the population L times into two subpopulations, perhaps first men and women, perhaps then smokers and nonsmokers, computing a test statistic from each subpopulation, and appends the test statistic for the whole population, making 2L+1 test statistics in total. Although L binary covariates define 2L interaction groups, only 2L+1 tests are performed, and at least L+1 of these tests use at least half of the data. The submax‐method achieves the highest design sensitivity and the highest Bahadur efficiency of its component tests. Moreover, the form of the test is sufficiently tractable that its large sample power may be studied analytically. The simulation suggests that the submax method exhibits superior performance, in comparison with an approach using CART, when there is effect modification of moderate size. Using data from the NHANES I epidemiologic follow‐up survey, an observational study of the effects of physical activity on survival is used to illustrate the method. The method is implemented in the R package submax which contains the NHANES example. An online Appendix provides simulation results and further analysis of the example.