How do we know we need to control for selectivity?

In the previous two decades there has been considerable progress in recognizing biases due to selectivity that are associated with the use of observational data to make causal inferences and in developing models to control for these biases statistically. Often there is a difference between estimates produced by models that attempt to control for selectivity and those that do not. Since a difference alone does not persuasively argue for one model over another, analysts typically rely on their a priori expectations of selectivity based on theory or intuition. Here we suggest that the analyst’s judgement about the appropriate analytical model may be informed by simple descriptive statistics and qualitative data. We use data on social networks collected in rural Kenya, since the analysis of networks is likely to raise questions of selectivity, and simple examples. Although we do not provide general rules for assessing when models that control for selectivity should be used, we conclude by recommending that analysts inform their judgement rather than rely on theory and intuition to justify controlling for selectivity. Although our data are particular, the implications of our approach are general, since a priori evaluations of the credibility of assumptions on which analytic models are based can be made in other settings and for other research questions.