Incorporating complex sample design effects when only final survey weights are available

In this article, we consider the situation that arises when a survey data producer has collected data from a sample with a complex design (possibly featuring stratification of the population, cluster sampling, and unequal proba- bilities of selection) and for various reasons only provides secondary analysts of those survey data with a final survey weight for each respondent and “average” design effects for survey estimates computed from the data. In general, these “av- erage” design effects, presumably computed by the data producer in a way that fully accounts for all the complex sampling features, already incorporate possible increases in sampling variance due to the use of the survey weights in estimation. The secondary analyst of the survey data—who uses the provided information to compute weighted estimates; computes design-based standard errors reflecting variance in the weights (by using Taylor series linearization, for example); and inflates the estimated variances using the “average” design effects provided—is applying a “double” adjustment to the standard errors for the effect of weighting on the variance estimates, leading to overly conservative inferences. We propose a simple method to prevent this problem and provide a Stata program for applying appropriate adjustments to variance estimates in this situation. We illustrate two applications of the method with survey data from the Monitoring the Future study and conclude with suggested directions for future research in this area.