Assessing the Performance of Nonexperimental Estimators for Evaluating Head Start

This paper uses experimental data from the Head Start Impact Study (HSIS) combined with nonexperimental data from the Early Childhood Longitudinal Study–Birth Cohort (ECLS-B) to study the performance of nonexperimental estimators for evaluating Head Start program impacts. The estimators studied include parametric cross-section and difference-in-differences regression estimators and nonparametric cross-section and difference-in-differences matching estimators. The estimators are used to generate program impacts on cognitive achievement test scores, child health measures, parenting behaviors, and parent labor market outcomes. Some of the estimators closely reproduce the experimental results, but a priori it would be difficult to know whether the estimator works well for any particular outcome. Pre-program exogeneity tests eliminate some outcomes and estimators with the worst biases, but estimators/outcomes with substantial biases pass the tests. The difference-in-differences matching estimator exhibits the best performance in terms of low bias values and capturing the pattern of statistically significant treatment effects. However, the variation in bias is greater across outcomes examined than across methods.