Panel Data Designs and Estimators as Substitutes for Randomized Controlled Trials in the Evaluation of Public Programs

In the evaluation of public programs, experimental designs are rare. Researchers instead rely on observational designs. Observational designs that use panel data are widely portrayed as superior to time-series or cross-sectional designs because they provide opportunities to control for observable and unobservable variables correlated with outcomes and exposure to a program. The most popular panel data evaluation designs use linear, fixed-effects estimators with additive individual and time effects. To assess the ability of observational designs to replicate results from experimental designs, scholars use design replications. No such replications have assessed popular, fixed-effects panel data models that exploit repeated observations before and after treatment assignment. We implement such a study using, as a benchmark, results from a randomized environmental program that included effective and ineffective treatments. The popular linear, fixed-effects estimator fails to generate impact estimates or statistical inferences similar to the experimental estimator. Applying common flexible model specifications or trimming procedures also fail to yield accurate estimates or inferences. However, following best practices for selecting a nonexperimental comparison group and combining matching methods with panel data estimators, we replicate the experimental benchmarks. We demonstrate how the combination of panel and matching methods mitigates common concerns about specifying the correct functional form, the nature of treatment effect heterogeneity, and the way in which time enters the model. Our results are consistent with recent claims that design trumps methods in estimating treatment effects and that combining designs is more likely to approximate a randomized controlled trial than applying a single design.