Inappropriate Comparisons as a Basis for Policy: Two Recent Examples from the Social Experiments

This paper examines the problem of inappropriate comparisons in evaluating social programs and the erroneous policy conclusions which can be derived from such comparisons. The paper examines two cases from the United States of America which fail to meet the criterion that comparison groups (if necessary after statistical adjustment) should be identical in all essential respects except in their exposure to the program. In the first of the two cases, involving the measurement of work-effort reduction in negative income tax (NIT) experiments, the inappropriate comparison was made in conducting a statistical analysis of program effects. In the second case, involving the analysis of changes in housing consumption in a housing allowance experiment, the statistical analysis of program effects appears valid, but the correctly measured program outcomes were themselves inappropriately used by policy-makers in drawing policy inferences. The conclusion draws out major lessons for policy analysis and policy-making.