Unintended Policy Impacts of Outcome-Based Measures in a New Era of Service Delivery: Examples from Preschool Immunization

As public programs move to block grants and other more decentralized forms of government intervention, one of the most vexing problems is accountability. A commonly suggested solution is to develop an outcome-based approach to measuring program success. One measure that is already widely used to assess the quality of children's health care is immunization rates among preschool children. This paper examines the construction and use of immunization rates and discusses the lessons learned as new performance standards are debated. We probe the details of how group rates of immunizations are constructed, and how these data are used in the process of evaluating quality of care. Immunization rates seem highly appropriate for accountability: they have strong face validity and are assumed to be a useful proxy for good preventive care for children. But when the data and the process of rate construction are examined closely, we find many ambiguities. We suggest that while data point to a recent increase in immunization rates, these higher rates do not necessarily indicate that children's access to health care has improved. We will discuss how recent efforts to increase immunization rates might have unintended and even negative consequences for the well being of children. The experience with measuring immunization rates underscores that performance measures are inherently socially constructed facts, not perfect mirrors of actions. Policy lessons include: 1) The resources and attention devoted to a measure will change over time depending on public and political interest; 2) even in the best designed projects, critical information about how data are gathered remains mysterious; 3) even seemingly simple outcomes are rarely, if ever, simple to measure; and 4) finally, and perhaps most important, documentation requirements can corrupt the social processes the records are intended to measure.