Policy evaluation during a pandemic

National and local governments have implemented a large number of policies in response to the Covid-19 pandemic. Evaluating the effects of these policies, both on the number of Covid-19 cases as well as on other economic outcomes is a key ingredient for policymakers to be able to determine which policies are most effective as well as the relative costs and benefits of particular policies. In this paper, we consider the relative merits of common identification strategies that exploit variation in the timing of policies across different locations by checking whether the identification strategies are compatible with leading epidemic models in the epidemiology literature. We argue that unconfoundedness type approaches, that condition on the pre-treatment “state” of the pandemic, are likely to be more useful for evaluating policies than difference-in-differences type approaches due to the highly nonlinear spread of cases during a pandemic. For difference-in-differences, we further show that a version of this problem continues to exist even when one is interested in understanding the effect of a policy on other economic outcomes when those outcomes also depend on the number of Covid-19 cases. We propose alternative approaches that are able to circumvent these issues. We apply our proposed approach to study the effect of state level shelter-in-place orders early in the pandemic.