Bottlenecks for Evidence Adoption

Governments increasingly use RCTs to test innovations before scale up. Yet, we know little about whether and how they incorporate the results of the experiments into policy-making. We follow up with 67 U.S. city departments which collectively ran 73 RCTs in collaboration with a national Nudge Unit. Compared to most contexts, the barriers to adoption are low. Yet, city departments adopt a nudge treatment in follow-on communication in 27% of cases. As potential determinants of adoption we consider (i) the strength of the evidence, as determined by the RCT itself, (ii) features of the organization, such as “organizational capacity” of the city and whether the city staff member working on the RCT has been retained, and (iii) the experimental design, such as whether the RCT was implemented as part of pre-existing communication. We find (i) a limited impact of strength of the evidence and (ii) some impact of city features, especially the retention of the original staff member. By far, the largest predictor of adoption is (iii) whether the communication was pre-existing, as opposed to a new communication. We consider two main interpretations of this finding: organizational inertia, in that changes to pre-existing communications are more naturally folded into year-to-year city processes, and costs, since new communications may require additional funding. We find the same pattern for electronic communications, with zero marginal costs, supporting the organizational inertia explanation. The pattern of results differs from the predictions of both experts and practitioners, who over-estimate the extent of evidence-based adoption. Our results underline the importance of considering the barriers to evidence adoption, beginning at the stage of experimental design and continuing after the RCT completion.