IDEAS home Printed from https://ideas.repec.org/p/osf/osfxxx/k5uap.html
   My bibliography  Save this paper

Scaling Behavioral Interventions in the Presence of Spillover

Author

Listed:
  • Ternovski, John
  • Keppeler, Florian
  • Jilke, Sebastian

    (Georgetown University)

  • Vogel, Dominik

    (University of Hamburg)

Abstract

Randomized Controlled Trials (RCTs) are increasingly relied upon by policymakers as part of efforts to incorporate evidence into the policymaking process, a movement known as evidence-based policymaking, or EBPM. Testing possible policy interventions via RCTs before full rollout is commonly thought to be the gold standard of evidence in the EBPM process. However, real-world policy changes do not always scale up as expected. Even large-N RCTs targeting a random sample of policy beneficiaries do not capture the influence of social networks and risk missing consequential spillover effects. We illustrate this issue by assessing the efficacy of monetary incentives to increase COVID-19 vaccination in an RCT over the entire population of a medium-sized European town (~40,000 residents). We use administrative vaccination data as our primary outcome. Since the entire population was randomized, we are able to estimate spillover effects within households. There were significant negative spillover effects on booster vaccinations that we attribute to a displacement effect, potentially driven by long lines at the vaccination events. Our results illustrate that using a population-level RCT to test whether a policy scales can help avoid costly, ineffective, or even counterproductive policy outcomes.

Suggested Citation

  • Ternovski, John & Keppeler, Florian & Jilke, Sebastian & Vogel, Dominik, 2023. "Scaling Behavioral Interventions in the Presence of Spillover," OSF Preprints k5uap, Center for Open Science.
    • Handle: RePEc:osf:osfxxx:k5uap
    DOI: 10.31219/osf.io/k5uap
    as

    Download full text from publisher

    File URL: https://osf.io/download/653aa37528274513f5b864ed/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/k5uap?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Michael McGann & Emma Blomkamp & Jenny M. Lewis, 2018. "The rise of public sector innovation labs: experiments in design thinking for policy," Policy Sciences, Springer;Society of Policy Sciences, vol. 51(3), pages 249-267, September.
    2. Jon Baron, 2018. "A Brief History of Evidence-Based Policy," The ANNALS of the American Academy of Political and Social Science, , vol. 678(1), pages 40-50, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ryan Anders Whitney & David López-García, 2023. "Fast-track institutionalization: The opening of urban planning best practice agencies in Mexico City," Environment and Planning C, , vol. 41(3), pages 600-616, May.
    2. Alfonso Unceta & Xabier Barandiaran & Natalia Restrepo, 2019. "The Role of Public Innovation Labs in Collaborative Governance—The Case of the Gipuzkoa Lab in the Basque Country, Spain," Sustainability, MDPI, vol. 11(21), pages 1-16, November.
    3. Makoza, Frank, 2023. "Analyzing policy change of Malawi ICT and Digitalization policy: Policy Assemblage Perspective," EconStor Preprints 273309, ZBW - Leibniz Information Centre for Economics.
    4. Strzałkowski, Andrzej, 2024. "Adaptation and operationalisation of sustainable degrowth for policy: Why we need to translate research papers into legislative drafts?," Ecological Economics, Elsevier, vol. 220(C).
    5. Simon Hull & Jennifer Whittal, 2021. "Do Design Science Research and Design Thinking Processes Improve the ‘Fit’ of the Fit-For-Purpose Approach to Securing Land Tenure for All in South Africa?," Land, MDPI, vol. 10(5), pages 1-26, May.
    6. Lars Fuglsang & Anne Vorre Hansen & Ines Mergel & Maria Taivalsaari Røhnebæk, 2021. "Living Labs for Public Sector Innovation: An Integrative Literature Review," Administrative Sciences, MDPI, vol. 11(2), pages 1-19, June.
    7. Sojeong Kim & Adam M. Wellstead & Tanya Heikkila, 2023. "Policy capacity and rise of data‐based policy innovation labs," Review of Policy Research, Policy Studies Organization, vol. 40(3), pages 341-362, May.
    8. Masahiro Sugiyama & Jun Muto, 2024. "Measuring policy analytical capacity in renewable energy policy: Germany‐Japan‐US comparison," Review of Policy Research, Policy Studies Organization, vol. 41(1), pages 184-209, January.
    9. Yongjin Choi & Ashley M. Fox & Jennifer Dodge, 2022. "What counts? Policy evidence in public hearing testimonies: the case of single-payer healthcare in New York State," Policy Sciences, Springer;Society of Policy Sciences, vol. 55(4), pages 631-660, December.
    10. Triyuth Promsiri & Krisakorn Sukavejworakit & Vasu Keerativutisest & Thanaphol Virasa & Krischanan Kampanthong, 2022. "Sustaining Thai Government Agency Innovation through Design Thinking Learning Effectiveness," Sustainability, MDPI, vol. 14(12), pages 1-13, June.
    11. Jesper Dahl Kelstrup & Jonas Videbæk Jørgensen, 2024. "Explaining differences in research utilization in evidence-based government ministries," Policy Sciences, Springer;Society of Policy Sciences, vol. 57(2), pages 257-280, June.
    12. Jesper Dahl Kelstrup, 2024. "Understanding the role of practical knowledge in evidence-based welfare reform—a three-stage model," Palgrave Communications, Palgrave Macmillan, vol. 11(1), pages 1-8, December.
    13. Joshua B Cohen, 2022. "Institutionalizing public engagement in research and innovation: Toward the construction of institutional entrepreneurial collectives [Limits of Decentered Governance in Science-society Policies]," Science and Public Policy, Oxford University Press, vol. 49(5), pages 673-685.
    14. John, Laura, 2022. "Rethinking digital governance - How collaborative innovation strategies advance the development of digital innovations in public organisations," Junior Management Science (JUMS), Junior Management Science e. V., vol. 7(5), pages 1400-1418.
    15. Kawther Saeedi & Anna Visvizi & Dimah Alahmadi & Amal Babour, 2023. "Smart Cities and Households’ Recyclable Waste Management: The Case of Jeddah," Sustainability, MDPI, vol. 15(8), pages 1-23, April.
    16. Schiuma, Giovanni & Santarsiero, Francesco, 2023. "Innovation labs as organisational catalysts for innovation capacity development: A systematic literature review," Technovation, Elsevier, vol. 123(C).
    17. Alessandro Pollini & Alessandro Caforio, 2021. "Participation and Iterative Experiments: Designing Alternative Futures with Migrants and Service Providers," Social Sciences, MDPI, vol. 10(10), pages 1-13, September.
    18. Vasco Fonseca & Joaquim Caeiro & Fernanda Nogueira, 2021. "Social Model—Innovation and Behavioural Intervention as a Public Policy of Action within an Oncology and Loneliness Scope," Sustainability, MDPI, vol. 13(3), pages 1-7, February.
    19. Meister Broekema Peter & Bulder Elisabeth A. M. & Horlings Lummina G., 2022. "Same same, but different…? The emergence of Public Sector Innovation Labs in theory and practice," Management & Marketing, Sciendo, vol. 17(s1), pages 344-363, October.
    20. Vassallo, Jarrod P. & Banerjee, Sourindra & Zaman, Hasanuzzaman & Prabhu, Jaideep C., 2023. "Design thinking and public sector innovation: The divergent effects of risk-taking, cognitive empathy and emotional empathy on individual performance," Research Policy, Elsevier, vol. 52(6).

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:osfxxx:k5uap. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.