IDEAS home Printed from https://ideas.repec.org/p/wbk/wbrwps/10098.html
   My bibliography  Save this paper

Adaptive Experiments for Policy Choice : Phone Calls for Home Reading in Kenya

Author

Listed:
  • Esposito Acosta,Bruno Nicola
  • Sautmann,Anja

Abstract

Adaptive sampling in experiments with multiple waves can improve learning for “policy choiceproblems” where the goal is to select the optimal intervention or treatment among several options. This paperuses a real-world policy choice problem to demonstrate the advantages of adaptive sampling and propose solutions tocommon issues in applying the method. The application is a test of six formats for automated calls to parents in Kenyathat encourage reading with children at home. The adaptive ‘exploration sampling’ algorithm is used to efficientlyidentify the call with the highest rate of engagement. Simulations show that adaptive sampling increased theposterior probability of the chosen arm being optimal from 86 to 93 percent and more than halved the posterior expected regret. The paper discusses a range of implementationaspects, including how to decide about research design parameters such as the number of experimental waves.

Suggested Citation

  • Esposito Acosta,Bruno Nicola & Sautmann,Anja, 2022. "Adaptive Experiments for Policy Choice : Phone Calls for Home Reading in Kenya," Policy Research Working Paper Series 10098, The World Bank.
    • Handle: RePEc:wbk:wbrwps:10098
    as

    Download full text from publisher

    File URL: http://documents.worldbank.org/curated/en/099337506232226851/pdf/IDU078c6dff10259d0409d0997f0d753a7e2cb72.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Abhijit Banerjee & Rukmini Banerji & James Berry & Esther Duflo & Harini Kannan & Shobhini Mukerji & Marc Shotland & Michael Walton, 2017. "From Proof of Concept to Scalable Policies: Challenges and Solutions, with an Application," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 73-102, Fall.
    2. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    3. Abhijit V. Banerjee & Shawn Cole & Esther Duflo & Leigh Linden, 2007. "Remedying Education: Evidence from Two Randomized Experiments in India," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 122(3), pages 1235-1264.
    4. Susan Athey & Raj Chetty & Guido W. Imbens & Hyunseung Kang, 2019. "The Surrogate Index: Combining Short-Term Proxies to Estimate Long-Term Treatment Effects More Rapidly and Precisely," NBER Working Papers 26463, National Bureau of Economic Research, Inc.
    5. Karthik Muralidharan & Abhijeet Singh & Alejandro J. Ganimian, 2019. "Disrupting Education? Experimental Evidence on Technology-Aided Instruction in India," American Economic Review, American Economic Association, vol. 109(4), pages 1426-1460, April.
    6. Rodriguez-Segura, Daniel & Campton, Cole & Crouch, Luis & Slade, Timothy S., 2021. "Looking beyond changes in averages in evaluating foundational learning: Some inequality measures," International Journal of Educational Development, Elsevier, vol. 84(C).
    7. Duflo, Esther & Glennerster, Rachel & Kremer, Michael, 2008. "Using Randomization in Development Economics Research: A Toolkit," Handbook of Development Economics, in: T. Paul Schultz & John A. Strauss (ed.), Handbook of Development Economics, edition 1, volume 4, chapter 61, pages 3895-3962, Elsevier.
    8. Peter Bergman & Eric W. Chan, 2021. "Leveraging Parents through Low-Cost Technology: The Impact of High-Frequency Information on Student Achievement," Journal of Human Resources, University of Wisconsin Press, vol. 56(1), pages 125-158.
    9. Bahety, Girija & Bauhoff, Sebastian & Patel, Dev & Potter, James, 2021. "Texts don’t nudge: An adaptive trial to prevent the spread of COVID-19 in India," Journal of Development Economics, Elsevier, vol. 153(C).
    10. Isaiah Andrews & Maximilian Kasy, 2019. "Identification of and Correction for Publication Bias," American Economic Review, American Economic Association, vol. 109(8), pages 2766-2794, August.
    11. Peter Bergman, 2021. "Parent-Child Information Frictions and Human Capital Investment: Evidence from a Field Experiment," Journal of Political Economy, University of Chicago Press, vol. 129(1), pages 286-322.
    12. A Stefano Caria & Grant Gordon & Maximilian Kasy & Simon Quinn & Soha Osman Shami & Alexander Teytelboym, 2024. "An Adaptive Targeted Field Experiment: Job Search Assistance for Refugees in Jordan," Journal of the European Economic Association, European Economic Association, vol. 22(2), pages 781-836.
    13. Benjamin N. York & Susanna Loeb & Christopher Doss, 2019. "One Step at a Time: The Effects of an Early Literacy Text-Messaging Program for Parents of Preschoolers," Journal of Human Resources, University of Wisconsin Press, vol. 54(3), pages 537-566.
    14. Dinkelman, Taryn & Berlinski, Samuel & Busso, Matias & Martinez A., Claudia, 2021. "Reducing Parent-School Information Gaps and Improving Education Outcomes: Evidence from High-Frequency Text Messages," CEPR Discussion Papers 15949, C.E.P.R. Discussion Papers.
    15. Matthew A. Kraft & Manuel Monti-Nussbaum, 2017. "Can Schools Enable Parents to Prevent Summer Learning Loss? A Text-Messaging Field Experiment to Promote Literacy Skills," The ANNALS of the American Academy of Political and Social Science, , vol. 674(1), pages 85-112, November.
    16. Kasy, Maximilian, 2016. "Why Experimenters Might Not Always Want to Randomize, and What They Could Do Instead," Political Analysis, Cambridge University Press, vol. 24(3), pages 324-338, July.
    17. Susan E. Mayer & Ariel Kalil & Philip Oreopoulos & Sebastian Gallegos, 2019. "Using Behavioral Insights to Increase Parental Engagement: The Parents and Children Together Intervention," Journal of Human Resources, University of Wisconsin Press, vol. 54(4), pages 900-925.
    18. Frederico Finan & Demian Pouzo, 2021. "Reinforcing RCTs with Multiple Priors while Learning about External Validity," Papers 2112.09170, arXiv.org, revised Sep 2024.
    19. Frederico Finan & Demian Pouzo, 2022. "Reinforcing RCTs with Multiple Priors while Learning about External Validity," NBER Working Papers 29756, National Bureau of Economic Research, Inc.
    20. Eric Bettinger & Nina Cunha & Guilherme Lichand & Ricardo Madeira, 2020. "Are the effects of informational interventions driven by salience?," ECON - Working Papers 350, Department of Economics - University of Zurich, revised May 2021.
    21. Daniel Russo, 2020. "Simple Bayesian Algorithms for Best-Arm Identification," Operations Research, INFORMS, vol. 68(6), pages 1625-1647, November.
    22. Christopher Doss & Erin M. Fahle & Susanna Loeb & Benjamin N. York, 2019. "More Than Just a Nudge: Supporting Kindergarten Parents with Differentiated and Personalized Text Messages," Journal of Human Resources, University of Wisconsin Press, vol. 54(3), pages 567-603.
    23. Abhijit V. Banerjee & Sylvain Chassang & Sergio Montero & Erik Snowberg, 2020. "A Theory of Experimenters: Robustness, Randomization, and Balance," American Economic Review, American Economic Association, vol. 110(4), pages 1206-1230, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Noam Angrist & Peter Bergman & Moitshepi Matsheng, 2022. "Experimental evidence on learning using low-tech when school is out," Nature Human Behaviour, Nature, vol. 6(7), pages 941-950, July.
    2. Henning Hermes & Philipp Lergetporer & Frauke Peter & Simon Wiederhold, 2021. "Application Barriers and the Socioeconomic Gap in Child Care Enrollment," CESifo Working Paper Series 9282, CESifo.
    3. Maldonado, Joana Elisa & De Witte, Kristof, 2021. "The impact of information provision to parents: Experimental evidence on student outcomes," Journal of Behavioral and Experimental Finance, Elsevier, vol. 31(C).
    4. Hermes, Henning & Lergetporer, Philipp & Peter, Frauke & Wiederhold, Simon, 2021. "Behavioral Barriers and the Socioeconomic Gap in Child Care Enrollment," IZA Discussion Papers 14698, Institute of Labor Economics (IZA).
    5. Angrist, Noam & Bergman, Peter & Matsheng, Moitshepi, 2020. "School's Out: Experimental Evidence on Limiting Learning Loss Using," IZA Discussion Papers 14009, Institute of Labor Economics (IZA).
    6. Noam Angrist & Peter Bergman & Moitshepi Matsheng, 2020. "School’s Out: Experimental Evidence on Limiting Learning Loss Using “Low-Tech” in a Pandemic," NBER Working Papers 28205, National Bureau of Economic Research, Inc.
    7. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    8. Chen, Yefeng & Yang, Wenyuan & Luo, Gansong & Luo, Jun, 2024. "Choosing tournament for children: Parenting style and information intervention," China Economic Review, Elsevier, vol. 85(C).
    9. Karthik Muralidharan & Mauricio Romero & Kaspar Wüthrich, 2019. "Factorial Designs, Model Selection, and (Incorrect) Inference in Randomized Experiments," NBER Working Papers 26562, National Bureau of Economic Research, Inc.
    10. Ankel-Peters, Jörg & Schmidt, Christoph M., 2023. "Rural electrification, the credibility revolution, and the limits of evidence-based policy," Ruhr Economic Papers 1051, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
    11. Amaral, Sofia & Dinarte-Diaz, Lelys & Dominguez, Patricio & Perez-Vincent, Santiago M., 2024. "Helping families help themselves: The (Un)intended impacts of a digital parenting program," Journal of Development Economics, Elsevier, vol. 166(C).
    12. Simon Calmar Andersen & Ulrik Hvidman, 2020. "Implementing Educational Interventions at Scale," Working Papers 2020-039, Human Capital and Economic Opportunity Working Group.
    13. Hernández-Agramonte, Juan Manuel & Namen, Olga & Näslund-Hadley, Emma & Biehl, Maria Loreto, 2024. "Supporting early childhood development remotely: Experimental evidence from SMS messages," Journal of Development Economics, Elsevier, vol. 166(C).
    14. Michele Giannola, 2024. "Parental Investments and Intra-household Inequality in Child Human Capital: Evidence from a Survey Experiment," The Economic Journal, Royal Economic Society, vol. 134(658), pages 671-727.
    15. Stanley, T. D. & Doucouliagos, Chris, 2019. "Practical Significance, Meta-Analysis and the Credibility of Economics," IZA Discussion Papers 12458, Institute of Labor Economics (IZA).
    16. Emily Beam & Priya Mukherjee & Laia Navarro-Sola, 2022. "Lowering Barriers to Remote Education: Experimental Impacts on Parental Responses and Learning," Working Papers 2022-030, Human Capital and Economic Opportunity Working Group.
    17. Sabrin Beg & Waqas Halim & Adrienne M. Lucas & Umar Saif, 2022. "Engaging Teachers with Technology Increased Achievement, Bypassing Teachers Did Not," American Economic Journal: Economic Policy, American Economic Association, vol. 14(2), pages 61-90, May.
    18. Christopher Snyder & Ran Zhuo, 2018. "Sniff Tests as a Screen in the Publication Process: Throwing out the Wheat with the Chaff," NBER Working Papers 25058, National Bureau of Economic Research, Inc.
    19. Mo, Di & Bai, Yu & Shi, Yaojiang & Abbey, Cody & Zhang, Linxiu & Rozelle, Scott & Loyalka, Prashant, 2020. "Institutions, implementation, and program effectiveness: Evidence from a randomized evaluation of computer-assisted learning in rural China," Journal of Development Economics, Elsevier, vol. 146(C).
    20. Agostinelli,Francesco & Avitabile,Ciro & Bobba,Matteo, 2021. "Enhancing Human Capital in Children : A Case Study on Scaling," Policy Research Working Paper Series 9647, The World Bank.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wbk:wbrwps:10098. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Roula I. Yazigi (email available below). General contact details of provider: https://edirc.repec.org/data/dvewbus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.