IDEAS home Printed from https://ideas.repec.org/p/tse/wpaper/128965.html
   My bibliography  Save this paper

How Much Should We Trust Observational Estimates? Accumulating Evidence Using Randomized Controlled Trials with Imperfect Compliance

Author

Listed:
  • Rhys Bernard, David
  • Bryan, Gharad
  • Chabé-Ferret, Sylvain
  • De Quidt, Jonathan
  • Fliegner, Jasmin
  • Rathelot, Roland

Abstract

The use of observational methods remains common in program evaluation. How much should we trust these studies, which lack clear identifying variation? We propose adjusting confidence intervals to incorporate the uncertainty due to observational bias. Using data from 44 development RCTs with imperfect compliance (ICRCTs), we estimate the parameters required to construct our confidence intervals. The results show that, after accounting for potential bias, observational studies have low effective power. Using our adjusted confidence intervals, a hypothetical infinite sample size observational study has a minimum detectable effect size of over 0.3 standard deviations. We conclude that – given current evidence –observational studies are uninformative about many programs that in truth have important effects. There is a silver lining: collecting data from more ICRCTs may help to reduce uncertainty about bias, and increase the effective power of observational program evaluation in the future.

Suggested Citation

  • Rhys Bernard, David & Bryan, Gharad & Chabé-Ferret, Sylvain & De Quidt, Jonathan & Fliegner, Jasmin & Rathelot, Roland, 2024. "How Much Should We Trust Observational Estimates? Accumulating Evidence Using Randomized Controlled Trials with Imperfect Compliance," TSE Working Papers 24-1498, Toulouse School of Economics (TSE).
    • Handle: RePEc:tse:wpaper:128965
    as

    Download full text from publisher

    File URL: https://www.tse-fr.eu/sites/default/files/TSE/documents/doc/wp/2024/wp_tse_1498.pdf
    File Function: Full Text
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Anderson, Michael L, 2008. "Multiple Inference and Gender Differences in the Effects of Early Intervention: A Reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects," Department of Agricultural & Resource Economics, UC Berkeley, Working Paper Series qt15n8j26f, Department of Agricultural & Resource Economics, UC Berkeley.
    2. Nava Ashraf & Xavier Giné & Dean Karlan, 2009. "Finding Missing Markets (and a Disturbing Epilogue): Evidence from an Export Crop Adoption and Marketing Intervention in Kenya," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 91(4), pages 973-990.
    3. Karlan, Dean & Osman, Adam & Zinman, Jonathan, 2016. "Follow the money not the cash: Comparing methods for identifying consumption and investment responses to a liquidity shock," Journal of Development Economics, Elsevier, vol. 121(C), pages 11-23.
    4. Abhijit V. Banerjee & Shawn Cole & Esther Duflo & Leigh Linden, 2007. "Remedying Education: Evidence from Two Randomized Experiments in India," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 122(3), pages 1235-1264.
    5. Brett R. Gordon & Florian Zettelmeyer & Neha Bhargava & Dan Chapsky, 2019. "A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook," Marketing Science, INFORMS, vol. 38(2), pages 193-225, March.
    6. Miriam Bruhn & Dean Karlan & Antoinette Schoar, 2018. "The Impact of Consulting Services on Small and Medium Enterprises: Evidence from a Randomized Trial in Mexico," Journal of Political Economy, University of Chicago Press, vol. 126(2), pages 635-687.
    7. Joshua D. Angrist & Jörn-Steffen Pischke, 2009. "Mostly Harmless Econometrics: An Empiricist's Companion," Economics Books, Princeton University Press, edition 1, number 8769.
    8. Kate Ambler & Diego Aycinena & Dean Yang, 2015. "Channeling Remittances to Education: A Field Experiment among Migrants from El Salvador," American Economic Journal: Applied Economics, American Economic Association, vol. 7(2), pages 207-232, April.
    9. Alan S. Gerber & Dean Karlan & Daniel Bergan, 2009. "Does the Media Matter? A Field Experiment Measuring the Effect of Newspapers on Voting Behavior and Political Opinions," American Economic Journal: Applied Economics, American Economic Association, vol. 1(2), pages 35-52, April.
    10. Pascaline Dupas, 2011. "Do Teenagers Respond to HIV Risk Information? Evidence from a Field Experiment in Kenya," American Economic Journal: Applied Economics, American Economic Association, vol. 3(1), pages 1-34, January.
    11. Anderson, Michael L., 2008. "Multiple Inference and Gender Differences in the Effects of Early Intervention: A Reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects," Journal of the American Statistical Association, American Statistical Association, vol. 103(484), pages 1481-1495.
    12. Luc Behaghel & Clément de Chaisemartin & Marc Gurgand, 2017. "Ready for Boarding? The Effects of a Boarding School for Disadvantaged Students," American Economic Journal: Applied Economics, American Economic Association, vol. 9(1), pages 140-164, January.
    13. Xavier Giné & Dean Karlan & Jonathan Zinman, 2010. "Put Your Money Where Your Butt Is: A Commitment Contract for Smoking Cessation," American Economic Journal: Applied Economics, American Economic Association, vol. 2(4), pages 213-235, October.
    14. Blattman, Christopher & Annan, Jeannie, 2016. "Can Employment Reduce Lawlessness and Rebellion? A Field Experiment with High-Risk Men in a Fragile State," American Political Science Review, Cambridge University Press, vol. 110(1), pages 1-17, February.
    15. Andrew S. Griffen & Petra E. Todd, 2017. "Assessing the Performance of Nonexperimental Estimators for Evaluating Head Start," Journal of Labor Economics, University of Chicago Press, vol. 35(S1), pages 7-63.
    16. Rukmini Banerji & James Berry & Marc Shotland, 2017. "The Impact of Maternal Literacy and Participation Programs: Evidence from a Randomized Evaluation in India," American Economic Journal: Applied Economics, American Economic Association, vol. 9(4), pages 303-337, October.
    17. Ashraf, Nava & Giné, Xavier & Karlan, Dean S., 2009. "AJAE appendix for “Finding Missing Markets (and a Disturbing Epilogue): Evidence from an Export Crop Adoption and Marketing Intervention in Kenya”," American Journal of Agricultural Economics APPENDICES, Agricultural and Applied Economics Association, vol. 91(4), pages 1-9, February.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Magnan, Nicholas & Hoffmann, Vivian & Opoku, Nelson & Gajate Garrido, Gissele & Kanyam, Daniel Akwasi, 2021. "Information, technology, and market rewards: Incentivizing aflatoxin control in Ghana," Journal of Development Economics, Elsevier, vol. 151(C).
    2. Bjorn Van Campenhout & David J. Spielman & Els Lecoutere, 2021. "Information and Communication Technologies to Provide Agricultural Advice to Smallholder Farmers: Experimental Evidence from Uganda," American Journal of Agricultural Economics, John Wiley & Sons, vol. 103(1), pages 317-337, January.
    3. Derksen, Laura & Leclerc, Catherine Michaud & Souza, Pedro CL, 2019. "Searching for Answers : The Impact of Student Access to Wikipedia," The Warwick Economics Research Paper Series (TWERPS) 1236, University of Warwick, Department of Economics.
    4. Kyle Emerick & Alain de Janvry & Elisabeth Sadoulet & Manzoor H. Dar, 2016. "Technological Innovations, Downside Risk, and the Modernization of Agriculture," American Economic Review, American Economic Association, vol. 106(6), pages 1537-1561, June.
    5. Arouna, Aminou & Michler, Jeffrey D. & Lokossou, Jourdain C., 2021. "Contract farming and rural transformation: Evidence from a field experiment in Benin," Journal of Development Economics, Elsevier, vol. 151(C).
    6. Aggarwal, Shilpa, 2021. "The long road to health: Healthcare utilization impacts of a road pavement policy in rural India," Journal of Development Economics, Elsevier, vol. 151(C).
    7. Brown, Annette N. & Wood, Benjamin Douglas Kuflick, 2018. "Which tests not witch hunts: A diagnostic approach for conducting replication research," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 12, pages 1-26.
    8. Bandiera, Oriana. & Buehren, Niklas. & Burgess, Robin & Goldstein, Markus P., & Gulesci, Selim. & Rasul, Imran. & Sulaiman, Munshi., 2015. "Women’s economic empowerment in action : evidence from a randomized control trial in Africa," ILO Working Papers 994874053402676, International Labour Organization.
    9. Derksen, Laura & Michaud-Leclerc, Catherine & Souza, Pedro C.L., 2022. "Restricted access: How the internet can be used to promote reading and learning," Journal of Development Economics, Elsevier, vol. 155(C).
    10. Pascaline Dupas & Edward Miguel, 2016. "Impacts and Determinants of Health Levels in Low-Income Countries," NBER Working Papers 22235, National Bureau of Economic Research, Inc.
    11. Derksen, Laura & Leclerc, Catherine Michaud & Souza, Pedro CL, 2019. "Searching for Answers: The Impact of Student Access to Wikipedia," CAGE Online Working Paper Series 450, Competitive Advantage in the Global Economy (CAGE).
    12. Sant’Anna, Pedro H.C. & Zhao, Jun, 2020. "Doubly robust difference-in-differences estimators," Journal of Econometrics, Elsevier, vol. 219(1), pages 101-122.
    13. Emilio Depetris-Chauvin & Ömer Özak, 2020. "The origins of the division of labor in pre-industrial times," Journal of Economic Growth, Springer, vol. 25(3), pages 297-340, September.
    14. Fitzsimons, Emla & Malde, Bansi & Mesnard, Alice & Vera-Hernández, Marcos, 2016. "Nutrition, information and household behavior: Experimental evidence from Malawi," Journal of Development Economics, Elsevier, vol. 122(C), pages 113-126.
    15. Emily Beam & Priya Mukherjee & Laia Navarro-Sola, 2022. "Lowering Barriers to Remote Education: Experimental Impacts on Parental Responses and Learning," Working Papers 2022-030, Human Capital and Economic Opportunity Working Group.
    16. Sule Alan & Gozde Corekcioglu & Matthias Sutter, 2023. "Improving Workplace Climate in Large Corporations: A Clustered Randomized Intervention," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 138(1), pages 151-203.
    17. Sonia Bhalotra & Martin Karlsson & Therese Nilsson & Nina Schwarz, 2022. "Infant Health, Cognitive Performance, and Earnings: Evidence from Inception of the Welfare State in Sweden," The Review of Economics and Statistics, MIT Press, vol. 104(6), pages 1138-1156, November.
    18. Daniela Scur & Raffaella Sadun & John Van Reenen & Renata Lemos & Nicholas Bloom, 2021. "The World Management Survey at 18: lessons and the way forward," Oxford Review of Economic Policy, Oxford University Press and Oxford Review of Economic Policy Limited, vol. 37(2), pages 231-258.
    19. Joshua D. Angrist & Sarah R. Cohodes & Susan M. Dynarski & Parag A. Pathak & Christopher R. Walters, 2016. "Stand and Deliver: Effects of Boston's Charter High Schools on College Preparation, Entry, and Choice," Journal of Labor Economics, University of Chicago Press, vol. 34(2), pages 275-318.
    20. Marshall Burke & Lauren Falcao Bergquist & Edward Miguel, 2019. "Sell Low and Buy High: Arbitrage and Local Price Effects in Kenyan Markets," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 134(2), pages 785-842.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:tse:wpaper:128965. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/tsetofr.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.