IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2405.17254.html
   My bibliography  Save this paper

Estimating treatment-effect heterogeneity across sites in multi-site randomized experiments with imperfect compliance

Author

Listed:
  • Cl'ement de Chaisemartin
  • Antoine Deeb

Abstract

We consider multi-site randomized controlled trials with a large number of small sites and imperfect compliance, conducted in non-random convenience samples in each site. We show that an Empirical-Bayes (EB) estimator can be used to estimate a lower bound of the variance of intention-to-treat (ITT) effects across sites. We also propose bounds for the coefficient from a regression of site-level ITTs on sites' control-group outcome. Turning to local average treatment effects (LATEs), the EB estimator cannot be used to estimate their variance, because site-level LATE estimators are biased. Instead, we propose two testable assumptions under which the LATEs' variance can be written as a function of sites' ITT and first-stage (FS) effects, thus allowing us to use an EB estimator leveraging only unbiased ITT and FS estimators. We revisit Behaghel et al. (2014), who study the effect of counselling programs on job seekers job-finding rate, in more than 200 job placement agencies in France. We find considerable ITT heterogeneity, and even more LATE heterogeneity: our lower bounds on ITTs' (resp. LATEs') standard deviation are more than three (resp. four) times larger than the average ITT (resp. LATE) across sites. Sites with a lower job-finding rate in the control group have larger ITT effects.

Suggested Citation

  • Cl'ement de Chaisemartin & Antoine Deeb, 2024. "Estimating treatment-effect heterogeneity across sites in multi-site randomized experiments with imperfect compliance," Papers 2405.17254, arXiv.org.
    • Handle: RePEc:arx:papers:2405.17254
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2405.17254
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Rajeev Dehejia & Cristian Pop-Eleches & Cyrus Samii, 2021. "From Local to Global: External Validity in a Fertility Natural Experiment," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 39(1), pages 217-243, January.
    2. Christopher R. Walters, 2015. "Inputs in the Production of Early Childhood Human Capital: Evidence from Head Start," American Economic Journal: Applied Economics, American Economic Association, vol. 7(4), pages 76-102, October.
    3. Karthik Muralidharan & Paul Niehaus, 2017. "Experimentation at Scale," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 103-124, Fall.
    4. Alberto Abadie & Susan Athey & Guido W. Imbens & Jeffrey M. Wooldridge, 2020. "Sampling‐Based versus Design‐Based Uncertainty in Regression Analysis," Econometrica, Econometric Society, vol. 88(1), pages 265-296, January.
    5. Guido W. Imbens & Charles F. Manski, 2004. "Confidence Intervals for Partially Identified Parameters," Econometrica, Econometric Society, vol. 72(6), pages 1845-1857, November.
    6. Hunt Allcott, 2015. "Site Selection Bias in Program Evaluation," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 130(3), pages 1117-1165.
    7. Karun Adusumilli & Francesco Agostinelli & Emilio Borghesan, 2024. "Heterogeneity and Endogenous Compliance: Implications for Scaling Class Size Interventions," NBER Working Papers 32338, National Bureau of Economic Research, Inc.
    8. Karun Adusumilli & Francesco Agostinelli & Emilio Borghesan, 2024. "Heterogeneity and Endogenous Compliance: Implications for Scaling Class Size Interventions," Working Papers 2024-007, Human Capital and Economic Opportunity Working Group.
    9. Xinran Li & Peng Ding, 2017. "General Forms of Finite Population Central Limit Theorems with Applications to Causal Inference," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(520), pages 1759-1769, October.
    10. Imbens, Guido W & Angrist, Joshua D, 1994. "Identification and Estimation of Local Average Treatment Effects," Econometrica, Econometric Society, vol. 62(2), pages 467-475, March.
    11. Joseph Hotz, V. & Imbens, Guido W. & Mortimer, Julie H., 2005. "Predicting the efficacy of future training programs using past experiences at other locations," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 241-270.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Denis Fougère & Nicolas Jacquemet, 2020. "Policy Evaluation Using Causal Inference Methods," SciencePo Working papers Main hal-03455978, HAL.
    2. Michael Gechter & Keisuke Hirano & Jean Lee & Mahreen Mahmud & Orville Mondal & Jonathan Morduch & Saravana Ravindran & Abu S. Shonchoy, 2024. "Selecting Experimental Sites for External Validity," Papers 2405.13241, arXiv.org.
    3. Stefano DellaVigna & Elizabeth Linos, 2022. "RCTs to Scale: Comprehensive Evidence From Two Nudge Units," Econometrica, Econometric Society, vol. 90(1), pages 81-116, January.
    4. Vishal Kamat, 2017. "Identifying the Effects of a Program Offer with an Application to Head Start," Papers 1711.02048, arXiv.org, revised Aug 2023.
    5. Andor, Mark A. & Gerster, Andreas & Peters, Jörg & Schmidt, Christoph M., 2020. "Social Norms and Energy Conservation Beyond the US," Journal of Environmental Economics and Management, Elsevier, vol. 103(C).
    6. Ashesh Rambachan & Jonathan Roth, 2020. "Design-Based Uncertainty for Quasi-Experiments," Papers 2008.00602, arXiv.org, revised Feb 2024.
    7. Benjamin Lu & Eli Ben-Michael & Avi Feller & Luke Miratrix, 2023. "Is It Who You Are or Where You Are? Accounting for Compositional Differences in Cross-Site Treatment Effect Variation," Journal of Educational and Behavioral Statistics, , vol. 48(4), pages 420-453, August.
    8. Sianesi, Barbara, 2017. "Evidence of randomisation bias in a large-scale social experiment: The case of ERA," Journal of Econometrics, Elsevier, vol. 198(1), pages 41-64.
    9. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    10. Susan Athey & Guido W. Imbens, 2017. "The State of Applied Econometrics: Causality and Policy Evaluation," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 3-32, Spring.
    11. Abramovsky, Laura & Augsburg, Britta & Lührmann, Melanie & Oteiza, Francisco & Rud, Juan Pablo, 2023. "Community matters: Heterogeneous impacts of a sanitation intervention," World Development, Elsevier, vol. 165(C).
    12. Goodman-Bacon, Andrew, 2021. "Difference-in-differences with variation in treatment timing," Journal of Econometrics, Elsevier, vol. 225(2), pages 254-277.
    13. Isaiah Andrews & Emily Oster, 2017. "A Simple Approximation for Evaluating External Validity Bias," NBER Working Papers 23826, National Bureau of Economic Research, Inc.
    14. Konrad Menzel, 2023. "Transfer Estimates for Causal Effects across Heterogeneous Sites," Papers 2305.01435, arXiv.org, revised May 2024.
    15. Paul Hunermund & Elias Bareinboim, 2019. "Causal Inference and Data Fusion in Econometrics," Papers 1912.09104, arXiv.org, revised Mar 2023.
    16. Andrews, Isaiah & Oster, Emily, 2019. "A simple approximation for evaluating external validity bias," Economics Letters, Elsevier, vol. 178(C), pages 58-62.
    17. Yuehao Bai & Azeem M. Shaikh & Max Tabord-Meehan, 2024. "A Primer on the Analysis of Randomized Experiments and a Survey of some Recent Advances," Papers 2405.03910, arXiv.org.
    18. Daido Kido, 2022. "Distributionally Robust Policy Learning with Wasserstein Distance," Papers 2205.04637, arXiv.org, revised Aug 2022.
    19. Omar Al-Ubaydli & John List & Claire Mackevicius & Min Sok Lee & Dana Suskind, 2019. "How Can Experiments Play a Greater Role in Public Policy? 12 Proposals from an Economic Model of Scaling," Artefactual Field Experiments 00679, The Field Experiments Website.
    20. Xintong Wang & Carlos A. Flores & Alfonso Flores-Lagunes, 2020. "The Effects of Vietnam-Era Military Service on the Long-Term Health of Veterans: A Bounds Analysis," Center for Policy Research Working Papers 234, Center for Policy Research, Maxwell School, Syracuse University.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2405.17254. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.