IDEAS home Printed from https://ideas.repec.org/p/hal/spmain/hal-03873897.html
   My bibliography  Save this paper

At What Level Should One Cluster Standard Errors in Paired and Small-Strata Experiments?

Author

Listed:
  • Clément de Chaisemartin

    (ECON - Département d'économie (Sciences Po) - Sciences Po - Sciences Po - CNRS - Centre National de la Recherche Scientifique)

  • Jaime Ramirez-Cuellar

    (Microsoft - Microsoft Research [Cambridge] - Microsoft Research)

Abstract

In clustered and paired experiments, to estimate treatment effects, researchers often regress their outcome on the treatment and pair fixed effects, clustering standard errors at the unit-ofrandomization level. We show that even if the treatment has no effect, a 5%-level t-test based on this regression will wrongly conclude that the treatment has an effect up to 16.5% of the time, an error rate much larger than the researcher's 5% target. To achieve their targeted error rate, researchers should instead cluster standard errors at the pair level. Using simulations, we show that similar results apply to clustered experiments with small strata.

Suggested Citation

  • Clément de Chaisemartin & Jaime Ramirez-Cuellar, 2022. "At What Level Should One Cluster Standard Errors in Paired and Small-Strata Experiments?," SciencePo Working papers Main hal-03873897, HAL.
    • Handle: RePEc:hal:spmain:hal-03873897
    DOI: 10.2139/ssrn.3520820
    Note: View the original document on HAL open archive server: https://sciencespo.hal.science/hal-03873897
    as

    Download full text from publisher

    File URL: https://sciencespo.hal.science/hal-03873897/document
    Download Restriction: no
    File URL: https://libkey.io/10.2139/ssrn.3520820?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Clément de Chaisemartin & Jaime Ramirez-Cuellar, 2024. "At What Level Should One Cluster Standard Errors in Paired and Small-Strata Experiments?," American Economic Journal: Applied Economics, American Economic Association, vol. 16(1), pages 193-212, January.
    2. Yuehao Bai & Joseph P. Romano & Azeem M. Shaikh, 2022. "Inference in Experiments With Matched Pairs," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 117(540), pages 1726-1737, October.
    3. Vincent Somville & Lore Vandewalle, 2018. "Saving by Default: Evidence from a Field Experiment in Rural India," American Economic Journal: Applied Economics, American Economic Association, vol. 10(3), pages 39-66, July.
    4. Miriam Bruhn & David McKenzie, 2009. "In Pursuit of Balance: Randomization in Practice in Development Field Experiments," American Economic Journal: Applied Economics, American Economic Association, vol. 1(4), pages 200-232, October.
    5. Bruno Crépon & Florencia Devoto & Esther Duflo & William Parienté, 2015. "Estimating the Impact of Microcredit on Those Who Take It Up: Evidence from a Randomized Experiment in Morocco," American Economic Journal: Applied Economics, American Economic Association, vol. 7(1), pages 123-150, January.
    6. Alberto Abadie & Guido W. Imbens, 2008. "Estimation of the Conditional Variance in Paired Experiments," Annals of Economics and Statistics, GENES, issue 91-92, pages 175-187.
    7. Gary King & Emmanuela Gakidou & Nirmala Ravishankar & Ryan T. Moore & Jason Lakin & Manett Vargas & Martha María Téllez-Rojo & Juan Eugenio Hernández Ávila & Mauricio Hernández Ávila & Héctor Hernánde, 2007. "A “politically robust” experimental design for public policy evaluation, with application to the Mexican Universal Health Insurance program," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 26(3), pages 479-506.
    8. Alberto Abadie & Susan Athey & Guido W Imbens & Jeffrey M Wooldridge, 2023. "When Should You Adjust Standard Errors for Clustering?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 138(1), pages 1-35.
    9. Manuela Angelucci & Dean Karlan & Jonathan Zinman, 2015. "Microcredit Impacts: Evidence from a Randomized Microcredit Program Placement Experiment by Compartamos Banco," American Economic Journal: Applied Economics, American Economic Association, vol. 7(1), pages 151-182, January.
    10. Diether W. Beuermann & Julian Cristia & Santiago Cueto & Ofer Malamud & Yyannu Cruz-Aguayo, 2015. "One Laptop per Child at Home: Short-Term Impacts from a Randomized Experiment in Peru," American Economic Journal: Applied Economics, American Economic Association, vol. 7(2), pages 53-80, April.
    11. Andrew V. Carter & Kevin T. Schnepel & Douglas G. Steigerwald, 2017. "Asymptotic Behavior of a t -Test Robust to Cluster Heterogeneity," The Review of Economics and Statistics, MIT Press, vol. 99(4), pages 698-709, July.
    12. Kate Ambler & Diego Aycinena & Dean Yang, 2015. "Channeling Remittances to Education: A Field Experiment among Migrants from El Salvador," American Economic Journal: Applied Economics, American Economic Association, vol. 7(2), pages 207-232, April.
    13. A. Colin Cameron & Douglas L. Miller, 2015. "A Practitioner’s Guide to Cluster-Robust Inference," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 317-372.
    14. Glewwe, Paul & Park, Albert & Zhao, Meng, 2016. "A better vision for development: Eyeglasses and academic performance in rural primary schools in China," Journal of Development Economics, Elsevier, vol. 122(C), pages 170-182.
    15. Miriam Bruhn & Luciana de Souza Leão & Arianna Legovini & Rogelio Marchetti & Bilal Zia, 2016. "The Impact of High School Financial Education: Evidence from a Large-Scale Evaluation in Brazil," American Economic Journal: Applied Economics, American Economic Association, vol. 8(4), pages 256-295, October.
    16. Roland G. Fryer, Jr, 2017. "Management and Student Achievement: Evidence from a Randomized Field Experiment," NBER Working Papers 23437, National Bureau of Economic Research, Inc.
    17. Orazio Attanasio & Britta Augsburg & Ralph De Haas & Emla Fitzsimons & Heike Harmgart, 2015. "The Impacts of Microfinance: Evidence from Joint-Liability Lending in Mongolia," American Economic Journal: Applied Economics, American Economic Association, vol. 7(1), pages 90-122, January.
    18. repec:adr:anecst:y:2008:i:91-92:p:09 is not listed on IDEAS
    19. Rukmini Banerji & James Berry & Marc Shotland, 2017. "The Impact of Maternal Literacy and Participation Programs: Evidence from a Randomized Evaluation in India," American Economic Journal: Applied Economics, American Economic Association, vol. 9(4), pages 303-337, October.
    20. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, September.
    21. Jeanne Lafortune & Julio Riutort & José Tessada, 2018. "Role Models or Individual Consulting: The Impact of Personalizing Micro-entrepreneurship Training," American Economic Journal: Applied Economics, American Economic Association, vol. 10(4), pages 222-245, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. MacKinnon, James G. & Nielsen, Morten Ørregaard & Webb, Matthew D., 2023. "Testing for the appropriate level of clustering in linear regression models," Journal of Econometrics, Elsevier, vol. 235(2), pages 2027-2056.
    2. Federico Bugni & Ivan Canay & Azeem Shaikh & Max Tabord-Meehan, 2022. "Inference for Cluster Randomized Experiments with Non-ignorable Cluster Sizes," Papers 2204.08356, arXiv.org, revised Apr 2024.
    3. Clément de Chaisemartin & Jaime Ramirez-Cuellar, 2024. "At What Level Should One Cluster Standard Errors in Paired and Small-Strata Experiments?," American Economic Journal: Applied Economics, American Economic Association, vol. 16(1), pages 193-212, January.
    4. Bruno Ferman, 2019. "Assessing Inference Methods," Papers 1912.08772, arXiv.org, revised Oct 2022.
    5. Yuehao Bai & Azeem M. Shaikh & Max Tabord-Meehan, 2024. "A Primer on the Analysis of Randomized Experiments and a Survey of some Recent Advances," Papers 2405.03910, arXiv.org.
    6. Dominik Stelzeneder, 2023. "Does Schooling Affect Political Attitudes? Quasi-Experimental Evidence," Vienna Economics Papers vie2301, University of Vienna, Department of Economics.
    7. Denis Agniel & Jonathan H. Cantor & Johanna Catherine Maclean & Kosali I. Simon & Erin Taylor, 2023. "Insurance Coverage and Provision of Opioid Treatment: Evidence from Medicare," NBER Working Papers 31884, National Bureau of Economic Research, Inc.
    8. Yuehao Bai & Meng Hsuan Hsieh & Jizhou Liu & Max Tabord-Meehan, 2022. "Revisiting the Analysis of Matched-Pair and Stratified Experiments in the Presence of Attrition," Papers 2209.11840, arXiv.org, revised Oct 2023.
    9. Lafortune, Jeanne & Pugatch, Todd & Tessada, José & Ubfal, Diego, 2022. "Can Interactive Online Training Make High School Students More Entrepreneurial? Experimental Evidence from Rwanda," IZA Discussion Papers 15064, Institute of Labor Economics (IZA).
    10. Fonseca, Julia & Matray, Adrien, 2024. "Financial inclusion, economic development, and inequality: Evidence from Brazil," Journal of Financial Economics, Elsevier, vol. 156(C).
    11. Fenoll, Ainoa Aparicio & Moscarola, Flavia Coda & Zaccagni, Sarah, 2021. "Mathematics camps: A gift for gifted students?," Journal of Economic Behavior & Organization, Elsevier, vol. 191(C), pages 738-751.
    12. Meinzen-Dick, Laura, 2020. "Decentralization and Elections in Burkina Faso," 2020 Annual Meeting, July 26-28, Kansas City, Missouri 304447, Agricultural and Applied Economics Association.
    13. Alice Guerra & Tatyana Zhuravleva, 2022. "Do women always behave as corruption cleaners?," Public Choice, Springer, vol. 191(1), pages 173-192, April.
    14. Ferman, Bruno & Lima, Lycia & Riva, Flávio, 2021. "Artificial Intelligence, Teacher Tasks and Individualized Pedagogy," SocArXiv qw249, Center for Open Science.
    15. Haoge Chang, 2023. "Design-based Estimation Theory for Complex Experiments," Papers 2311.06891, arXiv.org.
    16. Yuehao Bai & Jizhou Liu & Max Tabord-Meehan, 2022. "Inference for Matched Tuples and Fully Blocked Factorial Designs," Papers 2206.04157, arXiv.org, revised Nov 2023.
    17. Damon Jones & David Molitor & Julian Reif, 2024. "Incentives and Habit Formation in Health Screenings: Evidence from the Illinois Workplace Wellness Study," NBER Working Papers 32745, National Bureau of Economic Research, Inc.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Federico Bugni & Ivan Canay & Azeem Shaikh & Max Tabord-Meehan, 2022. "Inference for Cluster Randomized Experiments with Non-ignorable Cluster Sizes," Papers 2204.08356, arXiv.org, revised Apr 2024.
    2. Pedro Carneiro & Sokbae Lee & Daniel Wilhelm, 2020. "Optimal data collection for randomized control trials," The Econometrics Journal, Royal Economic Society, vol. 23(1), pages 1-31.
    3. Liang Jiang & Xiaobin Liu & Peter C. B. Phillips & Yichong Zhang, 2024. "Bootstrap Inference for Quantile Treatment Effects in Randomized Experiments with Matched Pairs," The Review of Economics and Statistics, MIT Press, vol. 106(2), pages 542-556, March.
    4. Yuehao Bai, 2022. "Optimality of Matched-Pair Designs in Randomized Controlled Trials," Papers 2206.07845, arXiv.org.
    5. Derksen, Laura & Leclerc, Catherine Michaud & Souza, Pedro CL, 2019. "Searching for Answers : The Impact of Student Access to Wikipedia," The Warwick Economics Research Paper Series (TWERPS) 1236, University of Warwick, Department of Economics.
    6. Bruno Ferman, 2019. "Assessing Inference Methods," Papers 1912.08772, arXiv.org, revised Oct 2022.
    7. MacKinnon, James G. & Nielsen, Morten Ørregaard & Webb, Matthew D., 2023. "Cluster-robust inference: A guide to empirical practice," Journal of Econometrics, Elsevier, vol. 232(2), pages 272-299.
    8. Jeffrey D. Michler & Anna Josephson, 2022. "Recent developments in inference: practicalities for applied economics," Chapters, in: A Modern Guide to Food Economics, chapter 11, pages 235-268, Edward Elgar Publishing.
    9. Yuehao Bai & Azeem M. Shaikh & Max Tabord-Meehan, 2024. "A Primer on the Analysis of Randomized Experiments and a Survey of some Recent Advances," Papers 2405.03910, arXiv.org.
    10. MacKinnon, James G. & Nielsen, Morten Ørregaard & Webb, Matthew D., 2023. "Testing for the appropriate level of clustering in linear regression models," Journal of Econometrics, Elsevier, vol. 235(2), pages 2027-2056.
    11. Derksen, Laura & Leclerc, Catherine Michaud & Souza, Pedro CL, 2019. "Searching for Answers: The Impact of Student Access to Wikipedia," CAGE Online Working Paper Series 450, Competitive Advantage in the Global Economy (CAGE).
    12. James G. MacKinnon & Matthew D. Webb, 2020. "When and How to Deal with Clustered Errors in Regression Models," Working Paper 1421, Economics Department, Queen's University.
    13. Cai, Shu, 2020. "Migration under liquidity constraints: Evidence from randomized credit access in China," Journal of Development Economics, Elsevier, vol. 142(C).
    14. Jonathan Fu & Annette Krauss, 2024. "Preparing fertile ground: how does the quality of business environments affect MSE growth?," Small Business Economics, Springer, vol. 63(1), pages 51-103, June.
    15. Nakano, Yuko & Magezi, Eustadius F., 2020. "The impact of microcredit on agricultural technology adoption and productivity: Evidence from randomized control trial in Tanzania," World Development, Elsevier, vol. 133(C).
    16. Dario Tortarolo & Guillermo Cruces & Gonzalo Vazquez-Bare, 2023. "Design of partial population experiments with an application to spillovers in tax compliance," IFS Working Papers W23/17, Institute for Fiscal Studies.
    17. Bai, Yuehao & Jiang, Liang & Romano, Joseph P. & Shaikh, Azeem M. & Zhang, Yichong, 2024. "Covariate adjustment in experiments with matched pairs," Journal of Econometrics, Elsevier, vol. 241(1).
    18. Victor Chernozhukov & Mert Demirer & Esther Duflo & Ivan Fernandez-Val, 2017. "Generic machine learning inference on heterogenous treatment effects in randomized experiments," CeMMAP working papers 61/17, Institute for Fiscal Studies.
    19. Yuehao Bai & Meng Hsuan Hsieh & Jizhou Liu & Max Tabord‐Meehan, 2024. "Revisiting the analysis of matched‐pair and stratified experiments in the presence of attrition," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 39(2), pages 256-268, March.
    20. Hoffmann, Vivian & Rao, Vijayendra & Surendra, Vaishnavi & Datta, Upamanyu, 2021. "Relief from usury: Impact of a self-help group lending program in rural India," Journal of Development Economics, Elsevier, vol. 148(C).

    More about this item

    Keywords

    clustered standard errors; clustering; paired experiments; stratified experiments; randomized experiments; RCT;
    All these keywords.

    JEL classification:

    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models
    • C90 - Mathematical and Quantitative Methods - - Design of Experiments - - - General
    • G21 - Financial Economics - - Financial Institutions and Services - - - Banks; Other Depository Institutions; Micro Finance Institutions; Mortgages
    • O16 - Economic Development, Innovation, Technological Change, and Growth - - Economic Development - - - Financial Markets; Saving and Capital Investment; Corporate Finance and Governance
    • O18 - Economic Development, Innovation, Technological Change, and Growth - - Economic Development - - - Urban, Rural, Regional, and Transportation Analysis; Housing; Infrastructure

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:hal:spmain:hal-03873897. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Contact - Sciences Po Departement of Economics (email available below). General contact details of provider: https://hal.archives-ouvertes.fr/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.