IDEAS home Printed from https://ideas.repec.org/p/osf/osfxxx/ac5ru_v1.html
   My bibliography  Save this paper

How many is enough? Sample Size in Staggered Difference-in-Differences Designs

Author

Listed:
  • Egerod, Benjamin
  • Hollenbach, Florian M

    (Copenhagen Business School)

Abstract

In difference-in-differences designs with staggered treatment timing and dynamic treatment effects, the two-way fixed effects estimator fails to recover an interpretable causal estimate. A large number of estimators have been proposed to remedy this issue. The flexibility of these estimators, however, increases their variance. This can lead to statistical tests with low statistical power. As a consequence, small effects are unlikely to be discovered. Additionally, under low power, if a statistically significant estimate is recovered, the estimate is often wrongly signed and/or greatly exaggerated. Using simulations on real-world data on US States, we show that effect sizes of 10 to 15% are necessary for the recently developed estimators for staggered difference-in-differences to produce statistical tests that achieve 80% power. Further, conditional on statistical significance, when the intervention generates weak effects, estimators recover the wrong sign in approximately 10% of the simulations and overestimate the true effect by several hundred percent on average. We use data on publicly traded firms to investigate which sample size is needed for a staggered difference-in-differences analysis to be informative. We find that depending on the dependent variable and effect size, even the most efficient estimators generally need more than 250 units to achieve reasonable power. We conclude with a discussion of how this type of ‘design analysis’ ought to be used by researchers before estimating staggered difference-in-differences models. We also discuss how power may under certain conditions be improved if a study is re-designed, e.g., by examining county-level outcomes with state-level interventions.

Suggested Citation

  • Egerod, Benjamin & Hollenbach, Florian M, 2024. "How many is enough? Sample Size in Staggered Difference-in-Differences Designs," OSF Preprints ac5ru_v1, Center for Open Science.
  • Handle: RePEc:osf:osfxxx:ac5ru_v1
    DOI: 10.31219/osf.io/ac5ru_v1
    as

    Download full text from publisher

    File URL: https://osf.io/download/6672d409b5a03600dc9ff454/
    Download Restriction: no

    File URL: https://libkey.io/10.31219/osf.io/ac5ru_v1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Clément de Chaisemartin & Xavier D'Haultfœuille, 2020. "Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects," American Economic Review, American Economic Association, vol. 110(9), pages 2964-2996, September.
    2. Alberto Abadie & Susan Athey & Guido W. Imbens & Jeffrey M. Wooldridge, 2020. "Sampling‐Based versus Design‐Based Uncertainty in Regression Analysis," Econometrica, Econometric Society, vol. 88(1), pages 265-296, January.
    3. Alberto Abadie & Susan Athey & Guido W Imbens & Jeffrey M Wooldridge, 2023. "When Should You Adjust Standard Errors for Clustering?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 138(1), pages 1-35.
    4. Brewer Mike & Crossley Thomas F. & Joyce Robert, 2018. "Inference with Difference-in-Differences Revisited," Journal of Econometric Methods, De Gruyter, vol. 7(1), pages 1-16, January.
    5. de Chaisemartin, Clement & D'Haultfoeuille, Xavier, 2014. "Fuzzy Changes-in Changes," CAGE Online Working Paper Series 184, Competitive Advantage in the Global Economy (CAGE).
    6. Andrew Goodman-Bacon, 2018. "Difference-in-Differences with Variation in Treatment Timing," NBER Working Papers 25018, National Bureau of Economic Research, Inc.
    7. Duflo, Esther & Glennerster, Rachel & Kremer, Michael, 2008. "Using Randomization in Development Economics Research: A Toolkit," Handbook of Development Economics, in: T. Paul Schultz & John A. Strauss (ed.), Handbook of Development Economics, edition 1, volume 4, chapter 61, pages 3895-3962, Elsevier.
    8. Susan Athey & Mohsen Bayati & Nikolay Doudchenko & Guido Imbens & Khashayar Khosravi, 2021. "Matrix Completion Methods for Causal Panel Data Models," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(536), pages 1716-1730, October.
    9. C de Chaisemartin & X D’HaultfŒuille, 2018. "Fuzzy Differences-in-Differences," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 85(2), pages 999-1028.
    10. Marianne Bertrand & Esther Duflo & Sendhil Mullainathan, 2004. "How Much Should We Trust Differences-In-Differences Estimates?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 119(1), pages 249-275.
    11. Sun, Liyang & Abraham, Sarah, 2021. "Estimating dynamic treatment effects in event studies with heterogeneous treatment effects," Journal of Econometrics, Elsevier, vol. 225(2), pages 175-199.
    12. Black, Bernard & Hollingsworth, Alex & Nunes, Letícia & Simon, Kosali, 2022. "Simulated power analyses for observational studies: An application to the Affordable Care Act Medicaid expansion," Journal of Public Economics, Elsevier, vol. 213(C).
    13. Imai, Kosuke & Kim, In Song, 2021. "On the Use of Two-Way Fixed Effects Regression Models for Causal Inference with Panel Data," Political Analysis, Cambridge University Press, vol. 29(3), pages 405-415, July.
    14. Alberto Abadie & Susan Athey & Guido W. Imbens & Jeffrey M. Wooldridge, 2014. "Finite Population Causal Standard Errors," NBER Working Papers 20325, National Bureau of Economic Research, Inc.
    15. Campbell, Douglas & Brodeur, Abel & Dreber, Anna & Johannesson, Magnus & Kopecky, Joseph & Lusher, Lester & Tsoy, Nikita, 2024. "The Robustness Reproducibility of the American Economic Review," I4R Discussion Paper Series 124, The Institute for Replication (I4R).
    16. Dahal, Mahesh & Fiala, Nathan, 2020. "What do we know about the impact of microfinance? The problems of statistical power and precision," World Development, Elsevier, vol. 128(C).
    17. Baker, Andrew C. & Larcker, David F. & Wang, Charles C.Y., 2022. "How much should we trust staggered difference-in-differences estimates?," Journal of Financial Economics, Elsevier, vol. 144(2), pages 370-395.
    18. Callaway, Brantly & Sant’Anna, Pedro H.C., 2021. "Difference-in-Differences with multiple time periods," Journal of Econometrics, Elsevier, vol. 225(2), pages 200-230.
    19. Michelle Marcus & Pedro H. C. Sant’Anna, 2021. "The Role of Parallel Trends in Event Study Settings: An Application to Environmental Economics," Journal of the Association of Environmental and Resource Economists, University of Chicago Press, vol. 8(2), pages 235-275.
    20. Jonathan Roth, 2022. "Pretest with Caution: Event-Study Estimates after Testing for Parallel Trends," American Economic Review: Insights, American Economic Association, vol. 4(3), pages 305-322, September.
    21. Broockman, David E. & Kalla, Joshua L. & Sekhon, Jasjeet S., 2017. "The Design of Field Experiments With Survey Outcomes: A Framework for Selecting More Efficient, Robust, and Ethical Designs," Political Analysis, Cambridge University Press, vol. 25(4), pages 435-464, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hollenbach, Florian M & Egerod, Benjamin, 2024. "How many is enough? Sample Size in Staggered Difference-in-Differences Designs," OSF Preprints ac5ru, Center for Open Science.
    2. Roth, Jonathan & Sant’Anna, Pedro H.C. & Bilinski, Alyssa & Poe, John, 2023. "What’s trending in difference-in-differences? A synthesis of the recent econometrics literature," Journal of Econometrics, Elsevier, vol. 235(2), pages 2218-2244.
    3. Bruno Ferman, 2023. "Inference in difference‐in‐differences: How much should we trust in independent clusters?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(3), pages 358-369, April.
    4. Clément de Chaisemartin & Xavier D’Haultfœuille, 2023. "Two-way fixed effects and differences-in-differences with heterogeneous treatment effects: a survey," The Econometrics Journal, Royal Economic Society, vol. 26(3), pages 1-30.
    5. Dmitry Arkhangelsky & Guido Imbens, 2023. "Causal Models for Longitudinal and Panel Data: A Survey," Papers 2311.15458, arXiv.org, revised Jun 2024.
    6. Kotyrlo, Elena, 2024. "Simple and complex difference-in-differences approach," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 73, pages 119-142.
    7. Ashesh Rambachan & Jonathan Roth, 2020. "Design-Based Uncertainty for Quasi-Experiments," Papers 2008.00602, arXiv.org, revised Oct 2024.
    8. Simon Hartmann & Rok Spruk, 2023. "The impact of unilateral BIT terminations on FDI: Quasi-experimental evidence from India," The Review of International Organizations, Springer, vol. 18(2), pages 259-296, April.
    9. Peter Z. Schochet, 2021. "Statistical Power for Estimating Treatment Effects Using Difference-in-Differences and Comparative Interrupted Time Series Designs with Variation in Treatment Timing," Papers 2102.06770, arXiv.org, revised Oct 2021.
    10. Jack (Peiyao) Ma & Andrea Mantovani & Carlo Reggiani & Annette Broocks & Néstor Duch-Brown, 2024. "The Price Effects of Prohibiting Price Parity Clauses: Evidence from International Hotel Groups," Economics Series Working Papers 1043, University of Oxford, Department of Economics.
    11. Mikhail Mamonov & Anna Pestova & Steven Ongena, 2023. "“Crime and Punishment”? How Banks Anticipate and Propagate Global Financial Sanctions," CERGE-EI Working Papers wp753, The Center for Economic Research and Graduate Education - Economics Institute, Prague.
    12. Clément de Chaisemartin & Xavier D'Haultfœuille, 2020. "Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects," American Economic Review, American Economic Association, vol. 110(9), pages 2964-2996, September.
    13. Bhattacharjee, Arnab & Aravena, Claudia & Castillo, Natalia & Ehrlich, Marco & Taou, Nadia & Wagner, Thomas, 2022. "Agroforestry Programs in the Colombian Amazon: Selection, Treatment and Exposure Effects on Deforestation," National Institute of Economic and Social Research (NIESR) Discussion Papers 537, National Institute of Economic and Social Research.
    14. Eli Ben‐Michael & Avi Feller & Jesse Rothstein, 2022. "Synthetic controls with staggered adoption," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 84(2), pages 351-381, April.
    15. Brick, Kerri & De Martino, Samantha & Visser, Martine, 2023. "Behavioural nudges for water conservation in unequal settings: Experimental evidence from Cape Town," Journal of Environmental Economics and Management, Elsevier, vol. 121(C).
    16. Tamara Bischof & Boris Kaiser, 2021. "Who cares when you close down? The effects of primary care practice closures on patients," Health Economics, John Wiley & Sons, Ltd., vol. 30(9), pages 2004-2025, September.
    17. Goodman-Bacon, Andrew, 2021. "Difference-in-differences with variation in treatment timing," Journal of Econometrics, Elsevier, vol. 225(2), pages 254-277.
    18. Stefan Szymanski, 2023. "Anticipating the honeymoon: Event study estimation of new stadium effects in Major League Baseball using the imputation method," Economic Inquiry, Western Economic Association International, vol. 61(4), pages 1077-1102, October.
    19. Liyang Sun & Jesse M. Shapiro, 2022. "A Linear Panel Model with Heterogeneous Coefficients and Variation in Exposure," Journal of Economic Perspectives, American Economic Association, vol. 36(4), pages 193-204, Fall.
    20. Kamila Cygan-Rehm, 2022. "Lifetime Consequences of Lost Instructional Time in the Classroom: Evidence from Shortened School Years," CESifo Working Paper Series 9892, CESifo.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:osfxxx:ac5ru_v1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.