IDEAS home Printed from https://ideas.repec.org/p/cdp/texdis/td662.html
   My bibliography  Save this paper

Evaluating difference-in-differences models under different treatment assignment mechanism and in the presence of spillover effects

Author

Listed:
  • Guilherme Araújo Lima

    (UFMG)

  • Igor Viveiros Melo Souza

    (UFMG)

  • Mauro Sayar Ferreira

    (UFMG)

Abstract

We conduct Monte Carlo experiments to evaluate the performance of different Difference-in-Differences estimators under treatment assignment mechanisms affected by shocks suffered by treated units and also in contexts where the treatment effect spills over to units in the control group. In particular, we compare the estimators proposed by Callaway and Sant'Anna (2021), Borusyak et al. (2021), and Sun and Abraham (2021), as well as the two-way fixed effects (TWFE) estimator. The results demonstrate that the treatment assignment mechanisms we design, and the presence of spillover effects can severely compromise the performance of the considered estimators, leading to bias and, even more importantly, inconsistency. Therefore, cautious for interpreting the results should be taken in applications where the environment studied resembles those we consider. The development of more robust estimators is a necessity and a prosperous research venue.

Suggested Citation

  • Guilherme Araújo Lima & Igor Viveiros Melo Souza & Mauro Sayar Ferreira, 2023. "Evaluating difference-in-differences models under different treatment assignment mechanism and in the presence of spillover effects," Textos para Discussão Cedeplar-UFMG 662, Cedeplar, Universidade Federal de Minas Gerais.
    • Handle: RePEc:cdp:texdis:td662
    as

    Download full text from publisher

    File URL: https://www.cedeplar.ufmg.br/pesquisas/td/TD%20662.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Clément de Chaisemartin & Xavier D'Haultfœuille, 2020. "Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects," American Economic Review, American Economic Association, vol. 110(9), pages 2964-2996, September.
    2. Ashenfelter, Orley C, 1978. "Estimating the Effect of Training Programs on Earnings," The Review of Economics and Statistics, MIT Press, vol. 60(1), pages 47-57, February.
    3. Joshua D. Angrist & Jörn-Steffen Pischke, 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 3-30, Spring.
    4. Clément de Chaisemartin & Xavier D’Haultfœuille, 2023. "Two-way fixed effects and differences-in-differences with heterogeneous treatment effects: a survey," The Econometrics Journal, Royal Economic Society, vol. 26(3), pages 1-30.
    5. Lechner, Michael, 2011. "The Estimation of Causal Effects by Difference-in-Difference Methods," Foundations and Trends(R) in Econometrics, now publishers, vol. 4(3), pages 165-224, November.
    6. Callaway, Brantly & Sant’Anna, Pedro H.C., 2021. "Difference-in-Differences with multiple time periods," Journal of Econometrics, Elsevier, vol. 225(2), pages 200-230.
    7. Michelle Marcus & Pedro H. C. Sant’Anna, 2021. "The Role of Parallel Trends in Event Study Settings: An Application to Environmental Economics," Journal of the Association of Environmental and Resource Economists, University of Chicago Press, vol. 8(2), pages 235-275.
    8. Sun, Liyang & Abraham, Sarah, 2021. "Estimating dynamic treatment effects in event studies with heterogeneous treatment effects," Journal of Econometrics, Elsevier, vol. 225(2), pages 175-199.
    9. John Gardner, 2022. "Two-stage differences in differences," Papers 2207.05943, arXiv.org.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Roth, Jonathan & Sant’Anna, Pedro H.C. & Bilinski, Alyssa & Poe, John, 2023. "What’s trending in difference-in-differences? A synthesis of the recent econometrics literature," Journal of Econometrics, Elsevier, vol. 235(2), pages 2218-2244.
    2. Ridwan Ah Sheikh & Sunil Kanwar, 2024. "Revisiting the Impact of TRIPS on IPR-intensive Export Flows: Evidence from Staggered Difference-in-Differences," Working papers 351, Centre for Development Economics, Delhi School of Economics.
    3. Dalia Ghanem & Pedro H. C. Sant'Anna & Kaspar Wuthrich, 2022. "Selection and parallel trends," Papers 2203.09001, arXiv.org, revised Mar 2024.
    4. Yadav, Anil & McHale, John & Harold, Jason & O'Neill, Stephen, 2024. "Estimating effects of staggered intervention with count and binary outcomes: a simulation study," Research Technical Papers 4/RT/24, Central Bank of Ireland.
    5. Gregory Faletto, 2023. "Fused Extended Two-Way Fixed Effects for Difference-in-Differences With Staggered Adoptions," Papers 2312.05985, arXiv.org, revised Oct 2024.
    6. Alex Hollingsworth & Krzysztof Karbownik & Melissa A. Thomasson & Anthony Wray, 2024. "The Gift of a Lifetime: The Hospital, Modern Medicine, and Mortality," American Economic Review, American Economic Association, vol. 114(7), pages 2201-2238, July.
    7. Simon Freyaldenhoven & Christian Hansen & Jorge Perez Perez & Jesse Shapiro, 2021. "Visualization, Identification, and stimation in the Linear Panel Event-Study Design," Working Papers 21-44, Federal Reserve Bank of Philadelphia.
    8. Bhattacharjee, Arnab & Aravena, Claudia & Castillo, Natalia & Ehrlich, Marco & Taou, Nadia & Wagner, Thomas, 2022. "Agroforestry Programs in the Colombian Amazon: Selection, Treatment and Exposure Effects on Deforestation," National Institute of Economic and Social Research (NIESR) Discussion Papers 537, National Institute of Economic and Social Research.
    9. Mark Kattenberg & Bas Scheer & Jurre Thiel, 2023. "Causal forests with fixed effects for treatment effect heterogeneity in difference-in-differences," CPB Discussion Paper 452, CPB Netherlands Bureau for Economic Policy Analysis.
    10. Philipp Barteska & Jay Euijung Lee, 2024. "Bureaucrats and the Korean export miracle," Discussion Papers 2024-11, Nottingham Interdisciplinary Centre for Economic and Political Research (NICEP).
    11. Rotunno, Lorenzo & Roy, Sanchari & Sakakibara, Anri & Vezina, Pierre-Louis, 2023. "Trade Policy and Jobs in Vietnam: The Unintended Consequences of Trump’s Trade War," SocArXiv 9rdne, Center for Open Science.
    12. Balmford, Ben & Collins, Joseph & Day, Brett & Lindsay, Luke & Peacock, James, 2023. "Pricing rules for PES auctions: Evidence from a natural experiment," Journal of Environmental Economics and Management, Elsevier, vol. 122(C).
    13. Nieto, Adrián, 2022. "Can subsidies to permanent employment change fertility decisions?," Labour Economics, Elsevier, vol. 78(C).
    14. Bruno Ferman, 2023. "Inference in difference‐in‐differences: How much should we trust in independent clusters?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(3), pages 358-369, April.
    15. Chareyron, Sylvain & Goffette-Nagot, Florence & Letrouit, Lucie, 2022. "Externalities from urban renewal: Evidence from a French program," Regional Science and Urban Economics, Elsevier, vol. 95(C).
    16. Mensah, Edouard R. & Filipski, Mateusz J., 2022. "Saving for a rainy day: the impact of natural disasters on savings rates," 2022 Annual Meeting, July 31-August 2, Anaheim, California 322266, Agricultural and Applied Economics Association.
    17. Rannveig Kaldager Hart & Janna Bergsvik & Agnes Fauske & Wookun Kim, 2023. "Causal Analysis of Policy Effects on Fertility," CESifo Working Paper Series 10690, CESifo.
    18. Francesca Truffa & Ashley Wong, 2024. "Undergraduate Gender Diversity and the Direction of Scientific Research," CESifo Working Paper Series 11294, CESifo.
    19. de Chaisemartin, Clément & D’Haultfœuille, Xavier, 2023. "Two-way fixed effects and differences-in-differences estimators with several treatments," Journal of Econometrics, Elsevier, vol. 236(2).
    20. Hollenbach, Florian M & Egerod, Benjamin, 2024. "How many is enough? Sample Size in Staggered Difference-in-Differences Designs," OSF Preprints ac5ru, Center for Open Science.

    More about this item

    Keywords

    Difference-in-Differences; Causal Inference; Treatment assignment mechanisms; Spillover effects.;
    All these keywords.

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cdp:texdis:td662. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Gustavo Britto (email available below). General contact details of provider: https://edirc.repec.org/data/pufmgbr.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.