IDEAS home Printed from https://ideas.repec.org/p/cdp/texdis/td662.html
   My bibliography  Save this paper

Evaluating difference-in-differences models under different treatment assignment mechanism and in the presence of spillover effects

Author

Listed:
  • Guilherme Araújo Lima

    (UFMG)

  • Igor Viveiros Melo Souza

    (UFMG)

  • Mauro Sayar Ferreira

    (UFMG)

Abstract

We conduct Monte Carlo experiments to evaluate the performance of different Difference-in-Differences estimators under treatment assignment mechanisms affected by shocks suffered by treated units and also in contexts where the treatment effect spills over to units in the control group. In particular, we compare the estimators proposed by Callaway and Sant'Anna (2021), Borusyak et al. (2021), and Sun and Abraham (2021), as well as the two-way fixed effects (TWFE) estimator. The results demonstrate that the treatment assignment mechanisms we design, and the presence of spillover effects can severely compromise the performance of the considered estimators, leading to bias and, even more importantly, inconsistency. Therefore, cautious for interpreting the results should be taken in applications where the environment studied resembles those we consider. The development of more robust estimators is a necessity and a prosperous research venue.

Suggested Citation

  • Guilherme Araújo Lima & Igor Viveiros Melo Souza & Mauro Sayar Ferreira, 2023. "Evaluating difference-in-differences models under different treatment assignment mechanism and in the presence of spillover effects," Textos para Discussão Cedeplar-UFMG 662, Cedeplar, Universidade Federal de Minas Gerais.
    • Handle: RePEc:cdp:texdis:td662
    as

    Download full text from publisher

    File URL: https://www.cedeplar.ufmg.br/pesquisas/td/TD%20662.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Clément de Chaisemartin & Xavier D'Haultfœuille, 2020. "Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects," American Economic Review, American Economic Association, vol. 110(9), pages 2964-2996, September.
    2. Ashenfelter, Orley C, 1978. "Estimating the Effect of Training Programs on Earnings," The Review of Economics and Statistics, MIT Press, vol. 60(1), pages 47-57, February.
    3. Joshua D. Angrist & Jörn-Steffen Pischke, 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 3-30, Spring.
    4. Clément de Chaisemartin & Xavier D’Haultfœuille, 2023. "Two-way fixed effects and differences-in-differences with heterogeneous treatment effects: a survey," The Econometrics Journal, Royal Economic Society, vol. 26(3), pages 1-30.
    5. Lechner, Michael, 2011. "The Estimation of Causal Effects by Difference-in-Difference Methods," Foundations and Trends(R) in Econometrics, now publishers, vol. 4(3), pages 165-224, November.
    6. Callaway, Brantly & Sant’Anna, Pedro H.C., 2021. "Difference-in-Differences with multiple time periods," Journal of Econometrics, Elsevier, vol. 225(2), pages 200-230.
    7. Michelle Marcus & Pedro H. C. Sant’Anna, 2021. "The Role of Parallel Trends in Event Study Settings: An Application to Environmental Economics," Journal of the Association of Environmental and Resource Economists, University of Chicago Press, vol. 8(2), pages 235-275.
    8. Athey, Susan & Imbens, Guido W., 2022. "Design-based analysis in Difference-In-Differences settings with staggered adoption," Journal of Econometrics, Elsevier, vol. 226(1), pages 62-79.
    9. Goodman-Bacon, Andrew, 2021. "Difference-in-differences with variation in treatment timing," Journal of Econometrics, Elsevier, vol. 225(2), pages 254-277.
    10. Sun, Liyang & Abraham, Sarah, 2021. "Estimating dynamic treatment effects in event studies with heterogeneous treatment effects," Journal of Econometrics, Elsevier, vol. 225(2), pages 175-199.
    11. John Gardner, 2022. "Two-stage differences in differences," Papers 2207.05943, arXiv.org.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Roth, Jonathan & Sant’Anna, Pedro H.C. & Bilinski, Alyssa & Poe, John, 2023. "What’s trending in difference-in-differences? A synthesis of the recent econometrics literature," Journal of Econometrics, Elsevier, vol. 235(2), pages 2218-2244.
    2. Gregory Faletto, 2023. "Fused Extended Two-Way Fixed Effects for Difference-in-Differences With Staggered Adoptions," Papers 2312.05985, arXiv.org, revised Oct 2024.
    3. Arne Henningsen & Guy Low & David Wuepper & Tobias Dalhaus & Hugo Storm & Dagim Belay & Stefan Hirsch, 2024. "Estimating Causal Effects with Observational Data: Guidelines for Agricultural and Applied Economists," IFRO Working Paper 2024/03, University of Copenhagen, Department of Food and Resource Economics.
    4. Philipp Barteska & Jay Euijung Lee, 2024. "Bureaucrats and the Korean export miracle," Discussion Papers 2024-11, Nottingham Interdisciplinary Centre for Economic and Political Research (NICEP).
    5. Cl'ement de Chaisemartin & Xavier D'Haultf{oe}uille, 2021. "Two-Way Fixed Effects and Differences-in-Differences with Heterogeneous Treatment Effects: A Survey," Papers 2112.04565, arXiv.org, revised Jun 2022.
    6. Giulia Bettin & Isabella Giorgetti & Stefano Staffolani, 2024. "The impact of Covid-19 lockdown on the gender gap in the Italian labour market," Review of Economics of the Household, Springer, vol. 22(1), pages 1-33, March.
    7. Andrea Mattozzi & Samuel Nocito & Francesco Sobbrio, 2022. "Fact-Checking Politicians," CESifo Working Paper Series 10122, CESifo.
    8. Shanxia Sun & Michael S. Delgado, 2024. "Local spatial difference-in-differences models: treatment correlations, response interactions, and expanded local models," Empirical Economics, Springer, vol. 67(5), pages 2077-2107, November.
    9. Dalia Ghanem & Pedro H. C. Sant'Anna & Kaspar Wüthrich, 2022. "Selection and Parallel Trends," CESifo Working Paper Series 9910, CESifo.
    10. Clément de Chaisemartin & Xavier D’Haultfœuille, 2023. "Two-way fixed effects and differences-in-differences with heterogeneous treatment effects: a survey," The Econometrics Journal, Royal Economic Society, vol. 26(3), pages 1-30.
    11. Kotyrlo, Elena, 2024. "Simple and complex difference-in-differences approach," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 73, pages 119-142.
    12. Ridwan Ah Sheikh & Sunil Kanwar, 2024. "Revisiting the Impact of TRIPS on IPR-intensive Export Flows: Evidence from Staggered Difference-in-Differences," Working papers 351, Centre for Development Economics, Delhi School of Economics.
    13. Jon Ellingsen & Caroline Espegren, 2022. "Lost in transition? Earnings losses of displaced petroleum workers," Working Papers No 06/2022, Centre for Applied Macro- and Petroleum economics (CAMP), BI Norwegian Business School.
    14. Callaway, Brantly & Li, Tong, 2023. "Policy evaluation during a pandemic," Journal of Econometrics, Elsevier, vol. 236(1).
    15. Yadav, Anil & McHale, John & Harold, Jason & O'Neill, Stephen, 2024. "Estimating effects of staggered intervention with count and binary outcomes: a simulation study," Research Technical Papers 4/RT/24, Central Bank of Ireland.
    16. Mac Clay, Pablo & Börner, Jan & Sellare, Jorge, 2023. "Institutional and macroeconomic stability mediate the effect of auctions on renewable energy capacity," Energy Policy, Elsevier, vol. 180(C).
    17. Rik Chakraborti & Gavin Roberts, 2023. "How price-gouging regulation undermined COVID-19 mitigation: county-level evidence of unintended consequences," Public Choice, Springer, vol. 196(1), pages 51-83, July.
    18. Mayberry, Anthony A., 2023. "Demilitarization and economic growth: Empirical evidence in support of a peace dividend," Journal of Comparative Economics, Elsevier, vol. 51(3), pages 960-988.
    19. Simon Freyaldenhoven & Christian Hansen & Jorge Perez Perez & Jesse Shapiro, 2021. "Visualization, Identification, and stimation in the Linear Panel Event-Study Design," Working Papers 21-44, Federal Reserve Bank of Philadelphia.
    20. Kurt Schmidheiny & Sebastian Siegloch, 2023. "On event studies and distributed‐lags in two‐way fixed effects models: Identification, equivalence, and generalization," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(5), pages 695-713, August.

    More about this item

    Keywords

    Difference-in-Differences; Causal Inference; Treatment assignment mechanisms; Spillover effects.;
    All these keywords.

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cdp:texdis:td662. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Gustavo Britto (email available below). General contact details of provider: https://edirc.repec.org/data/pufmgbr.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.