IDEAS home Printed from https://ideas.repec.org/p/tse/wpaper/26567.html
   My bibliography  Save this paper

Matching vs Differencing when Estimating Treatment Effects with Panel Data: the Example of the Effect of Job Training Programs on Earnings

Author

Listed:
  • Chabé-Ferret, Sylvain

Abstract

This paper compares matching and Difference-In-Difference matching (DID) when estimating the effect of a program on a dynamic outcome. I detail the sources of bias of each estimator in a model of entry into a Job Training Program (JTP) and earnings dynamics that I use as a working example. I show that there are plausible settings in which DID is consistent while matching on past outcomes is not. Unfortunately, the consistency of both estimators relies on conditions that are at odds with properties of earnings dynamics. Using calibration and Monte-Carlo simulations, I show that deviations from the most favorable conditions severely bias both estimators. The behavior of matching is nevertheless less erratic: its bias generally decreases when controlling for more past outcomes and it generally provides a lower bound on the true treatment effect. I finally point to previously unnoticed empirical results that confirm that DID does well, and generally better than matching on past outcomes, at replicating the results of an experimental benchmark.

Suggested Citation

  • Chabé-Ferret, Sylvain, 2012. "Matching vs Differencing when Estimating Treatment Effects with Panel Data: the Example of the Effect of Job Training Programs on Earnings," TSE Working Papers 12-356, Toulouse School of Economics (TSE).
    • Handle: RePEc:tse:wpaper:26567
    as

    Download full text from publisher

    File URL: http://www.tse-fr.eu/sites/default/files/medias/doc/wp/env/wp_tse_356.pdf
    File Function: Full text
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
    2. Mayer, Thierry & Mayneris, Florian & Py, Loriane, 2012. "The Impact of Urban Enterprise Zones on Establishments' Location Decisions: Evidence from French ZFUs," CEPR Discussion Papers 9074, C.E.P.R. Discussion Papers.
    3. Gobillon, Laurent & Magnac, Thierry & Selod, Harris, 2012. "Do unemployed workers benefit from enterprise zones? The French experience," Journal of Public Economics, Elsevier, vol. 96(9-10), pages 881-892.
    4. Flavio Cunha & James Heckman & Salvador Navarro, 2005. "Separating uncertainty from heterogeneity in life cycle earnings," Oxford Economic Papers, Oxford University Press, vol. 57(2), pages 191-261, April.
    5. Martin Browning & Mette Ejrnæs & Javier Alvarez, 2010. "Modelling Income Processes with Lots of Heterogeneity," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 77(4), pages 1353-1381.
    6. Ashenfelter, Orley C, 1978. "Estimating the Effect of Training Programs on Earnings," The Review of Economics and Statistics, MIT Press, vol. 60(1), pages 47-57, February.
    7. Markus Frlich, 2004. "Finite-Sample Properties of Propensity-Score Matching and Weighting Estimators," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 77-90, February.
    8. Balineau, Gaëlle, 2013. "Disentangling the Effects of Fair Trade on the Quality of Malian Cotton," World Development, Elsevier, vol. 44(C), pages 241-255.
    9. Keisuke Hirano & Guido W. Imbens & Geert Ridder, 2003. "Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score," Econometrica, Econometric Society, vol. 71(4), pages 1161-1189, July.
    10. Jay Bhattacharya & William B. Vogt, 2007. "Do Instrumental Variables Belong in Propensity Scores?," NBER Technical Working Papers 0343, National Bureau of Economic Research, Inc.
    11. Barry Arnold & Robert Beaver & Richard Groeneveld & William Meeker, 1993. "The nontruncated marginal of a truncated bivariate normal distribution," Psychometrika, Springer;The Psychometric Society, vol. 58(3), pages 471-488, September.
    12. Ziebarth, Nicolas R. & Karlsson, Martin, 2010. "A natural experiment on sick pay cuts, sickness absence, and labor costs," Journal of Public Economics, Elsevier, vol. 94(11-12), pages 1108-1122, December.
    13. MaCurdy, Thomas E., 1982. "The use of time series processes to model the error structure of earnings in a longitudinal data analysis," Journal of Econometrics, Elsevier, vol. 18(1), pages 83-114, January.
    14. Puhani, Patrick A. & Sonderhof, Katja, 2010. "The effects of a sick pay reform on absence and on health-related outcomes," Journal of Health Economics, Elsevier, vol. 29(2), pages 285-302, March.
    15. Meghir, Costas & Pistaferri, Luigi, 2011. "Earnings, Consumption and Life Cycle Choices," Handbook of Labor Economics, in: O. Ashenfelter & D. Card (ed.), Handbook of Labor Economics, edition 1, volume 4, chapter 9, pages 773-854, Elsevier.
    16. Heckman, James J. & Lalonde, Robert J. & Smith, Jeffrey A., 1999. "The economics and econometrics of active labor market programs," Handbook of Labor Economics, in: O. Ashenfelter & D. Card (ed.), Handbook of Labor Economics, edition 1, volume 3, chapter 31, pages 1865-2097, Elsevier.
    17. Jinyong Hahn, 1998. "On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects," Econometrica, Econometric Society, vol. 66(2), pages 315-332, March.
    18. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    19. Fatih Guvenen, 2007. "Learning Your Earning: Are Labor Income Shocks Really Very Persistent?," American Economic Review, American Economic Association, vol. 97(3), pages 687-712, June.
    20. Wooldridge, Jeffrey M., 2005. "Violating Ignorability Of Treatment By Controlling For Too Many Factors," Econometric Theory, Cambridge University Press, vol. 21(5), pages 1026-1028, October.
    21. Heckman, J.J. & Hotz, V.J., 1988. "Choosing Among Alternative Nonexperimental Methods For Estimating The Impact Of Social Programs: The Case Of Manpower Training," University of Chicago - Economics Research Center 88-12, Chicago - Economics Research Center.
    22. James Heckman & Salvador Navarro-Lozano, 2004. "Using Matching, Instrumental Variables, and Control Functions to Estimate Economic Choice Models," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 30-57, February.
    23. Alberto Abadie, 2005. "Semiparametric Difference-in-Differences Estimators," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 72(1), pages 1-19.
    24. James Heckman & Hidehiko Ichimura & Jeffrey Smith & Petra Todd, 1998. "Characterizing Selection Bias Using Experimental Data," Econometrica, Econometric Society, vol. 66(5), pages 1017-1098, September.
    25. James J. Heckman & Hidehiko Ichimura & Petra Todd, 1998. "Matching As An Econometric Evaluation Estimator," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 65(2), pages 261-294.
    26. Dmytro Hryshko, 2012. "Labor income profiles are not heterogeneous: Evidence from income growth rates," Quantitative Economics, Econometric Society, vol. 3(2), pages 177-209, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Fredrik Andersson & Harry J. Holzer & Julia I. Lane & David Rosenblum & Jeffrey Smith, 2024. "Does Federally Funded Job Training Work? Nonexperimental Estimates of WIA Training Impacts Using Longitudinal Data on Workers and Firms," Journal of Human Resources, University of Wisconsin Press, vol. 59(4), pages 1244-1283.
    2. Pei‐Yu Chi & Ting‐Yu Chang & Kuo‐I Chang, 2022. "Evaluating the impact of preferential trade agreement on fishery imports: An application of difference‐in‐differences with matching method," Agricultural Economics, International Association of Agricultural Economists, vol. 53(1), pages 90-124, January.
    3. Cisilino, Federica & Bodini, Antonella & Zanoli, Agostina & Lasorella, Maria Valentina, 2018. "Exploring Agri-environmental effectiveness using counterfactual analysis," 162nd Seminar, April 26-27, 2018, Budapest, Hungary 271958, European Association of Agricultural Economists.
    4. Andrea ALBANESE & Bart COCKX, 2015. "Permanent Wage Cost Subsidies for Older Workers. An Effective Tool for Increasing Working Time and Postponing Early Retirement?," LIDAM Discussion Papers IRES 2015006, Université catholique de Louvain, Institut de Recherches Economiques et Sociales (IRES).
    5. von Eije, Henk & Goyal, Abhinav & Muckley, Cal B., 2014. "Does the information content of payout initiations and omissions influence firm risks?," Journal of Econometrics, Elsevier, vol. 183(2), pages 222-229.
    6. Lechner, Michael, 2013. "Treatment effects and panel data," Economics Working Paper Series 1314, University of St. Gallen, School of Economics and Political Science.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chabé-Ferret, Sylvain, 2015. "Analysis of the bias of Matching and Difference-in-Difference under alternative earnings and selection processes," Journal of Econometrics, Elsevier, vol. 185(1), pages 110-123.
    2. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    3. Carlos A. Flores & Oscar A. Mitnik, 2009. "Evaluating Nonexperimental Estimators for Multiple Treatments: Evidence from Experimental Data," Working Papers 2010-10, University of Miami, Department of Economics.
    4. Huber, Martin & Lechner, Michael & Wunsch, Conny, 2010. "How to Control for Many Covariates? Reliable Estimators Based on the Propensity Score," IZA Discussion Papers 5268, Institute of Labor Economics (IZA).
    5. Peter R. Mueser & Kenneth R. Troske & Alexey Gorislavsky, 2007. "Using State Administrative Data to Measure Program Performance," The Review of Economics and Statistics, MIT Press, vol. 89(4), pages 761-783, November.
    6. Richard Blundell & Monica Costa Dias, 2009. "Alternative Approaches to Evaluation in Empirical Microeconomics," Journal of Human Resources, University of Wisconsin Press, vol. 44(3).
    7. James J. Heckman, 2008. "The Principles Underlying Evaluation Estimators with an Application to Matching," Annals of Economics and Statistics, GENES, issue 91-92, pages 9-73.
    8. Dettmann, E. & Becker, C. & Schmeißer, C., 2011. "Distance functions for matching in small samples," Computational Statistics & Data Analysis, Elsevier, vol. 55(5), pages 1942-1960, May.
    9. Huber, Martin & Lechner, Michael & Wunsch, Conny, 2013. "The performance of estimators based on the propensity score," Journal of Econometrics, Elsevier, vol. 175(1), pages 1-21.
    10. Heckman, James J., 2010. "The Assumptions Underlying Evaluation Estimators," Brazilian Review of Econometrics, Sociedade Brasileira de Econometria - SBE, vol. 30(2), December.
    11. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
    12. Frölich, Markus & Huber, Martin & Wiesenfarth, Manuel, 2017. "The finite sample performance of semi- and non-parametric estimators for treatment effects and policy evaluation," Computational Statistics & Data Analysis, Elsevier, vol. 115(C), pages 91-102.
    13. John C. Ham & Xianghong Li & Patricia B. Reagan, 2004. "Propensity Score Matching, a Distance-Based Measure of Migration, and the Wage Growth of Young Men," Working Papers 2004_3, York University, Department of Economics.
    14. Lechner, Michael & Wunsch, Conny, 2013. "Sensitivity of matching-based program evaluations to the availability of control variables," Labour Economics, Elsevier, vol. 21(C), pages 111-121.
    15. Jones A.M & Rice N, 2009. "Econometric Evaluation of Health Policies," Health, Econometrics and Data Group (HEDG) Working Papers 09/09, HEDG, c/o Department of Economics, University of York.
    16. Zhao, Zhong, 2008. "Sensitivity of propensity score methods to the specifications," Economics Letters, Elsevier, vol. 98(3), pages 309-319, March.
    17. James J. Heckman & Petra E. Todd, 2009. "A note on adapting propensity score matching and selection models to choice based samples," Econometrics Journal, Royal Economic Society, vol. 12(s1), pages 230-234, January.
    18. Advani, Arun & Sloczynski, Tymon, 2013. "Mostly Harmless Simulations? On the Internal Validity of Empirical Monte Carlo Studies," IZA Discussion Papers 7874, Institute of Labor Economics (IZA).
    19. Chabé-Ferret, Sylvain, 2017. "Should We Combine Difference In Differences with Conditioning on Pre-Treatment Outcomes?," TSE Working Papers 17-824, Toulouse School of Economics (TSE).
    20. Dehejia Rajeev, 2015. "Experimental and Non-Experimental Methods in Development Economics: A Porous Dialectic," Journal of Globalization and Development, De Gruyter, vol. 6(1), pages 47-69, June.

    More about this item

    Keywords

    Matching - Difference in Difference - Evaluation of Job training Programs;

    JEL classification:

    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models
    • C23 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Models with Panel Data; Spatio-temporal Models

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:tse:wpaper:26567. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/tsetofr.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.