IDEAS home Printed from https://ideas.repec.org/a/spr/jlabrs/v57y2023i1d10.1186_s12651-023-00352-9.html
   My bibliography  Save this article

How sensitive are matching estimates of active labor market policy effects to typically unobserved confounders?

Author

Listed:
  • Stefan Tübbicke

    (Institute for Employment Research (IAB))

Abstract

Using a rich and unique combined administrative-survey dataset, this paper explores how sensitive propensity score (PS) matching estimates of Active Labor Market Policies (ALMPs) based on the selection-on-observables assumption are to typically unobserved covariates. Using a sample of German unemployed welfare recipients, the analysis shows that typically unobserved factors such job search behavior, concessions willing to make for a job as well as (mental) health are in fact relevant confounders. However, results also show that matching on the PS using only typically observed covariates reduces imbalance in terms of typically unobserved covariates by about 46 percent in this setting. In line with this finding, the inclusion of typically unobserved covariates yields very similar estimates to estimates based on a standard specification. Hence, a standard matching approach based on rich and high quality administrative data appears to be sufficient to obtain estimates that are rather robust to unobserved confounding.

Suggested Citation

  • Stefan Tübbicke, 2023. "How sensitive are matching estimates of active labor market policy effects to typically unobserved confounders?," Journal for Labour Market Research, Springer;Institute for Employment Research/ Institut für Arbeitsmarkt- und Berufsforschung (IAB), vol. 57(1), pages 1-16, December.
  • Handle: RePEc:spr:jlabrs:v:57:y:2023:i:1:d:10.1186_s12651-023-00352-9
    DOI: 10.1186/s12651-023-00352-9
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1186/s12651-023-00352-9
    File Function: Abstract
    Download Restriction: no

    File URL: https://libkey.io/10.1186/s12651-023-00352-9?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Lechner, Michael & Wunsch, Conny, 2013. "Sensitivity of matching-based program evaluations to the availability of control variables," Labour Economics, Elsevier, vol. 21(C), pages 111-121.
    2. Lichter, Andreas & Schiprowski, Amelie, 2021. "Benefit duration, job search behavior and re-employment," Journal of Public Economics, Elsevier, vol. 193(C).
    3. Andrey Fradkin & Frédéric Panier & Ilan Tojerow, 2019. "Blame the Parents? How Parental Unemployment Affects Labor Supply and Job Quality for Young Adults," Journal of Labor Economics, University of Chicago Press, vol. 37(1), pages 35-100.
    4. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
    5. Caliendo, Marco & Mahlstedt, Robert & Mitnik, Oscar A., 2017. "Unobservable, but unimportant? The relevance of usually unobserved variables for the evaluation of labor market policies," Labour Economics, Elsevier, vol. 46(C), pages 14-25.
    6. Andrea Ichino & Fabrizia Mealli & Tommaso Nannicini, 2008. "From temporary help jobs to permanent employment: what can we learn from matching estimators and their sensitivity?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 23(3), pages 305-327.
    7. Patrick Bayer & Stephen L. Ross & Giorgio Topa, 2008. "Place of Work and Place of Residence: Informal Hiring Networks and Labor Market Outcomes," Journal of Political Economy, University of Chicago Press, vol. 116(6), pages 1150-1196, December.
    8. Bernd Fitzenberger & Aderonke Osikominu & Robert Völter, 2008. "Get Training or Wait? Long-Run Employment Effects of Training Programs for the Unemployed in West Germany," Annals of Economics and Statistics, GENES, issue 91-92, pages 321-355.
    9. Böheim, Renè & Horvath, Gerard Thomas & Winter-Ebmer, Rudolf, 2011. "Great expectations: Past wages and unemployment durations," Labour Economics, Elsevier, vol. 18(6), pages 778-785.
    10. Gerard J. van den Berg & Arne Uhlendorff & Joachim Wolff, 2022. "The Impact of Sanctions for Young Welfare Recipients on Transitions to Work and Wages, and on Dropping Out," Economica, London School of Economics and Political Science, vol. 89(353), pages 1-28, January.
    11. Michael Lechner & Conny Wunsch, 2009. "Are Training Programs More Effective When Unemployment Is High?," Journal of Labor Economics, University of Chicago Press, vol. 27(4), pages 653-692, October.
    12. Hugo Bodory & Lorenzo Camponovo & Martin Huber & Michael Lechner, 2020. "The Finite Sample Performance of Inference Methods for Propensity Score Matching and Weighting Estimators," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 38(1), pages 183-200, January.
    13. Gerard J. Berg & Johan Vikström, 2014. "Monitoring Job Offer Decisions, Punishments, Exit to Work, and Job Quality," Scandinavian Journal of Economics, Wiley Blackwell, vol. 116(2), pages 284-334, April.
    14. JAMES G. MacKINNON, 2006. "Bootstrap Methods in Econometrics," The Economic Record, The Economic Society of Australia, vol. 82(s1), pages 2-18, September.
    15. Caliendo, Marco & Künn, Steffen & Uhlendorff, Arne, 2016. "Earnings exemptions for unemployed workers: The relationship between marginal employment, unemployment duration and job quality," Labour Economics, Elsevier, vol. 42(C), pages 177-193.
    16. Emily Oster, 2019. "Unobservable Selection and Coefficient Stability: Theory and Evidence," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 37(2), pages 187-204, April.
    17. Caliendo, Marco & Künn, Steffen, 2011. "Start-up subsidies for the unemployed: Long-term evidence and effect heterogeneity," Journal of Public Economics, Elsevier, vol. 95(3), pages 311-331.
    18. Marco Caliendo & Sabine Kopeinig, 2008. "Some Practical Guidance For The Implementation Of Propensity Score Matching," Journal of Economic Surveys, Wiley Blackwell, vol. 22(1), pages 31-72, February.
    19. Alan B. Krueger & Andreas I. Mueller, 2016. "A Contribution to the Empirics of Reservation Wages," American Economic Journal: Economic Policy, American Economic Association, vol. 8(1), pages 142-179, February.
    20. Schubert, Michael & Parthier, Katrin & Kupka, Peter & Krüger, Ulrich & Holke, Jörg & Fuchs, Philipp, 2013. "Menschen mit psychischen Störungen im SGB II," IAB-Forschungsbericht 201312, Institut für Arbeitsmarkt- und Berufsforschung (IAB), Nürnberg [Institute for Employment Research, Nuremberg, Germany].
    21. Altmann, Steffen & Falk, Armin & Jäger, Simon & Zimmermann, Florian, 2018. "Learning about job search: A field experiment with job seekers in Germany," Journal of Public Economics, Elsevier, vol. 164(C), pages 33-49.
    22. Huber Martin & Lechner Michael & Wunsch Conny & Walter Thomas, 2011. "Do German Welfare-to-Work Programmes Reduce Welfare Dependency and Increase Employment?," German Economic Review, De Gruyter, vol. 12(2), pages 182-204, May.
    23. Torsten Lietzmann & Paul Schmelzer & Jürgen Wiemers, 2017. "Marginal employment for welfare recipients: stepping stone or obstacle?," LABOUR, CEIS, vol. 31(4), pages 394-414, December.
    24. Tamara Harrer & Andreas Moczall & Joachim Wolff, 2020. "Free, free, set them free? Are programmes effective that allow job centres considerable freedom to choose the exact design?," International Journal of Social Welfare, John Wiley & Sons, vol. 29(2), pages 154-167, April.
    25. Gerard J. van den Berg & Annette H. Bergemann & Marco Caliendo, 2009. "The Effect of Active Labor Market Programs on Not-Yet Treated Unemployed Individuals," Journal of the European Economic Association, MIT Press, vol. 7(2-3), pages 606-616, 04-05.
    26. Arni, Patrick & Schiprowski, Amelie, 2019. "Job search requirements, effort provision and labor market outcomes," Journal of Public Economics, Elsevier, vol. 169(C), pages 65-88.
    27. Michael Lechner, 2002. "Some practical issues in the evaluation of heterogeneous labour market programmes by matching methods," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 165(1), pages 59-82, February.
    28. John V. Pepper, 2000. "The Intergenerational Transmission Of Welfare Receipt: A Nonparametric Bounds Analysis," The Review of Economics and Statistics, MIT Press, vol. 82(3), pages 472-488, August.
    29. Fitzenberger, Bernd & Volter, Robert, 2007. "Long-run effects of training programs for the unemployed in East Germany," Labour Economics, Elsevier, vol. 14(4), pages 730-755, August.
    30. Montgomery, James D, 1991. "Social Networks and Labor-Market Outcomes: Toward an Economic Analysis," American Economic Review, American Economic Association, vol. 81(5), pages 1407-1418, December.
    31. James Heckman & Hidehiko Ichimura & Jeffrey Smith & Petra Todd, 1998. "Characterizing Selection Bias Using Experimental Data," Econometrica, Econometric Society, vol. 66(5), pages 1017-1098, September.
    32. Rose, Damaris & Stavrova, Olga, 2019. "Does life satisfaction predict reemployment? Evidence form German panel data," Journal of Economic Psychology, Elsevier, vol. 72(C), pages 1-11.
    33. Martin Huber & Michael Lechner & Andreas Steinmayr, 2015. "Radius matching on the propensity score with bias adjustment: tuning parameters and finite sample behaviour," Empirical Economics, Springer, vol. 49(1), pages 1-31, August.
    34. Heckman, James J. & Robb, Richard Jr., 1985. "Alternative methods for evaluating the impact of interventions : An overview," Journal of Econometrics, Elsevier, vol. 30(1-2), pages 239-267.
    35. Lietzmann, Torsten & Schmelzer, Paul & Wiemers, Jürgen, 2017. "Marginal employment for welfare recipients: stepping stone or obstacle?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 31(4), pages 394-414.
    36. A. D. Roy, 1951. "Some Thoughts On The Distribution Of Earnings," Oxford Economic Papers, Oxford University Press, vol. 3(2), pages 135-146.
    37. Jeffrey J. Yankow, 2003. "Migration, Job Change, and Wage Growth: A New Perspective on the Pecuniary Return to Geographic Mobility," Journal of Regional Science, Wiley Blackwell, vol. 43(3), pages 483-516, August.
    38. Pilar García-Gómez & Hans van Kippersluis & Owen O’Donnell & Eddy van Doorslaer, 2013. "Long-Term and Spillover Effects of Health Shocks on Employment and Income," Journal of Human Resources, University of Wisconsin Press, vol. 48(4), pages 873-909.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Caliendo, Marco & Mahlstedt, Robert & Mitnik, Oscar A., 2017. "Unobservable, but unimportant? The relevance of usually unobserved variables for the evaluation of labor market policies," Labour Economics, Elsevier, vol. 46(C), pages 14-25.
    2. Marco Caliendo & Stefan Tübbicke, 2019. "Do Start-Up Subsidies for the Unemployed Affect Participants’ Well-Being? A Rigorous Look at (Un-)Intended Consequences of Labor Market Policies," CEPA Discussion Papers 14, Center for Economic Policy Analysis.
    3. Marco Caliendo & Steffen Künn, 2015. "Getting back into the labor market: the effects of start-up subsidies for unemployed females," Journal of Population Economics, Springer;European Society for Population Economics, vol. 28(4), pages 1005-1043, October.
    4. Marco Caliendo & Stefan Tübbicke, 2020. "New evidence on long-term effects of start-up subsidies: matching estimates and their robustness," Empirical Economics, Springer, vol. 59(4), pages 1605-1631, October.
    5. Lutz Bellmann & Marco Caliendo & Stefan Tübbicke, 2018. "The Post‐Reform Effectiveness of the New German Start‐Up Subsidy for the Unemployed," LABOUR, CEIS, vol. 32(3), pages 293-319, September.
    6. Schmidl, Ricarda, 2015. "The Effectiveness of Early Vacancy Information in the Presence of Monitoring and ALMP," IZA Discussion Papers 9575, Institute of Labor Economics (IZA).
    7. Goller, Daniel & Lechner, Michael & Moczall, Andreas & Wolff, Joachim, 2020. "Does the estimation of the propensity score by machine learning improve matching estimation? The case of Germany's programmes for long term unemployed," Labour Economics, Elsevier, vol. 65(C).
    8. Caliendo, Marco & Künn, Steffen, 2011. "Start-up subsidies for the unemployed: Long-term evidence and effect heterogeneity," Journal of Public Economics, Elsevier, vol. 95(3), pages 311-331.
    9. Hagen, Tobias, 2016. "Econometric Evaluation of a Placement Coaching Program for Recipients of Disability Insurance Benefits in Switzerland," VfS Annual Conference 2016 (Augsburg): Demographic Change 145736, Verein für Socialpolitik / German Economic Association.
    10. Ulf Rinne & Arne Uhlendorff & Zhong Zhao, 2013. "Vouchers and caseworkers in training programs for the unemployed," Empirical Economics, Springer, vol. 45(3), pages 1089-1127, December.
    11. Hagen, Tobias, 2016. "Econometric evaluation of a placement coaching program for recipients of disability insurance benefits in Switzerland," Working Paper Series 10, Frankfurt University of Applied Sciences, Faculty of Business and Law.
    12. Lombardi, Stefano & van den Berg, Gerard J. & Vikström, Johan, 2020. "Empirical Monte Carlo evidence on estimation of Timing-of-Events models," Working Paper Series 2020:26, IFAU - Institute for Evaluation of Labour Market and Education Policy, revised 05 Jan 2021.
    13. Volpe Martincus, Christian & Carballo, Jerónimo, 2010. "Export Promotion: Heterogeneous Programs and Heterogeneous Effects," IDB Publications (Working Papers) 3176, Inter-American Development Bank.
    14. Caliendo, Marco & Künn, Steffen & Weißenberger, Martin, 2016. "Personality traits and the evaluation of start-up subsidies," European Economic Review, Elsevier, vol. 86(C), pages 87-108.
    15. Manuela Deidda & Adriana Di Liberto & Marta Foddi & Giovanni Sulis, 2015. "Employment subsidies, informal economy and women’s transition into work in a depressed area: evidence from a matching approach," IZA Journal of Labor Policy, Springer;Forschungsinstitut zur Zukunft der Arbeit GmbH (IZA), vol. 4(1), pages 1-25, December.
    16. Jeffrey Smith & Arthur Sweetman, 2016. "Viewpoint: Estimating the causal effects of policies and programs," Canadian Journal of Economics, Canadian Economics Association, vol. 49(3), pages 871-905, August.
    17. Marco Caliendo & Steffen K�nn, 2014. "Regional Effect Heterogeneity of Start-up Subsidies for the Unemployed," Regional Studies, Taylor & Francis Journals, vol. 48(6), pages 1108-1134, June.
    18. Christian Volpe Martincus & Jerónimo Carballo, 2012. "Export promotion activities in developing countries: What kind of trade do they promote?," The Journal of International Trade & Economic Development, Taylor & Francis Journals, vol. 21(4), pages 539-578, June.
    19. Christian Volpe Martincus, 2010. "Odyssey in International Markets: An Assessment of the Effectiveness of Export Promotion in Latin America and the Caribbean," IDB Publications (Books), Inter-American Development Bank, number 16458, February.
    20. Lechner, Michael & Wunsch, Conny, 2013. "Sensitivity of matching-based program evaluations to the availability of control variables," Labour Economics, Elsevier, vol. 21(C), pages 111-121.

    More about this item

    Keywords

    Propensity score matching; Observational studies; Selection bias; Active labor market policy; Evaluation;
    All these keywords.

    JEL classification:

    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models
    • D04 - Microeconomics - - General - - - Microeconomic Policy: Formulation; Implementation; Evaluation
    • J68 - Labor and Demographic Economics - - Mobility, Unemployment, Vacancies, and Immigrant Workers - - - Public Policy

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:jlabrs:v:57:y:2023:i:1:d:10.1186_s12651-023-00352-9. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.