IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/20142.html
   My bibliography  Save this paper

Can Variation in Subgroups' Average Treatment Effects Explain Treatment Effect Heterogeneity? Evidence from a Social Experiment

Author

Listed:
  • Marianne P. Bitler
  • Jonah B. Gelbach
  • Hilary W. Hoynes

Abstract

In this paper, we assess whether welfare reform affects earnings only through mean impacts that are constant within but vary across subgroups. This is important because researchers interested in treatment effect heterogeneity typically restrict their attention to estimating mean impacts that are only allowed to vary across subgroups. Using a novel approach to simulating treatment group earnings under the constant mean-impacts within subgroup model, we find that this model does a poor job of capturing the treatment effect heterogeneity for Connecticut's Jobs First welfare reform experiment using quantile treatment effects. Notably, ignoring within-group heterogeneity would lead one to miss evidence that the Jobs First experiment's effects are consistent with central predictions of basic labor supply theory.

Suggested Citation

  • Marianne P. Bitler & Jonah B. Gelbach & Hilary W. Hoynes, 2014. "Can Variation in Subgroups' Average Treatment Effects Explain Treatment Effect Heterogeneity? Evidence from a Social Experiment," NBER Working Papers 20142, National Bureau of Economic Research, Inc.
  • Handle: RePEc:nbr:nberwo:20142
    Note: CH LS PE
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w20142.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Anderson, Michael L., 2008. "Multiple Inference and Gender Differences in the Effects of Early Intervention: A Reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects," Journal of the American Statistical Association, American Statistical Association, vol. 103(484), pages 1481-1495.
    2. Richard K. Crump & V. Joseph Hotz & Guido W. Imbens & Oscar A. Mitnik, 2008. "Nonparametric Tests for Treatment Effect Heterogeneity," The Review of Economics and Statistics, MIT Press, vol. 90(3), pages 389-405, August.
    3. Sergio Firpo, 2007. "Efficient Semiparametric Estimation of Quantile Treatment Effects," Econometrica, Econometric Society, vol. 75(1), pages 259-276, January.
    4. Alberto Abadie & Joshua Angrist & Guido Imbens, 2002. "Instrumental Variables Estimates of the Effect of Subsidized Training on the Quantiles of Trainee Earnings," Econometrica, Econometric Society, vol. 70(1), pages 91-117, January.
    5. James J. Heckman & Jeffrey Smith & Nancy Clements, 1997. "Making The Most Out Of Programme Evaluations and Social Experiments: Accounting For Heterogeneity in Programme Impacts," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 64(4), pages 487-535.
    6. James Heckman & Hidehiko Ichimura & Jeffrey Smith & Petra Todd, 1998. "Characterizing Selection Bias Using Experimental Data," Econometrica, Econometric Society, vol. 66(5), pages 1017-1098, September.
    7. Sergio Firpo & Nicole M. Fortin & Thomas Lemieux, 2009. "Unconditional Quantile Regressions," Econometrica, Econometric Society, vol. 77(3), pages 953-973, May.
    8. Patrick Kline & Melissa Tartari, 2016. "Bounding the Labor Supply Responses to a Randomized Welfare Experiment: A Revealed Preference Approach," American Economic Review, American Economic Association, vol. 106(4), pages 972-1014, April.
    9. Oliver Linton & Esfandiar Maasoumi & Yoon-Jae Whang, 2005. "Consistent Testing for Stochastic Dominance under General Sampling Schemes," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 72(3), pages 735-765.
    10. Charles F. Manski, 2004. "Statistical Treatment Rules for Heterogeneous Populations," Econometrica, Econometric Society, vol. 72(4), pages 1221-1246, July.
    11. Dehejia, Rajeev H., 2005. "Program evaluation as a decision problem," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 141-173.
    12. Bhattacharya, Debopam & Dupas, Pascaline, 2012. "Inferring welfare maximizing treatment assignment under budget constraints," Journal of Econometrics, Elsevier, vol. 167(1), pages 168-196.
    13. Roger Koenker & Zhijie Xiao, 2002. "Inference on the Quantile Regression Process," Econometrica, Econometric Society, vol. 70(4), pages 1583-1612, July.
    14. Abadie A., 2002. "Bootstrap Tests for Distributional Treatment Effects in Instrumental Variable Models," Journal of the American Statistical Association, American Statistical Association, vol. 97, pages 284-292, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Blog mentions

    As found by EconAcademics.org, the blog aggregator for Economics research:
    1. Sam Watson’s journal round up for 16th October 2017
      by Sam Watson in The Academic Health Economists' Blog on 2017-10-16 16:00:00

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Callaway, Brantly, 2021. "Bounds on distributional treatment effect parameters using panel data with an application on job displacement," Journal of Econometrics, Elsevier, vol. 222(2), pages 861-881.
    2. Lazuka, Volha, 2017. "The lasting health and income effects of public health formation in Sweden," Lund Papers in Economic History 153, Lund University, Department of Economic History.
    3. Robert Garlick, 2018. "Academic Peer Effects with Different Group Assignment Policies: Residential Tracking versus Random Assignment," American Economic Journal: Applied Economics, American Economic Association, vol. 10(3), pages 345-369, July.
    4. Alejandro Sanchez-Becerra, 2023. "Robust inference for the treatment effect variance in experiments using machine learning," Papers 2306.03363, arXiv.org.
    5. Xiao, Zhijie & Xu, Lan, 2019. "What do mean impacts miss? Distributional effects of corporate diversification," Journal of Econometrics, Elsevier, vol. 213(1), pages 92-120.
    6. Afrouz Azadikhah Jahromi & Brantly Callaway, 2022. "Heterogeneous Effects of Job Displacement on Earnings," Empirical Economics, Springer, vol. 62(1), pages 213-245, January.
    7. Gibbons Charles E. & Suárez Serrato Juan Carlos & Urbancic Michael B., 2019. "Broken or Fixed Effects?," Journal of Econometric Methods, De Gruyter, vol. 8(1), pages 1-12, January.
    8. Maike Hohberg & Peter Pütz & Thomas Kneib, 2020. "Treatment effects beyond the mean using distributional regression: Methods and guidance," PLOS ONE, Public Library of Science, vol. 15(2), pages 1-29, February.
    9. Deirdre Bloome & Daniel Schrage, 2021. "Covariance Regression Models for Studying Treatment Effect Heterogeneity Across One or More Outcomes: Understanding How Treatments Shape Inequality," Sociological Methods & Research, , vol. 50(3), pages 1034-1072, August.
    10. Xavier D’Haultfoeuille & Pauline Givord, 2014. "La régression quantile en pratique," Économie et Statistique, Programme National Persée, vol. 471(1), pages 85-111.
    11. Chung, EunYi & Olivares, Mauricio, 2021. "Permutation test for heterogeneous treatment effects with a nuisance parameter," Journal of Econometrics, Elsevier, vol. 225(2), pages 148-174.
    12. Jeffrey Smith, 2022. "Treatment Effect Heterogeneity," Evaluation Review, , vol. 46(5), pages 652-677, October.
    13. Likai Chen & Georg Keilbar & Liangjun Su & Weining Wang, 2023. "Inference on many jumps in nonparametric panel regression models," Papers 2312.01162, arXiv.org, revised Aug 2024.
    14. Horacio Alvarez Marinelli & Samuel Berlinski & Matias Busso, 2024. "Remedial Education: Evidence from a Sequence of Experiments in Colombia," Journal of Human Resources, University of Wisconsin Press, vol. 59(1), pages 141-174.
    15. Brigham R. Frandsen & Lars J. Lefgren, 2021. "Partial identification of the distribution of treatment effects with an application to the Knowledge is Power Program (KIPP)," Quantitative Economics, Econometric Society, vol. 12(1), pages 143-171, January.
    16. Strittmatter, Anthony, 2019. "Heterogeneous earnings effects of the job corps by gender: A translated quantile approach," Labour Economics, Elsevier, vol. 61(C).
    17. Matthew L. Comey & Amanda R. Eng & Zhuan Pei, 2022. "Supercompliers," Papers 2212.14105, arXiv.org, revised Aug 2023.
    18. Jaime Ramirez-Cuellar, 2023. "Testing for idiosyncratic Treatment Effect Heterogeneity," Papers 2304.01141, arXiv.org.
    19. Strittmatter, Anthony, 2023. "What is the value added by using causal machine learning methods in a welfare experiment evaluation?," Labour Economics, Elsevier, vol. 84(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    2. Victor Chernozhukov & Iván Fernández‐Val & Blaise Melly, 2013. "Inference on Counterfactual Distributions," Econometrica, Econometric Society, vol. 81(6), pages 2205-2268, November.
    3. Steven F. Lehrer & R. Vincent Pohl & Kyungchul Song, 2016. "Targeting Policies: Multiple Testing and Distributional Treatment Effects," NBER Working Papers 22950, National Bureau of Economic Research, Inc.
    4. Steven F. Lehrer & R. Vincent Pohl & Kyungchul Song, 2022. "Multiple Testing and the Distributional Effects of Accountability Incentives in Education," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 40(4), pages 1552-1568, October.
    5. Peng Ding & Avi Feller & Luke Miratrix, 2016. "Randomization inference for treatment effect variation," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 78(3), pages 655-671, June.
    6. Sokbae Lee & Yoon-Jae Whang, 2009. "Nonparametric Tests of Conditional Treatment Effects," Cowles Foundation Discussion Papers 1740, Cowles Foundation for Research in Economics, Yale University.
    7. Qu, Zhongjun & Yoon, Jungmo, 2015. "Nonparametric estimation and inference on conditional quantile processes," Journal of Econometrics, Elsevier, vol. 185(1), pages 1-19.
    8. Timothy B. Armstrong & Shu Shen, 2013. "Inference on Optimal Treatment Assignments," Cowles Foundation Discussion Papers 1927RR, Cowles Foundation for Research in Economics, Yale University, revised Apr 2015.
    9. Firpo, Sergio & Galvao, Antonio F. & Kobus, Martyna & Parker, Thomas & Rosa-Dias, Pedro, 2020. "Loss Aversion and the Welfare Ranking of Policy Interventions," IZA Discussion Papers 13176, Institute of Labor Economics (IZA).
    10. Ferreira, Francisco H. G. & Firpo, Sergio & Galvao, Antonio F., 2017. "Estimation and Inference for Actual and Counterfactual Growth Incidence Curves," IZA Discussion Papers 10473, Institute of Labor Economics (IZA).
    11. Abhijit V. Banerjee & Esther Duflo, 2009. "The Experimental Approach to Development Economics," Annual Review of Economics, Annual Reviews, vol. 1(1), pages 151-178, May.
    12. Zongwu Cai & Ying Fang & Ming Lin & Shengfang Tang, 2021. "A Nonparametric Test for Testing Heterogeneity in Conditional Quantile Treatment Effects," WORKING PAPERS SERIES IN THEORETICAL AND APPLIED ECONOMICS 202117, University of Kansas, Department of Economics, revised Aug 2021.
    13. Ozkan Eren & Serkan Ozbeklik, 2014. "Who Benefits From Job Corps? A Distributional Analysis Of An Active Labor Market Program," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 29(4), pages 586-611, June.
    14. Marianne P. Bitler & Jonah B. Gelbach & Hilary W. Hoynes, 2006. "What Mean Impacts Miss: Distributional Effects of Welfare Reform Experiments," American Economic Review, American Economic Association, vol. 96(4), pages 988-1012, September.
    15. Zongwu Cai & Ying Fang & Ming Lin & Shengfang Tang, 2020. "Inferences for Partially Conditional Quantile Treatment Effect Model," WORKING PAPERS SERIES IN THEORETICAL AND APPLIED ECONOMICS 202005, University of Kansas, Department of Economics, revised Feb 2020.
    16. Susan Athey & Guido Imbens, 2016. "The Econometrics of Randomized Experiments," Papers 1607.00698, arXiv.org.
    17. Callaway, Brantly & Li, Tong & Oka, Tatsushi, 2018. "Quantile treatment effects in difference in differences models under dependence restrictions and with only two time periods," Journal of Econometrics, Elsevier, vol. 206(2), pages 395-413.
    18. Sakos, Grayson & Cerulli, Giovanni & Garbero, Alessandra, 2021. "Beyond the ATE: Idiosyncratic Effect Estimation to Uncover Distributional Impacts Results from 17 Impact Evaluations," 2021 Annual Meeting, August 1-3, Austin, Texas 314017, Agricultural and Applied Economics Association.
    19. Undral Byambadalai & Tatsushi Oka & Shota Yasui, 2024. "Estimating Distributional Treatment Effects in Randomized Experiments: Machine Learning for Variance Reduction," Papers 2407.16037, arXiv.org.
    20. Xavier D’Haultfoeuille & Pauline Givord, 2014. "La régression quantile en pratique," Économie et Statistique, Programme National Persée, vol. 471(1), pages 85-111.

    More about this item

    JEL classification:

    • H75 - Public Economics - - State and Local Government; Intergovernmental Relations - - - State and Local Government: Health, Education, and Welfare
    • I38 - Health, Education, and Welfare - - Welfare, Well-Being, and Poverty - - - Government Programs; Provision and Effects of Welfare Programs
    • J18 - Labor and Demographic Economics - - Demographic Economics - - - Public Policy

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:20142. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.