IDEAS home Printed from https://ideas.repec.org/p/iza/izadps/dp15580.html
   My bibliography  Save this paper

Effect or Treatment Heterogeneity? Policy Evaluation with Aggregated and Disaggregated Treatments

Author

Listed:
  • Heiler, Phillip

    (Aarhus University)

  • Knaus, Michael C.

    (University of Tübingen)

Abstract

Binary treatments are often ex-post aggregates of multiple treatments or can be disaggregated into multiple treatment versions. Thus, effects can be heterogeneous due to either effect or treatment heterogeneity. We propose a decomposition method that uncovers masked heterogeneity, avoids spurious discoveries, and evaluates treatment assignment quality. The estimation and inference procedure based on double/debiased machine learning allows for high-dimensional confounding, many treatments and extreme propensity scores. Our applications suggest that heterogeneous effects of smoking on birthweight are partially due to different smoking intensities and that gender gaps in Job Corps effectiveness are largely explained by differences in vocational training.

Suggested Citation

  • Heiler, Phillip & Knaus, Michael C., 2022. "Effect or Treatment Heterogeneity? Policy Evaluation with Aggregated and Disaggregated Treatments," IZA Discussion Papers 15580, Institute of Labor Economics (IZA).
    • Handle: RePEc:iza:izadps:dp15580
    as

    Download full text from publisher

    File URL: https://docs.iza.org/dp15580.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Sokbae Lee & Ryo Okui & Yoon†Jae Whang, 2017. "Doubly robust uniform confidence band for the conditional average treatment effect function," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(7), pages 1207-1225, November.
    2. Victor Chernozhukov & Denis Chetverikov & Mert Demirer & Esther Duflo & Christian Hansen & Whitney Newey & James Robins, 2018. "Double/debiased machine learning for treatment and structural parameters," Econometrics Journal, Royal Economic Society, vol. 21(1), pages 1-68, February.
    3. Heiler, Phillip & Kazak, Ekaterina, 2021. "Valid inference for treatment effect parameters under irregular identification and many extreme propensity scores," Journal of Econometrics, Elsevier, vol. 222(2), pages 1083-1108.
    4. Ozkan Eren & Serkan Ozbeklik, 2014. "Who Benefits From Job Corps? A Distributional Analysis Of An Active Labor Market Program," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 29(4), pages 586-611, June.
    5. S. Derya Uysal, 2015. "Doubly Robust Estimation of Causal Effects with Multivalued Treatments: An Application to the Returns to Schooling," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 30(5), pages 763-786, August.
    6. James J. Heckman, 1991. "Randomization and Social Policy Evaluation Revisited," NBER Technical Working Papers 0107, National Bureau of Economic Research, Inc.
    7. Jonathan M.V. Davis & Sara B. Heller, 2020. "Rethinking the Benefits of Youth Employment Programs: The Heterogeneous Effects of Summer Jobs," The Review of Economics and Statistics, MIT Press, vol. 102(4), pages 664-677, October.
    8. Lu Tian & Ash A. Alizadeh & Andrew J. Gentles & Robert Tibshirani, 2014. "A Simple Method for Estimating Interactions Between a Treatment and a Large Number of Covariates," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 109(508), pages 1517-1532, December.
    9. Matias D. Cattaneo, 2010. "multi-valued treatment effects," The New Palgrave Dictionary of Economics,, Palgrave Macmillan.
    10. Michael C Knaus & Michael Lechner & Anthony Strittmatter, 2021. "Machine learning estimation of heterogeneous causal effects: Empirical Monte Carlo evidence," The Econometrics Journal, Royal Economic Society, vol. 24(1), pages 134-161.
    11. Christoph Rothe, 2017. "Robust Confidence Intervals for Average Treatment Effects Under Limited Overlap," Econometrica, Econometric Society, vol. 85, pages 645-660, March.
    12. Belloni, Alexandre & Chernozhukov, Victor & Chetverikov, Denis & Kato, Kengo, 2015. "Some new asymptotic theory for least squares series: Pointwise and uniform results," Journal of Econometrics, Elsevier, vol. 186(2), pages 345-366.
    13. Jorg Stoye, 2009. "More on Confidence Intervals for Partially Identified Parameters," Econometrica, Econometric Society, vol. 77(4), pages 1299-1315, July.
    14. Jason Abrevaya & Yu-Chin Hsu & Robert P. Lieli, 2015. "Estimating Conditional Average Treatment Effects," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 33(4), pages 485-505, October.
    15. Xinwei Ma & Jingshen Wang, 2020. "Robust Inference Using Inverse Probability Weighting," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 115(532), pages 1851-1860, December.
    16. Fan Li & Kari Lock Morgan & Alan M. Zaslavsky, 2018. "Balancing Covariates via Propensity Score Weighting," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(521), pages 390-400, January.
    17. Michael Lechner, 2002. "Program Heterogeneity And Propensity Score Matching: An Application To The Evaluation Of Active Labor Market Policies," The Review of Economics and Statistics, MIT Press, vol. 84(2), pages 205-220, May.
    18. Almond, Douglas & Currie, Janet, 2011. "Human Capital Development before Age Five," Handbook of Labor Economics, in: O. Ashenfelter & D. Card (ed.), Handbook of Labor Economics, edition 1, volume 4, chapter 15, pages 1315-1486, Elsevier.
    19. Jason Abrevaya, 2006. "Estimating the effect of smoking on birth outcomes using a matched panel data approach," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 21(4), pages 489-519.
    20. Shakeeb Khan & Elie Tamer, 2010. "Irregular Identification, Support Conditions, and Inverse Weight Estimation," Econometrica, Econometric Society, vol. 78(6), pages 2021-2042, November.
    21. repec:mpr:mprres:6097 is not listed on IDEAS
    22. Ye Luo & Martin Spindler & Jannis Kuck, 2016. "High-Dimensional $L_2$Boosting: Rate of Convergence," Papers 1602.08927, arXiv.org, revised Jul 2022.
    23. Alexandre Belloni & Victor Chernozhukov & Ying Wei, 2016. "Post-Selection Inference for Generalized Linear Models With Many Controls," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 34(4), pages 606-619, October.
    24. Michael Zimmert & Michael Lechner, 2019. "Nonparametric estimation of causal heterogeneity under high-dimensional confounding," Papers 1908.08779, arXiv.org.
    25. Joseph Hotz, V. & Imbens, Guido W. & Mortimer, Julie H., 2005. "Predicting the efficacy of future training programs using past experiences at other locations," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 241-270.
    26. Peter Z. Schochet & John Burghardt & Steven Glazerman, 2001. "National Job Corps Study: The Impacts of Job Corps on Participants' Employment and Related Outcomes," Mathematica Policy Research Reports db6c4204b8e1408bb0c6289ec, Mathematica Policy Research.
    27. Stefan Wager & Susan Athey, 2018. "Estimation and Inference of Heterogeneous Treatment Effects using Random Forests," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(523), pages 1228-1242, July.
    28. Michael C Knaus, 2022. "Double machine learning-based programme evaluation under unconfoundedness [Econometric methods for program evaluation]," The Econometrics Journal, Royal Economic Society, vol. 25(3), pages 602-627.
    29. Susan Athey & Guido W. Imbens, 2017. "The State of Applied Econometrics: Causality and Policy Evaluation," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 3-32, Spring.
    30. Vira Semenova & Victor Chernozhukov, 2021. "Debiased machine learning of conditional average treatment effects and other causal functions," The Econometrics Journal, Royal Economic Society, vol. 24(2), pages 264-289.
    31. Cattaneo, Matias D., 2010. "Efficient semiparametric estimation of multi-valued treatment effects under ignorability," Journal of Econometrics, Elsevier, vol. 155(2), pages 138-154, April.
    32. Max H. Farrell & Tengyuan Liang & Sanjog Misra, 2021. "Deep Neural Networks for Estimation and Inference," Econometrica, Econometric Society, vol. 89(1), pages 181-213, January.
    33. Han Hong & Michael P Leung & Jessie Li, 2020. "Inference on finite-population treatment effects under limited overlap," The Econometrics Journal, Royal Economic Society, vol. 23(1), pages 32-47.
    34. X Nie & S Wager, 2021. "Quasi-oracle estimation of heterogeneous treatment effects [TensorFlow: A system for large-scale machine learning]," Biometrika, Biometrika Trust, vol. 108(2), pages 299-319.
    35. Damian Kozbur, 2020. "Analysis of Testing‐Based Forward Model Selection," Econometrica, Econometric Society, vol. 88(5), pages 2147-2173, September.
    36. Newey, Whitney K., 1997. "Convergence rates and asymptotic normality for series estimators," Journal of Econometrics, Elsevier, vol. 79(1), pages 147-168, July.
    37. Strittmatter, Anthony, 2019. "Heterogeneous earnings effects of the job corps by gender: A translated quantile approach," Labour Economics, Elsevier, vol. 61(C).
    38. Farrell, Max H., 2015. "Robust inference on average treatment effects with possibly more covariates than observations," Journal of Econometrics, Elsevier, vol. 189(1), pages 1-23.
    39. Peter Z. Schochet & John Burghardt & Sheena McConnell, 2008. "Does Job Corps Work? Impact Findings from the National Job Corps Study," American Economic Review, American Economic Association, vol. 98(5), pages 1864-1886, December.
    40. Marshall, John, 2016. "Coarsening Bias: How Coarse Treatment Measurement Upwardly Biases Instrumental Variable Estimates," Political Analysis, Cambridge University Press, vol. 24(2), pages 157-171, April.
    41. Anthony Strittmatter, 2019. "Heterogeneous Earnings Effects of the Job Corps by Gender Earnings: A Translated Quantile Approach," Papers 1908.08721, arXiv.org.
    42. Qingliang Fan & Yu-Chin Hsu & Robert P. Lieli & Yichong Zhang, 2022. "Estimation of Conditional Average Treatment Effects With High-Dimensional Data," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 40(1), pages 313-327, January.
    43. Martin E Andresen & Martin Huber, 2021. "Instrument-based estimation with binarised treatments: issues and tests for the exclusion restriction," The Econometrics Journal, Royal Economic Society, vol. 24(3), pages 536-558.
    44. Carlos A. Flores & Alfonso Flores-Lagunes & Arturo Gonzalez & Todd C. Neumann, 2012. "Estimating the Effects of Length of Exposure to Instruction in a Training Program: The Case of Job Corps," The Review of Economics and Statistics, MIT Press, vol. 94(1), pages 153-171, February.
    45. Hande Inanc & Karen Needels & Jillian Berk, "undated". "Gender Segregation in Training Programs and the Wage Gap (Issue Brief)," Mathematica Policy Research Reports 80be3cf3d04b4f2f8b9c88e14, Mathematica Policy Research.
    46. Phillip Heiler, 2022. "Heterogeneous Treatment Effect Bounds under Sample Selection with an Application to the Effects of Social Media on Political Polarization," Papers 2209.04329, arXiv.org, revised Jul 2024.
    47. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, September.
    48. repec:mpr:mprres:2951 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Patrick Rehill & Nicholas Biddle, 2023. "Fairness Implications of Heterogeneous Treatment Effect Estimation with Machine Learning Methods in Policy-making," Papers 2309.00805, arXiv.org.
    2. Dan A. Black & Lars Skipper & Jeffrey A. Smith & Jeffrey Andrew Smith, 2023. "Firm Training," CESifo Working Paper Series 10268, CESifo.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ganesh Karapakula, 2023. "Stable Probability Weighting: Large-Sample and Finite-Sample Estimation and Inference Methods for Heterogeneous Causal Effects of Multivalued Treatments Under Limited Overlap," Papers 2301.05703, arXiv.org, revised Jan 2023.
    2. Michael Zimmert & Michael Lechner, 2019. "Nonparametric estimation of causal heterogeneity under high-dimensional confounding," Papers 1908.08779, arXiv.org.
    3. Phillip Heiler, 2022. "Heterogeneous Treatment Effect Bounds under Sample Selection with an Application to the Effects of Social Media on Political Polarization," Papers 2209.04329, arXiv.org, revised Jul 2024.
    4. Riccardo Di Francesco, 2022. "Aggregation Trees," CEIS Research Paper 546, Tor Vergata University, CEIS, revised 20 Nov 2023.
    5. Riccardo Di Francesco, 2024. "Aggregation Trees," Papers 2410.11408, arXiv.org.
    6. Michael C Knaus, 2022. "Double machine learning-based programme evaluation under unconfoundedness [Econometric methods for program evaluation]," The Econometrics Journal, Royal Economic Society, vol. 25(3), pages 602-627.
    7. Michael Lechner & Jana Mareckova, 2024. "Comprehensive Causal Machine Learning," Papers 2405.10198, arXiv.org.
    8. Daniel Goller, 2023. "Analysing a built-in advantage in asymmetric darts contests using causal machine learning," Annals of Operations Research, Springer, vol. 325(1), pages 649-679, June.
    9. Adam Baybutt & Manu Navjeevan, 2023. "Doubly-Robust Inference for Conditional Average Treatment Effects with High-Dimensional Controls," Papers 2301.06283, arXiv.org.
    10. Rahul Singh & Liyuan Xu & Arthur Gretton, 2020. "Kernel Methods for Causal Functions: Dose, Heterogeneous, and Incremental Response Curves," Papers 2010.04855, arXiv.org, revised Oct 2022.
    11. Gabriel Okasa, 2022. "Meta-Learners for Estimation of Causal Effects: Finite Sample Cross-Fit Performance," Papers 2201.12692, arXiv.org.
    12. Nora Bearth & Michael Lechner, 2024. "Causal Machine Learning for Moderation Effects," Papers 2401.08290, arXiv.org, revised Apr 2024.
    13. Kyle Colangelo & Ying-Ying Lee, 2020. "Double Debiased Machine Learning Nonparametric Inference with Continuous Treatments," Papers 2004.03036, arXiv.org, revised Sep 2023.
    14. Difang Huang & Jiti Gao & Tatsushi Oka, 2022. "Semiparametric Single-Index Estimation for Average Treatment Effects," Papers 2206.08503, arXiv.org, revised Apr 2024.
    15. Yang Ning & Sida Peng & Jing Tao, 2020. "Doubly Robust Semiparametric Difference-in-Differences Estimators with High-Dimensional Data," Papers 2009.03151, arXiv.org.
    16. Michael C Knaus & Michael Lechner & Anthony Strittmatter, 2021. "Machine learning estimation of heterogeneous causal effects: Empirical Monte Carlo evidence," The Econometrics Journal, Royal Economic Society, vol. 24(1), pages 134-161.
    17. Michael Lechner & Jana Mareckova, 2022. "Modified Causal Forest," Papers 2209.03744, arXiv.org.
    18. Su, Liangjun & Ura, Takuya & Zhang, Yichong, 2019. "Non-separable models with high-dimensional data," Journal of Econometrics, Elsevier, vol. 212(2), pages 646-677.
    19. Heejun Shin & Joseph Antonelli, 2023. "Improved inference for doubly robust estimators of heterogeneous treatment effects," Biometrics, The International Biometric Society, vol. 79(4), pages 3140-3152, December.
    20. Nan Liu & Yanbo Liu & Yuya Sasaki, 2024. "Estimation and Inference for Causal Functions with Multiway Clustered Data," Papers 2409.06654, arXiv.org.

    More about this item

    Keywords

    causal inference; causal machine learning; double machine learning; heterogeneous treatment effects; overlap; treatment versions;
    All these keywords.

    JEL classification:

    • C14 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Semiparametric and Nonparametric Methods: General
    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp15580. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Holger Hinte (email available below). General contact details of provider: https://edirc.repec.org/data/izaaade.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.