IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2211.01547.html
   My bibliography  Save this paper

A Systematic Paradigm for Detecting, Surfacing, and Characterizing Heterogeneous Treatment Effects (HTE)

Author

Listed:
  • John Cai
  • Weinan Wang

Abstract

To effectively optimize and personalize treatments, it is necessary to investigate the heterogeneity of treatment effects. With the wide range of users being treated over many online controlled experiments, the typical approach of manually investigating each dimension of heterogeneity becomes overly cumbersome and prone to subjective human biases. We need an efficient way to search through thousands of experiments with hundreds of target covariates and hundreds of breakdown dimensions. In this paper, we propose a systematic paradigm for detecting, surfacing and characterizing heterogeneous treatment effects. First, we detect if treatment effect variation is present in an experiment, prior to specifying any breakdowns. Second, we surface the most relevant dimensions for heterogeneity. Finally, we characterize the heterogeneity beyond just the conditional average treatment effects (CATE) by studying the conditional distributions of the estimated individual treatment effects. We show the effectiveness of our methods using simulated data and empirical studies.

Suggested Citation

  • John Cai & Weinan Wang, 2022. "A Systematic Paradigm for Detecting, Surfacing, and Characterizing Heterogeneous Treatment Effects (HTE)," Papers 2211.01547, arXiv.org.
    • Handle: RePEc:arx:papers:2211.01547
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2211.01547
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Peng Ding & Avi Feller & Luke Miratrix, 2016. "Randomization inference for treatment effect variation," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 78(3), pages 655-671, June.
    2. Richard K. Crump & V. Joseph Hotz & Guido W. Imbens & Oscar A. Mitnik, 2008. "Nonparametric Tests for Treatment Effect Heterogeneity," The Review of Economics and Statistics, MIT Press, vol. 90(3), pages 389-405, August.
    3. Minsu Chang & Sokbae Lee & Yoon‐Jae Whang, 2015. "Nonparametric tests of conditional treatment effects with an application to single‐sex schooling on academic achievements," Econometrics Journal, Royal Economic Society, vol. 18(3), pages 307-346, October.
    4. Peng Ding & Avi Feller & Luke Miratrix, 2019. "Decomposing Treatment Effect Variation," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 114(525), pages 304-317, January.
    5. James J. Heckman & Jeffrey Smith & Nancy Clements, 1997. "Making The Most Out Of Programme Evaluations and Social Experiments: Accounting For Heterogeneity in Programme Impacts," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 64(4), pages 487-535.
    6. Fan, Yanqin & Park, Sang Soo, 2010. "Sharp Bounds On The Distribution Of Treatment Effects And Their Statistical Inference," Econometric Theory, Cambridge University Press, vol. 26(3), pages 931-951, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Nathan Kallus, 2023. "Treatment Effect Risk: Bounds and Inference," Management Science, INFORMS, vol. 69(8), pages 4579-4590, August.
    2. Nathan Kallus, 2022. "Treatment Effect Risk: Bounds and Inference," Papers 2201.05893, arXiv.org, revised Jul 2022.
    3. Sungwon Lee, 2024. "Partial identification and inference for conditional distributions of treatment effects," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 39(1), pages 107-127, January.
    4. Alejandro Sanchez-Becerra, 2023. "Robust inference for the treatment effect variance in experiments using machine learning," Papers 2306.03363, arXiv.org.
    5. Sungwon Lee, 2021. "Partial Identification and Inference for Conditional Distributions of Treatment Effects," Papers 2108.00723, arXiv.org, revised Nov 2023.
    6. Bedoya Arguelles,Guadalupe & Bittarello,Luca & Davis,Jonathan Martin Villars & Mittag,Nikolas Karl & Bedoya Arguelles,Guadalupe & Bittarello,Luca & Davis,Jonathan Martin Villars & Mittag,Nikolas Karl, 2017. "Distributional impact analysis: toolkit and illustrations of impacts beyond the average treatment effect," Policy Research Working Paper Series 8139, The World Bank.
    7. Sung Jae Jun & Yoonseok Lee & Youngki Shin, 2016. "Treatment Effects With Unobserved Heterogeneity: A Set Identification Approach," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 34(2), pages 302-311, April.
    8. Julius Owusu, 2024. "A Nonparametric Test of Heterogeneous Treatment Effects under Interference," Papers 2410.00733, arXiv.org.
    9. Ismaël Mourifié & Marc Henry & Romuald Méango, 2020. "Sharp Bounds and Testability of a Roy Model of STEM Major Choices," Journal of Political Economy, University of Chicago Press, vol. 128(8), pages 3220-3283.
    10. Pablo Lavado & Gonzalo Rivera, 2016. "Identifying Treatment Effects with Data Combination and Unobserved Heterogeneity," Working Papers 79, Peruvian Economic Association.
    11. Jiannan Lu & Peng Ding & Tirthankar Dasgupta, 2018. "Treatment Effects on Ordinal Outcomes: Causal Estimands and Sharp Bounds," Journal of Educational and Behavioral Statistics, , vol. 43(5), pages 540-567, October.
    12. Steven F. Lehrer & R. Vincent Pohl & Kyungchul Song, 2016. "Targeting Policies: Multiple Testing and Distributional Treatment Effects," NBER Working Papers 22950, National Bureau of Economic Research, Inc.
    13. Undral Byambadalai & Tatsushi Oka & Shota Yasui, 2024. "Estimating Distributional Treatment Effects in Randomized Experiments: Machine Learning for Variance Reduction," Papers 2407.16037, arXiv.org.
    14. Timothy B. Armstrong & Shu Shen, 2013. "Inference on Optimal Treatment Assignments," Cowles Foundation Discussion Papers 1927RR, Cowles Foundation for Research in Economics, Yale University, revised Apr 2015.
    15. Halbert White & Karim Chalak, 2013. "Identification and Identification Failure for Treatment Effects Using Structural Systems," Econometric Reviews, Taylor & Francis Journals, vol. 32(3), pages 273-317, November.
    16. Julius Owusu, 2023. "Randomization Inference of Heterogeneous Treatment Effects under Network Interference," Papers 2308.00202, arXiv.org, revised Jan 2024.
    17. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    18. Pereda-Fernández, Santiago, 2023. "Identification and estimation of triangular models with a binary treatment," Journal of Econometrics, Elsevier, vol. 234(2), pages 585-623.
    19. Pablo Lavado & Gonzalo Rivera, 2015. "Identifying treatment effects and counterfactual distributions using data combination with unobserved heterogeneity," Working Papers 15-14, Centro de Investigación, Universidad del Pacífico.
    20. Augsburg, Britta & Bancalari, Antonella & Durrani, Zara & Vaidyanathan, Madhav & White, Zach, 2022. "When nature calls back: Sustaining behavioral change in rural Pakistan," Journal of Development Economics, Elsevier, vol. 158(C).

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2211.01547. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.