IDEAS home Printed from https://ideas.repec.org/a/bla/jorssa/v184y2021i2p504-521.html
   My bibliography  Save this article

Beyond generalization of the ATE: Designing randomized trials to understand treatment effect heterogeneity

Author

Listed:
  • Elizabeth Tipton

Abstract

Researchers conducting randomized trials have increasingly shifted focus from the average treatment effect to understanding moderators of treatment effects. Current methods for exploring moderation focus on model selection and hypothesis tests. At the same time, recent developments in the design of randomized trials have argued for the need for population‐based recruitment in order to generalize well. In this paper, we show that a different population‐based recruitment strategy can be implemented to increase the precision of estimates of treatment effect moderators, and we explore the trade‐offs between optimal designs for the average treatment effect and moderator effects.

Suggested Citation

  • Elizabeth Tipton, 2021. "Beyond generalization of the ATE: Designing randomized trials to understand treatment effect heterogeneity," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 184(2), pages 504-521, April.
    • Handle: RePEc:bla:jorssa:v:184:y:2021:i:2:p:504-521
    DOI: 10.1111/rssa.12629
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/rssa.12629
    Download Restriction: no
    File URL: https://libkey.io/10.1111/rssa.12629?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Robert B. Olsen & Larry L. Orr & Stephen H. Bell & Elizabeth A. Stuart, 2013. "External Validity in Policy Evaluations That Choose Sites Purposively," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 32(1), pages 107-121, January.
    2. Elizabeth A. Stuart & Stephen R. Cole & Catherine P. Bradshaw & Philip J. Leaf, 2011. "The use of propensity scores to assess the generalizability of results from randomized trials," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 174(2), pages 369-386, April.
    3. Erin Hartman & Richard Grieve & Roland Ramsahai & Jasjeet S. Sekhon, 2015. "From sample average treatment effect to population average treatment effect on the treated: combining experimental with observational studies to estimate population treatment effects," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 178(3), pages 757-778, June.
    4. David S. Yeager & Paul Hanselman & Gregory M. Walton & Jared S. Murray & Robert Crosnoe & Chandra Muller & Elizabeth Tipton & Barbara Schneider & Chris S. Hulleman & Cintia P. Hinojosa & David Paunesk, 2019. "A national experiment reveals where a growth mindset improves achievement," Nature, Nature, vol. 573(7774), pages 364-369, September.
    5. Robert B. Olsen & Stephen H. Bell & Austin Nichols, 2018. "Using Preferred Applicant Random Assignment (PARA) to Reduce Randomization Bias in Randomized Trials of Discretionary Programs," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 37(1), pages 167-180, January.
    6. Leonard Feldt, 1961. "The use of extreme groups to test for the presence of a relationship," Psychometrika, Springer;The Psychometric Society, vol. 26(3), pages 307-316, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Elizabeth Tipton & Robert B. Olsen, "undated". "Enhancing the Generalizability of Impact Studies in Education," Mathematica Policy Research Reports 35d5625333dc480aba9765b3b, Mathematica Policy Research.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rajeev Dehejia & Cristian Pop-Eleches & Cyrus Samii, 2021. "From Local to Global: External Validity in a Fertility Natural Experiment," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 39(1), pages 217-243, January.
    2. Wendy Chan, 2018. "Applications of Small Area Estimation to Generalization With Subclassification by Propensity Scores," Journal of Educational and Behavioral Statistics, , vol. 43(2), pages 182-224, April.
    3. Fan Li & Ashley L. Buchanan & Stephen R. Cole, 2022. "Generalizing trial evidence to target populations in non‐nested designs: Applications to AIDS clinical trials," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 71(3), pages 669-697, June.
    4. Dasom Lee & Shu Yang & Lin Dong & Xiaofei Wang & Donglin Zeng & Jianwen Cai, 2023. "Improving trial generalizability using observational studies," Biometrics, The International Biometric Society, vol. 79(2), pages 1213-1225, June.
    5. Sharples, Linda D., 2018. "The role of statistics in the era of big data: Electronic health records for healthcare research," Statistics & Probability Letters, Elsevier, vol. 136(C), pages 105-110.
    6. Elizabeth A. Stuart & Anna Rhodes, 2017. "Generalizing Treatment Effect Estimates From Sample to Population: A Case Study in the Difficulties of Finding Sufficient Data," Evaluation Review, , vol. 41(4), pages 357-388, August.
    7. Elizabeth Tipton & Kelly Hallberg & Larry V. Hedges & Wendy Chan, 2017. "Implications of Small Samples for Generalization: Adjustments and Rules of Thumb," Evaluation Review, , vol. 41(5), pages 472-505, October.
    8. Sarah A. Avellar & Jaime Thomas & Rebecca Kleinman & Emily Sama-Miller & Sara E. Woodruff & Rebecca Coughlin & T’Pring R. Westbrook, 2017. "External Validity: The Next Step for Systematic Reviews?," Evaluation Review, , vol. 41(4), pages 283-325, August.
    9. Kellie Ottoboni & Jason Poulos, 2019. "Estimating population average treatment effects from experiments with noncompliance," Papers 1901.02991, arXiv.org, revised Aug 2020.
    10. Esterling, Kevin & Brady, David & Schwitzgebel, Eric, 2021. "The Necessity of Construct and External Validity for Generalized Causal Claims," OSF Preprints 2s8w5, Center for Open Science.
    11. Isaiah Andrews & Emily Oster, 2017. "A Simple Approximation for Evaluating External Validity Bias," NBER Working Papers 23826, National Bureau of Economic Research, Inc.
    12. Ashley L. Buchanan & Michael G. Hudgens & Stephen R. Cole & Katie R. Mollan & Paul E. Sax & Eric S. Daar & Adaora A. Adimora & Joseph J. Eron & Michael J. Mugavero, 2018. "Generalizing evidence from randomized trials using inverse probability of sampling weights," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 181(4), pages 1193-1209, October.
    13. Xinyu Li & Wang Miao & Fang Lu & Xiao‐Hua Zhou, 2023. "Improving efficiency of inference in clinical trials with external control data," Biometrics, The International Biometric Society, vol. 79(1), pages 394-403, March.
    14. repec:mpr:mprres:8128 is not listed on IDEAS
    15. Lihua Lei & Emmanuel J. Candès, 2021. "Conformal inference of counterfactuals and individual treatment effects," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 83(5), pages 911-938, November.
    16. David M. Phillippo & Anthony E. Ades & Sofia Dias & Stephen Palmer & Keith R. Abrams & Nicky J. Welton, 2018. "Methods for Population-Adjusted Indirect Comparisons in Health Technology Appraisal," Medical Decision Making, , vol. 38(2), pages 200-211, February.
    17. Andrews, Isaiah & Oster, Emily, 2019. "A simple approximation for evaluating external validity bias," Economics Letters, Elsevier, vol. 178(C), pages 58-62.
    18. Ottoboni Kellie N. & Poulos Jason V., 2020. "Estimating population average treatment effects from experiments with noncompliance," Journal of Causal Inference, De Gruyter, vol. 8(1), pages 108-130, January.
    19. Elizabeth Tipton & Laura R. Peck, 2017. "A Design-Based Approach to Improve External Validity in Welfare Policy Evaluations," Evaluation Review, , vol. 41(4), pages 326-356, August.
    20. Elizabeth Tipton, 2014. "How Generalizable Is Your Experiment? An Index for Comparing Experimental Samples and Populations," Journal of Educational and Behavioral Statistics, , vol. 39(6), pages 478-501, December.
    21. Peter Z. Schochet, "undated". "Statistical Theory for the RCT-YES Software: Design-Based Causal Inference for RCTs," Mathematica Policy Research Reports a0c005c003c242308a92c02dc, Mathematica Policy Research.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jorssa:v:184:y:2021:i:2:p:504-521. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/rssssea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.