IDEAS home Printed from https://ideas.repec.org/p/osf/osfxxx/2gur9.html
   My bibliography  Save this paper

A new four-arm within-study comparison: Design, implementation, and data

Author

Listed:
  • Keller, Bryan
  • Wong, Vivian C

    (University of Virginia)

  • Park, Sangbaek
  • Zhang, Jingru
  • Sheehan, Patrick
  • Steiner, Peter M.

Abstract

Within-study comparisons (WSCs) use real, rather than simulated, data to compare estimates from observational studies against a benchmark randomized controlled trial (RCT). A primary goal of WSCs is to assess whether well-designed quasi-experimental designs (QEDs) can produce internally valid causal effect estimates comparable to those from RCTs. In this paper, we describe the design and implementation of a new type of WSC. Motivated by Shadish et al. (2008), we examine the impact of a mathematics training intervention and a vocabulary study session on posttest scores for mathematics and vocabulary, respectively. We extend the original design in three ways. First, before random assignment, we ask participants to express a preference for either the mathematics or vocabulary training session, after which they are randomly assigned regardless of preferences. This allows us to experimentally identify and estimate the overall average treatment effect (ATE) and two conditional ATEs: the average treatment effect on the treated (ATT) and the average treatment effect on the untreated (ATU). Second, participant recruitment and sample size (N = 2200) were determined through power analyses for comparing RCT and QED estimates, ensuring sufficient power for methodological comparisons. Finally, the study’s eligibility criteria, recruitment, treatment allocation, and analysis plan were preregistered on the Open Science Foundation platform, and the data are publicly accessible. We believe that this WSC design and the resulting data set will be valuable for researchers seeking to evaluate causal inference methods and test identification assumptions using real-world data.

Suggested Citation

  • Keller, Bryan & Wong, Vivian C & Park, Sangbaek & Zhang, Jingru & Sheehan, Patrick & Steiner, Peter M., 2024. "A new four-arm within-study comparison: Design, implementation, and data," OSF Preprints 2gur9, Center for Open Science.
  • Handle: RePEc:osf:osfxxx:2gur9
    DOI: 10.31219/osf.io/2gur9
    as

    Download full text from publisher

    File URL: https://osf.io/download/66df54047f573a030f77af7b/
    Download Restriction: no

    File URL: https://libkey.io/10.31219/osf.io/2gur9?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Steven Glazerman & Dan M. Levy & David Myers, 2003. "Nonexperimental Versus Experimental Estimates of Earnings Impacts," The ANNALS of the American Academy of Political and Social Science, , vol. 589(1), pages 63-93, September.
    2. Burt S. Barnow & Coady Wing & M. H. Clark, 2017. "What Can We Learn From A Doubly Randomized Preference Trial?—An Instrumental Variables Perspective," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 36(2), pages 418-437, March.
    3. Peter M. Steiner & Thomas D. Cook & William R. Shadish, 2011. "On the Importance of Reliable Covariate Measurement in Selection Bias Adjustments Using Propensity Scores," Journal of Educational and Behavioral Statistics, , vol. 36(2), pages 213-236, April.
    4. Heejung Bang & James M. Robins, 2005. "Doubly Robust Estimation in Missing Data and Causal Inference Models," Biometrics, The International Biometric Society, vol. 61(4), pages 962-973, December.
    5. Stephen H. Bell & Larry l. Orr & John D. Blomquist & Glen G. Cain, 1995. "Program Applicants as a Comparison Group in Evaluating Training Programs: Theory and a Test," Books from Upjohn Press, W.E. Upjohn Institute for Employment Research, number pacg, November.
    6. Guido W. Imbens, 2004. "Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 4-29, February.
    7. repec:mpr:mprres:3694 is not listed on IDEAS
    8. Friedlander, Daniel & Robins, Philip K, 1995. "Evaluating Program Evaluations: New Evidence on Commonly Used Nonexperimental Methods," American Economic Review, American Economic Association, vol. 85(4), pages 923-937, September.
    9. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, October.
    10. Thomas Fraker & Rebecca Maynard, 1987. "The Adequacy of Comparison Group Designs for Evaluations of Employment-Related Programs," Journal of Human Resources, University of Wisconsin Press, vol. 22(2), pages 194-227.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Katherine Baicker & Theodore Svoronos, 2019. "Testing the Validity of the Single Interrupted Time Series Design," NBER Working Papers 26080, National Bureau of Economic Research, Inc.
    2. Katherine Baicker & Theodore Svoronos, 2019. "Testing the Validity of the Single Interrupted Time Series Design," CID Working Papers 364, Center for International Development at Harvard University.
    3. Vivian C. Wong & Peter M. Steiner & Kylie L. Anglin, 2018. "What Can Be Learned From Empirical Evaluations of Nonexperimental Methods?," Evaluation Review, , vol. 42(2), pages 147-175, April.
    4. Travis St.Clair & Kelly Hallberg & Thomas D. Cook, 2016. "The Validity and Precision of the Comparative Interrupted Time-Series Design," Journal of Educational and Behavioral Statistics, , vol. 41(3), pages 269-299, June.
    5. Daniel Litwok, 2023. "Estimating the Impact of Emergency Assistance on Educational Progress for Low-Income Adults: Experimental and Nonexperimental Evidence," Evaluation Review, , vol. 47(2), pages 231-263, April.
    6. Plamen Nikolov & Hongjian Wang & Kevin Acker, 2020. "Wage premium of Communist Party membership: Evidence from China," Pacific Economic Review, Wiley Blackwell, vol. 25(3), pages 309-338, August.
    7. Carlos A. Flores & Oscar A. Mitnik, 2009. "Evaluating Nonexperimental Estimators for Multiple Treatments: Evidence from Experimental Data," Working Papers 2010-10, University of Miami, Department of Economics.
    8. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    9. Siying Guo & Jianxuan Liu & Qiu Wang, 2022. "Effective Learning During COVID-19: Multilevel Covariates Matching and Propensity Score Matching," Annals of Data Science, Springer, vol. 9(5), pages 967-982, October.
    10. Peter R. Mueser & Kenneth R. Troske & Alexey Gorislavsky, 2007. "Using State Administrative Data to Measure Program Performance," The Review of Economics and Statistics, MIT Press, vol. 89(4), pages 761-783, November.
    11. Dmitry Arkhangelsky & Guido Imbens, 2023. "Causal Models for Longitudinal and Panel Data: A Survey," Papers 2311.15458, arXiv.org, revised Jun 2024.
    12. Susan Athey & Guido W. Imbens, 2017. "The State of Applied Econometrics: Causality and Policy Evaluation," Journal of Economic Perspectives, American Economic Association, vol. 31(2), pages 3-32, Spring.
    13. Lechner, Michael & Wunsch, Conny, 2013. "Sensitivity of matching-based program evaluations to the availability of control variables," Labour Economics, Elsevier, vol. 21(C), pages 111-121.
    14. Kenneth Fortson & Philip Gleason & Emma Kopa & Natalya Verbitsky-Savitz, "undated". "Horseshoes, Hand Grenades, and Treatment Effects? Reassessing Bias in Nonexperimental Estimators," Mathematica Policy Research Reports 1c24988cd5454dd3be51fbc2c, Mathematica Policy Research.
    15. Kiran Tomlinson & Johan Ugander & Austin R. Benson, 2021. "Choice Set Confounding in Discrete Choice," Papers 2105.07959, arXiv.org, revised Aug 2021.
    16. Peter M. Steiner, 2011. "Propensity Score Methods for Causal Inference: On the Relative Importance of Covariate Selection, Reliable Measurement, and Choice of Propensity Score Technique," Working Papers 09, AlmaLaurea Inter-University Consortium.
    17. Davide Viviano & Jelena Bradic, 2020. "Fair Policy Targeting," Papers 2005.12395, arXiv.org, revised Jun 2022.
    18. Verena Lauber & Johanna Storck, 2016. "Helping with the Kids? How Family-Friendly Workplaces Affect Parental Well-Being and Behavior," Discussion Papers of DIW Berlin 1630, DIW Berlin, German Institute for Economic Research.
    19. J. R. Lockwood & Daniel F. McCaffrey, 2019. "Impact Evaluation Using Analysis of Covariance With Error-Prone Covariates That Violate Surrogacy," Evaluation Review, , vol. 43(6), pages 335-369, December.
    20. Fortson, Kenneth & Gleason, Philip & Kopa, Emma & Verbitsky-Savitz, Natalya, 2015. "Horseshoes, hand grenades, and treatment effects? Reassessing whether nonexperimental estimators are biased," Economics of Education Review, Elsevier, vol. 44(C), pages 100-113.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:osfxxx:2gur9. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.