IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v47y2023i4p599-629.html
   My bibliography  Save this article

Design Parameters for Planning the Sample Size of Individual-Level Randomized Controlled Trials in Community Colleges

Author

Listed:
  • Marie-Andrée Somers
  • Michael J. Weiss
  • Colin Hill

Abstract

The last two decades have seen a dramatic increase in randomized controlled trials (RCTs) conducted in community colleges. Yet, there is limited empirical information on the design parameters necessary to plan the sample size for RCTs in this context. For a blocked student-level random assignment research design, key design parameters for the minimum detectable true effect (MDTE) are the within-block outcome standard deviation ( σ | S ) and the within-block outcome variance explained by baseline covariates like student characteristics ( R | S 2 ) . We provide empirical estimates of these key design parameters, discussing the pattern of estimates across outcomes (enrollment, credits earned, credential attainment, and grade point average), semesters, and studies. The main analyses use student-level data from 8 to 14 RCTs including 5,649–7,099 students (depending on the outcome) with follow-up data for 3 years. The following patterns are observed: the within-block standard deviation ( σ | S ) and therefore the MDTE can be much larger in later semesters for enrollment outcomes and cumulative credits earned; there is substantial variation across studies in σ | S for degree attainment; and baseline covariates explain less than 10% of the variation in student outcomes. These findings indicate that when planning the sample size for a study, researchers should be mindful of the follow-up period, use a range of values to calculate the MDTE for outcomes that vary across studies, and assume a value of R | S 2 between 0 and 0.05. A public database created for this paper includes parameter estimates for additional RCTs and students.

Suggested Citation

  • Marie-Andrée Somers & Michael J. Weiss & Colin Hill, 2023. "Design Parameters for Planning the Sample Size of Individual-Level Randomized Controlled Trials in Community Colleges," Evaluation Review, , vol. 47(4), pages 599-629, August.
  • Handle: RePEc:sae:evarev:v:47:y:2023:i:4:p:599-629
    DOI: 10.1177/0193841X221121236
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X221121236
    Download Restriction: no

    File URL: https://libkey.io/10.1177/0193841X221121236?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Nianbo Dong & Benjamin Kelcey & Jessaca Spybrook, 2021. "Design Considerations in Multisite Randomized Trials Probing Moderated Treatment Effects," Journal of Educational and Behavioral Statistics, , vol. 46(5), pages 527-559, October.
    2. repec:mpr:mprres:5863 is not listed on IDEAS
    3. E. C. Hedberg, 2016. "Academic and Behavioral Design Parameters for Cluster Randomized Trials in Kindergarten," Evaluation Review, , vol. 40(4), pages 279-313, August.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.

      Corrections

      All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:47:y:2023:i:4:p:599-629. See general information about how to correct material in RePEc.

      If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

      If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

      If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

      For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

      Please note that corrections may take a couple of weeks to filter through the various RePEc services.

      IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.