IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v46y2021i5p527-559.html
   My bibliography  Save this article

Design Considerations in Multisite Randomized Trials Probing Moderated Treatment Effects

Author

Listed:
  • Nianbo Dong

    (2331University of North Carolina at Chapel Hill)

  • Benjamin Kelcey

    (2514University of Cincinnati)

  • Jessaca Spybrook

    (4175Western Michigan University)

Abstract

Past research has demonstrated that treatment effects frequently vary across sites (e.g., schools) and that such variation can be explained by site-level or individual-level variables (e.g., school size or gender). The purpose of this study is to develop a statistical framework and tools for the effective and efficient design of multisite randomized trials (MRTs) probing moderated treatment effects. The framework considers three core facets of such designs: (a) Level 1 and Level 2 moderators, (b) random and nonrandomly varying slopes (coefficients) of the treatment variable and its interaction terms with the moderators, and (c) binary and continuous moderators. We validate the formulas for calculating statistical power and the minimum detectable effect size difference with simulations, probe its sensitivity to model assumptions, execute the formulas in accessible software, demonstrate an application, and provide suggestions in designing MRTs probing moderated treatment effects.

Suggested Citation

  • Nianbo Dong & Benjamin Kelcey & Jessaca Spybrook, 2021. "Design Considerations in Multisite Randomized Trials Probing Moderated Treatment Effects," Journal of Educational and Behavioral Statistics, , vol. 46(5), pages 527-559, October.
    • Handle: RePEc:sae:jedbes:v:46:y:2021:i:5:p:527-559
    DOI: 10.3102/1076998620961492
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998620961492
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998620961492?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Allen Fleishman, 1978. "A method for simulating non-normal distributions," Psychometrika, Springer;The Psychometric Society, vol. 43(4), pages 521-532, December.
    2. Larry V. Hedges & E. C. Hedberg, 2013. "Intraclass Correlations and Covariate Outcome Correlations for Planning Two- and Three-Level Cluster-Randomized Experiments in Education," Evaluation Review, , vol. 37(6), pages 445-489, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Marie-Andrée Somers & Michael J. Weiss & Colin Hill, 2023. "Design Parameters for Planning the Sample Size of Individual-Level Randomized Controlled Trials in Community Colleges," Evaluation Review, , vol. 47(4), pages 599-629, August.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Doole, Graeme J. & Romera, Alvaro J. & Leslie, Jennifer E. & Chapman, David F. & Pinxterhuis, Ina (J.B.). & Kemp, Peter D., 2021. "Economic assessment of plantain (Plantago lanceolata) uptake in the New Zealand dairy sector," Agricultural Systems, Elsevier, vol. 187(C).
    2. Ben Kelcey & Zuchao Shen & Jessaca Spybrook, 2016. "Intraclass Correlation Coefficients for Designing Cluster-Randomized Trials in Sub-Saharan Africa Education," Evaluation Review, , vol. 40(6), pages 500-525, December.
    3. Headrick, Todd C. & Sheng, Yanyan & Hodis, Flaviu-Adrian, 2007. "Numerical Computing and Graphics for the Power Method Transformation Using Mathematica," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 19(i03).
    4. Nianbo Dong & Wendy M. Reinke & Keith C. Herman & Catherine P. Bradshaw & Desiree W. Murray, 2016. "Meaningful Effect Sizes, Intraclass Correlations, and Proportions of Variance Explained by Covariates for Planning Two- and Three-Level Cluster Randomized Trials of Social and Behavioral Outcomes," Evaluation Review, , vol. 40(4), pages 334-377, August.
    5. Pesaran, M. Hashem & Yamagata, Takashi, 2012. "Testing CAPM with a Large Number of Assets," IZA Discussion Papers 6469, Institute of Labor Economics (IZA).
    6. Schinckus, Christophe, 2015. "The valuation of social impact bonds: An introductory perspective with the Peterborough SIB," Research in International Business and Finance, Elsevier, vol. 35(C), pages 104-110.
    7. Elizabeth Tipton & Robert B. Olsen, "undated". "Enhancing the Generalizability of Impact Studies in Education," Mathematica Policy Research Reports 35d5625333dc480aba9765b3b, Mathematica Policy Research.
    8. Max Auerswald & Morten Moshagen, 2015. "Generating Correlated, Non-normally Distributed Data Using a Non-linear Structural Model," Psychometrika, Springer;The Psychometric Society, vol. 80(4), pages 920-937, December.
    9. Weihua Fan & Gregory R. Hancock, 2012. "Robust Means Modeling," Journal of Educational and Behavioral Statistics, , vol. 37(1), pages 137-156, February.
    10. Mohan D. Pant & Todd C. Headrick, 2017. "Simulating Uniform- and Triangular- Based Double Power Method Distributions," Journal of Statistical and Econometric Methods, SCIENPRESS Ltd, vol. 6(1), pages 1-1.
    11. Mahul, Olivier, 2002. "Hedging Price Risk in the Presence of Crop Yield and Revenue Insurance," 2002 International Congress, August 28-31, 2002, Zaragoza, Spain 24881, European Association of Agricultural Economists.
    12. Löhndorf, Nils, 2016. "An empirical analysis of scenario generation methods for stochastic optimization," European Journal of Operational Research, Elsevier, vol. 255(1), pages 121-132.
    13. Shaobo Jin & Fan Yang-Wallentin, 2017. "Asymptotic Robustness Study of the Polychoric Correlation Estimation," Psychometrika, Springer;The Psychometric Society, vol. 82(1), pages 67-85, March.
    14. Schinckus, Christophe, 2018. "The valuation of social impact bonds: An introductory perspective with the Peterborough SIB," Research in International Business and Finance, Elsevier, vol. 45(C), pages 1-6.
    15. Foss, Tron & Jöreskog, Karl G. & Olsson, Ulf H., 2011. "Testing structural equation models: The effect of kurtosis," Computational Statistics & Data Analysis, Elsevier, vol. 55(7), pages 2263-2275, July.
    16. Daniel McNeish & Jeffrey R. Harring & Denis Dumas, 2023. "A multilevel structured latent curve model for disaggregating student and school contributions to learning," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 32(2), pages 545-575, June.
    17. Pelagatti, Matteo M. & Sen, Pranab K., 2013. "Rank tests for short memory stationarity," Journal of Econometrics, Elsevier, vol. 172(1), pages 90-105.
    18. Nicolas Depraetere & Martina Vandebroek, 2014. "Order selection in finite mixtures of linear regressions," Statistical Papers, Springer, vol. 55(3), pages 871-911, August.
    19. Emanuela Raffinetti & Pier Alda Ferrari, 2021. "A dependence measure flow tree through Monte Carlo simulations," Quality & Quantity: International Journal of Methodology, Springer, vol. 55(2), pages 467-496, April.
    20. Al-Subaihi, Ali A., 2004. "Simulating Correlated Multivariate Pseudorandom Numbers," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 9(i04).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:46:y:2021:i:5:p:527-559. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.