IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v46y2021i5p527-559.html
   My bibliography  Save this article

Design Considerations in Multisite Randomized Trials Probing Moderated Treatment Effects

Author

Listed:
  • Nianbo Dong

    (2331University of North Carolina at Chapel Hill)

  • Benjamin Kelcey

    (2514University of Cincinnati)

  • Jessaca Spybrook

    (4175Western Michigan University)

Abstract

Past research has demonstrated that treatment effects frequently vary across sites (e.g., schools) and that such variation can be explained by site-level or individual-level variables (e.g., school size or gender). The purpose of this study is to develop a statistical framework and tools for the effective and efficient design of multisite randomized trials (MRTs) probing moderated treatment effects. The framework considers three core facets of such designs: (a) Level 1 and Level 2 moderators, (b) random and nonrandomly varying slopes (coefficients) of the treatment variable and its interaction terms with the moderators, and (c) binary and continuous moderators. We validate the formulas for calculating statistical power and the minimum detectable effect size difference with simulations, probe its sensitivity to model assumptions, execute the formulas in accessible software, demonstrate an application, and provide suggestions in designing MRTs probing moderated treatment effects.

Suggested Citation

  • Nianbo Dong & Benjamin Kelcey & Jessaca Spybrook, 2021. "Design Considerations in Multisite Randomized Trials Probing Moderated Treatment Effects," Journal of Educational and Behavioral Statistics, , vol. 46(5), pages 527-559, October.
    • Handle: RePEc:sae:jedbes:v:46:y:2021:i:5:p:527-559
    DOI: 10.3102/1076998620961492
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998620961492
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998620961492?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Larry V. Hedges & E. C. Hedberg, 2013. "Intraclass Correlations and Covariate Outcome Correlations for Planning Two- and Three-Level Cluster-Randomized Experiments in Education," Evaluation Review, , vol. 37(6), pages 445-489, December.
    2. Allen Fleishman, 1978. "A method for simulating non-normal distributions," Psychometrika, Springer;The Psychometric Society, vol. 43(4), pages 521-532, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Marie-Andrée Somers & Michael J. Weiss & Colin Hill, 2023. "Design Parameters for Planning the Sample Size of Individual-Level Randomized Controlled Trials in Community Colleges," Evaluation Review, , vol. 47(4), pages 599-629, August.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ben Kelcey & Zuchao Shen & Jessaca Spybrook, 2016. "Intraclass Correlation Coefficients for Designing Cluster-Randomized Trials in Sub-Saharan Africa Education," Evaluation Review, , vol. 40(6), pages 500-525, December.
    2. Nianbo Dong & Wendy M. Reinke & Keith C. Herman & Catherine P. Bradshaw & Desiree W. Murray, 2016. "Meaningful Effect Sizes, Intraclass Correlations, and Proportions of Variance Explained by Covariates for Planning Two- and Three-Level Cluster Randomized Trials of Social and Behavioral Outcomes," Evaluation Review, , vol. 40(4), pages 334-377, August.
    3. Schinckus, Christophe, 2015. "The valuation of social impact bonds: An introductory perspective with the Peterborough SIB," Research in International Business and Finance, Elsevier, vol. 35(C), pages 104-110.
    4. Max Auerswald & Morten Moshagen, 2015. "Generating Correlated, Non-normally Distributed Data Using a Non-linear Structural Model," Psychometrika, Springer;The Psychometric Society, vol. 80(4), pages 920-937, December.
    5. Mohan D. Pant & Todd C. Headrick, 2017. "Simulating Uniform- and Triangular- Based Double Power Method Distributions," Journal of Statistical and Econometric Methods, SCIENPRESS Ltd, vol. 6(1), pages 1-1.
    6. Schinckus, Christophe, 2018. "The valuation of social impact bonds: An introductory perspective with the Peterborough SIB," Research in International Business and Finance, Elsevier, vol. 45(C), pages 1-6.
    7. Daniel McNeish & Jeffrey R. Harring & Denis Dumas, 2023. "A multilevel structured latent curve model for disaggregating student and school contributions to learning," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 32(2), pages 545-575, June.
    8. Pelagatti, Matteo M. & Sen, Pranab K., 2013. "Rank tests for short memory stationarity," Journal of Econometrics, Elsevier, vol. 172(1), pages 90-105.
    9. Emanuela Raffinetti & Pier Alda Ferrari, 2021. "A dependence measure flow tree through Monte Carlo simulations," Quality & Quantity: International Journal of Methodology, Springer, vol. 55(2), pages 467-496, April.
    10. Al-Subaihi, Ali A., 2004. "Simulating Correlated Multivariate Pseudorandom Numbers," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 9(i04).
    11. Yen Lee & David Kaplan, 2018. "Generating Multivariate Ordinal Data via Entropy Principles," Psychometrika, Springer;The Psychometric Society, vol. 83(1), pages 156-181, March.
    12. M Hashem Pesaran & Takashi Yamagata, 2012. "Testing CAPM with a Large Number of Assets," Discussion Papers 12/05, Department of Economics, University of York.
    13. Beasley, T. Mark & Zumbo, Bruno D., 2003. "Comparison of aligned Friedman rank and parametric methods for testing interactions in split-plot designs," Computational Statistics & Data Analysis, Elsevier, vol. 42(4), pages 569-593, April.
    14. Irmer, Julien Patrick & Klein, Andreas & Schermelleh-Engel, Karin, 2024. "A General Model-Implied Simulation-Based Power Estimation Method for Correctly and Misspecfied Models: Applications to Nonlinear and Linear Structural Equation Models," OSF Preprints pe5bj, Center for Open Science.
    15. Fei Gu & Hao Wu, 2016. "Raw Data Maximum Likelihood Estimation for Common Principal Component Models: A State Space Approach," Psychometrika, Springer;The Psychometric Society, vol. 81(3), pages 751-773, September.
    16. Pasquale Dolce & Cristina Davino & Domenico Vistocco, 2022. "Quantile composite-based path modeling: algorithms, properties and applications," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 16(4), pages 909-949, December.
    17. Peter Wludyka, 1999. "Two non-parametric, analysis-of-means-type tests for homogeneity of variances," Journal of Applied Statistics, Taylor & Francis Journals, vol. 26(2), pages 243-256.
    18. Yadira Pazmiño & José Juan de Felipe & Marc Vallbé & Franklin Cargua & Luis Quevedo, 2021. "Identification of a Set of Variables for the Classification of Páramo Soils Using a Nonparametric Model, Remote Sensing, and Organic Carbon," Sustainability, MDPI, vol. 13(16), pages 1-22, August.
    19. Po-Hsien Huang, 2017. "Asymptotics of AIC, BIC, and RMSEA for Model Selection in Structural Equation Modeling," Psychometrika, Springer;The Psychometric Society, vol. 82(2), pages 407-426, June.
    20. Ke-Hai Yuan & Peter Bentler, 2002. "On robusiness of the normal-theory based asymptotic distributions of three reliability coefficient estimates," Psychometrika, Springer;The Psychometric Society, vol. 67(2), pages 251-259, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:46:y:2021:i:5:p:527-559. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.