IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v38y2013i5p443-469.html
   My bibliography  Save this article

Alternatives for Mixed-Effects Meta-Regression Models in the Reliability Generalization Approach

Author

Listed:
  • José Antonio López-López

    (University of Murcia, Spain)

  • Juan Botella

    (Autonomous University of Madrid, Spain)

  • Julio Sánchez-Meca

    (University of Murcia, Spain)

  • Fulgencio Marín-Martínez

Abstract

Since heterogeneity between reliability coefficients is usually found in reliability generalization studies, moderator analyses constitute a crucial step for that meta-analytic approach. In this study, different procedures for conducting mixed-effects meta-regression analyses were compared. Specifically, four transformation methods for the reliability coefficients, two estimators of the residual between-studies variance, and two methods for testing regression coefficients significance were combined in a Monte Carlo simulation study. The different methods were compared in terms of bias and mean square error (MSE) of the slope estimates, and Type I error and statistical power rates for the slope statistical tests. The results of the simulation study did not vary as a function of the residual variance estimator. All transformation methods provided negatively biased estimates, but both bias and MSE were reasonably small in all cases. In contrast, important differences were found regarding statistical tests, with the method proposed by Knapp and Hartung showing a better adjustment to the nominal significance level and higher power rates than the standard method.

Suggested Citation

  • José Antonio López-López & Juan Botella & Julio Sánchez-Meca & Fulgencio Marín-Martínez, 2013. "Alternatives for Mixed-Effects Meta-Regression Models in the Reliability Generalization Approach," Journal of Educational and Behavioral Statistics, , vol. 38(5), pages 443-469, October.
    • Handle: RePEc:sae:jedbes:v:38:y:2013:i:5:p:443-469
    DOI: 10.3102/1076998612466142
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998612466142
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998612466142?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. A. Hakstian & Thomas Whalen, 1976. "A k-sample significance test for independent alpha coefficients," Psychometrika, Springer;The Psychometric Society, vol. 41(2), pages 219-231, June.
    2. Roger M. Harbord & Julian P.T. Higgins, 2008. "Meta-regression in Stata," Stata Journal, StataCorp LLC, vol. 8(4), pages 493-519, December.
    3. Viechtbauer, Wolfgang, 2010. "Conducting Meta-Analyses in R with the metafor Package," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 36(i03).
    4. Leonard Feldt, 1969. "A test of the hypothesis that cronbach's alpha or kuder-richardson coefficent twenty is the same for two tests," Psychometrika, Springer;The Psychometric Society, vol. 34(3), pages 363-373, September.
    5. Martin, Andrew D. & Quinn, Kevin M. & Park, Jong Hee, 2011. "MCMCpack: Markov Chain Monte Carlo in R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 42(i09).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Peter M. Bentler, 2016. "Covariate-free and Covariate-dependent Reliability," Psychometrika, Springer;The Psychometric Society, vol. 81(4), pages 907-920, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. White, Robin R. & Brady, Michael, 2014. "Can consumers’ willingness to pay incentivize adoption of environmental impact reducing technologies in meat animal production?," Food Policy, Elsevier, vol. 49(P1), pages 41-49.
    2. Amin Haghnejad & Mahboobeh Farahati, 2024. "An overview of meta-analytic methods for economic research," Papers 2412.10608, arXiv.org, revised Mar 2025.
    3. Marsha Schroeder & A. Ralph Hakstian, 1990. "Inferential procedures for multifaceted coefficients of generalizability," Psychometrika, Springer;The Psychometric Society, vol. 55(3), pages 429-447, September.
    4. Whitney S Beck & Ed K Hall, 2018. "Confounding factors in algal phosphorus limitation experiments," PLOS ONE, Public Library of Science, vol. 13(10), pages 1-19, October.
    5. Tianyi Qiu & Yu Shi & Josep Peñuelas & Ji Liu & Qingliang Cui & Jordi Sardans & Feng Zhou & Longlong Xia & Weiming Yan & Shuling Zhao & Shushi Peng & Jinshi Jian & Qinsi He & Wenju Zhang & Min Huang &, 2024. "Optimizing cover crop practices as a sustainable solution for global agroecosystem services," Nature Communications, Nature, vol. 15(1), pages 1-14, December.
    6. Xindong Xue & W. Robert Reed & Robbie C.M. van Aert, 2022. "Social Capital and Economic Growth: A Meta-Analysis," Working Papers in Economics 22/20, University of Canterbury, Department of Economics and Finance.
    7. Hengtao Zhang & Guosheng Yin, 2021. "Response‐adaptive rerandomization," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 70(5), pages 1281-1298, November.
    8. Bart Verkuil & Serpil Atasayi & Marc L Molendijk, 2015. "Workplace Bullying and Mental Health: A Meta-Analysis on Cross-Sectional and Longitudinal Data," PLOS ONE, Public Library of Science, vol. 10(8), pages 1-16, August.
    9. Andrew Gelman & Daniel Lee & Jiqiang Guo, 2015. "Stan," Journal of Educational and Behavioral Statistics, , vol. 40(5), pages 530-543, October.
    10. Francesca Pilotto & Ingolf Kühn & Rita Adrian & Renate Alber & Audrey Alignier & Christopher Andrews & Jaana Bäck & Luc Barbaro & Deborah Beaumont & Natalie Beenaerts & Sue Benham & David S. Boukal & , 2020. "Meta-analysis of multidecadal biodiversity trends in Europe," Nature Communications, Nature, vol. 11(1), pages 1-11, December.
    11. repec:cup:judgdm:v:15:y:2020:i:6:p:972-988 is not listed on IDEAS
    12. Jonas Schmidt & Tammo H. A. Bijmolt, 2020. "Accurately measuring willingness to pay for consumer goods: a meta-analysis of the hypothetical bias," Journal of the Academy of Marketing Science, Springer, vol. 48(3), pages 499-518, May.
    13. Zabaloy, Maria Florencia & Viego, Valentina, 2022. "Household electricity demand in Latin America and the Caribbean: A meta-analysis of price elasticity," Utilities Policy, Elsevier, vol. 75(C).
    14. Zuoxian Gan & Tao Feng & Min Yang, 2018. "Exploring the Effects of Car Ownership and Commuting on Subjective Well-Being: A Nationwide Questionnaire Study," Sustainability, MDPI, vol. 11(1), pages 1-20, December.
    15. Mario Herberz & Tobias Brosch & Ulf J. J. Hahnel, 2020. "Kilo what? Default units increase value sensitivity in joint evaluations of energy efficiency," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 15(6), pages 972-988, November.
    16. Kate Birnie & Rachel Cooper & Richard M Martin & Diana Kuh & Avan Aihie Sayer & Beatriz E Alvarado & Antony Bayer & Kaare Christensen & Sung-il Cho & Cyrus Cooper & Janie Corley & Leone Craig & Ian J , 2011. "Childhood Socioeconomic Position and Objectively Measured Physical Capability Levels in Adulthood: A Systematic Review and Meta-Analysis," PLOS ONE, Public Library of Science, vol. 6(1), pages 1-13, January.
    17. Maier, Maximilian & VanderWeele, Tyler J. & Mathur, Maya B, 2021. "Using Selection Models to Assess Sensitivity to Publication Bias: A Tutorial and Call for More Routine Use," MetaArXiv tp45u_v1, Center for Open Science.
    18. Piers Steel & Sjoerd Beugelsdijk & Herman Aguinis, 2021. "The anatomy of an award-winning meta-analysis: Recommendations for authors, reviewers, and readers of meta-analytic reviews," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 52(1), pages 23-44, February.
    19. Antonio Parisi & B. Liseo, 2018. "Objective Bayesian analysis for the multivariate skew-t model," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 27(2), pages 277-295, June.
    20. Masood Gheasi & Noriko Ishikawa & Karima Kourtit & Peter Nijkamp, 2019. "A meta-analysis of human health differences in urban and rural environments," Letters in Spatial and Resource Sciences, Springer, vol. 12(3), pages 167-186, December.
    21. Molo, Fabio & Pawel, Samuel & Fraga González, Gorka, 2024. "A Robustness Reproduction of "A Systematic Review and Meta-Analysis of 90 Cohort Studies of Social Isolation, Loneliness and Mortality"," I4R Discussion Paper Series 169, The Institute for Replication (I4R).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:38:y:2013:i:5:p:443-469. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.