IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v38y2013i5p443-469.html
   My bibliography  Save this article

Alternatives for Mixed-Effects Meta-Regression Models in the Reliability Generalization Approach

Author

Listed:
  • José Antonio López-López

    (University of Murcia, Spain)

  • Juan Botella

    (Autonomous University of Madrid, Spain)

  • Julio Sánchez-Meca

    (University of Murcia, Spain)

  • Fulgencio Marín-Martínez

Abstract

Since heterogeneity between reliability coefficients is usually found in reliability generalization studies, moderator analyses constitute a crucial step for that meta-analytic approach. In this study, different procedures for conducting mixed-effects meta-regression analyses were compared. Specifically, four transformation methods for the reliability coefficients, two estimators of the residual between-studies variance, and two methods for testing regression coefficients significance were combined in a Monte Carlo simulation study. The different methods were compared in terms of bias and mean square error (MSE) of the slope estimates, and Type I error and statistical power rates for the slope statistical tests. The results of the simulation study did not vary as a function of the residual variance estimator. All transformation methods provided negatively biased estimates, but both bias and MSE were reasonably small in all cases. In contrast, important differences were found regarding statistical tests, with the method proposed by Knapp and Hartung showing a better adjustment to the nominal significance level and higher power rates than the standard method.

Suggested Citation

  • José Antonio López-López & Juan Botella & Julio Sánchez-Meca & Fulgencio Marín-Martínez, 2013. "Alternatives for Mixed-Effects Meta-Regression Models in the Reliability Generalization Approach," Journal of Educational and Behavioral Statistics, , vol. 38(5), pages 443-469, October.
    • Handle: RePEc:sae:jedbes:v:38:y:2013:i:5:p:443-469
    DOI: 10.3102/1076998612466142
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998612466142
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998612466142?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Roger M. Harbord & Julian P.T. Higgins, 2008. "Meta-regression in Stata," Stata Journal, StataCorp LP, vol. 8(4), pages 493-519, December.
    2. Viechtbauer, Wolfgang, 2010. "Conducting Meta-Analyses in R with the metafor Package," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 36(i03).
    3. A. Hakstian & Thomas Whalen, 1976. "A k-sample significance test for independent alpha coefficients," Psychometrika, Springer;The Psychometric Society, vol. 41(2), pages 219-231, June.
    4. Leonard Feldt, 1969. "A test of the hypothesis that cronbach's alpha or kuder-richardson coefficent twenty is the same for two tests," Psychometrika, Springer;The Psychometric Society, vol. 34(3), pages 363-373, September.
    5. Martin, Andrew D. & Quinn, Kevin M. & Park, Jong Hee, 2011. "MCMCpack: Markov Chain Monte Carlo in R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 42(i09).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Peter M. Bentler, 2016. "Covariate-free and Covariate-dependent Reliability," Psychometrika, Springer;The Psychometric Society, vol. 81(4), pages 907-920, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Marsha Schroeder & A. Ralph Hakstian, 1990. "Inferential procedures for multifaceted coefficients of generalizability," Psychometrika, Springer;The Psychometric Society, vol. 55(3), pages 429-447, September.
    2. White, Robin R. & Brady, Michael, 2014. "Can consumers’ willingness to pay incentivize adoption of environmental impact reducing technologies in meat animal production?," Food Policy, Elsevier, vol. 49(P1), pages 41-49.
    3. Bart Verkuil & Serpil Atasayi & Marc L Molendijk, 2015. "Workplace Bullying and Mental Health: A Meta-Analysis on Cross-Sectional and Longitudinal Data," PLOS ONE, Public Library of Science, vol. 10(8), pages 1-16, August.
    4. Francesca Pilotto & Ingolf Kühn & Rita Adrian & Renate Alber & Audrey Alignier & Christopher Andrews & Jaana Bäck & Luc Barbaro & Deborah Beaumont & Natalie Beenaerts & Sue Benham & David S. Boukal & , 2020. "Meta-analysis of multidecadal biodiversity trends in Europe," Nature Communications, Nature, vol. 11(1), pages 1-11, December.
    5. repec:cup:judgdm:v:15:y:2020:i:6:p:972-988 is not listed on IDEAS
    6. Jonas Schmidt & Tammo H. A. Bijmolt, 2020. "Accurately measuring willingness to pay for consumer goods: a meta-analysis of the hypothetical bias," Journal of the Academy of Marketing Science, Springer, vol. 48(3), pages 499-518, May.
    7. Zabaloy, Maria Florencia & Viego, Valentina, 2022. "Household electricity demand in Latin America and the Caribbean: A meta-analysis of price elasticity," Utilities Policy, Elsevier, vol. 75(C).
    8. Mario Herberz & Tobias Brosch & Ulf J. J. Hahnel, 2020. "Kilo what? Default units increase value sensitivity in joint evaluations of energy efficiency," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 15(6), pages 972-988, November.
    9. Piers Steel & Sjoerd Beugelsdijk & Herman Aguinis, 2021. "The anatomy of an award-winning meta-analysis: Recommendations for authors, reviewers, and readers of meta-analytic reviews," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 52(1), pages 23-44, February.
    10. Masood Gheasi & Noriko Ishikawa & Karima Kourtit & Peter Nijkamp, 2019. "A meta-analysis of human health differences in urban and rural environments," Letters in Spatial and Resource Sciences, Springer, vol. 12(3), pages 167-186, December.
    11. Augusteijn, Hilde Elisabeth Maria & van Aert, Robbie Cornelis Maria & van Assen, Marcel A. L. M., 2021. "Posterior Probabilities of Effect Sizes and Heterogeneity in Meta-Analysis: An Intuitive Approach of Dealing with Publication Bias," OSF Preprints avkgj, Center for Open Science.
    12. Bakar, Khandoker Shuvo & Sahu, Sujit K., 2015. "spTimer: Spatio-Temporal Bayesian Modeling Using R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 63(i15).
    13. Georgiou, George K. & Guo, Kan & Naveenkumar, Nithya & Vieira, Ana Paula Alves & Das, J.P., 2020. "PASS theory of intelligence and academic achievement: A meta-analytic review," Intelligence, Elsevier, vol. 79(C).
    14. Mehmet Ugur & Sefa Awaworyi Churchill & Edna Solomon, 2018. "Technological Innovation And Employment In Derived Labour Demand Models: A Hierarchical Meta†Regression Analysis," Journal of Economic Surveys, Wiley Blackwell, vol. 32(1), pages 50-82, February.
    15. Murat Genc & Masood Gheasi & Peter Nijkamp & Jacques Poot, 2012. "The impact of immigration on international trade: a meta-analysis," Chapters, in: Peter Nijkamp & Jacques Poot & Mediha Sahin (ed.), Migration Impact Assessment, chapter 9, pages 301-337, Edward Elgar Publishing.
    16. Stephan Kambach & Ingolf Kühn & Bastien Castagneyrol & Helge Bruelheide, 2016. "The Impact of Tree Diversity on Different Aspects of Insect Herbivory along a Global Temperature Gradient - A Meta-Analysis," PLOS ONE, Public Library of Science, vol. 11(11), pages 1-14, November.
    17. Baştürk, Nalan & Grassi, Stefano & Hoogerheide, Lennart & Opschoor, Anne & van Dijk, Herman K., 2017. "The R Package MitISEM: Efficient and Robust Simulation Procedures for Bayesian Inference," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 79(i01).
    18. Maria Jose Sanchez-Andrades & Maria Jesus Vinolo-Gil & María Jesús Casuso-Holgado & Javier Barón-López & Manuel Rodríguez-Huguet & Rocío Martín-Valero, 2023. "Measurement Properties of Self-Report Questionnaires for Amyotrophic Lateral Sclerosis: A Systematic Review and Meta-Analysis of Commonly Used Instruments," IJERPH, MDPI, vol. 20(4), pages 1-19, February.
    19. Nan Wang & Yuxiang Luan & Rui Ma, 2024. "Detecting causal relationships between work motivation and job performance: a meta-analytic review of cross-lagged studies," Palgrave Communications, Palgrave Macmillan, vol. 11(1), pages 1-10, December.
    20. repec:cup:judgdm:v:14:y:2019:i:3:p:234-279 is not listed on IDEAS
    21. de la Cruz, Vera Ysabel V. & Tantriani, & Cheng, Weiguo & Tawaraya, Keitaro, 2023. "Yield gap between organic and conventional farming systems across climate types and sub-types: A meta-analysis," Agricultural Systems, Elsevier, vol. 211(C).
    22. Brian K Miller & Kay M Nicols & Silvia Clark & Alison Daniels & Whitney Grant, 2018. "Meta-analysis of coefficient alpha for scores on the Narcissistic Personality Inventory," PLOS ONE, Public Library of Science, vol. 13(12), pages 1-16, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:38:y:2013:i:5:p:443-469. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.