IDEAS home Printed from https://ideas.repec.org/a/bla/biomet/v76y2020i4p1240-1250.html
   My bibliography  Save this article

Testing small study effects in multivariate meta‐analysis

Author

Listed:
  • Chuan Hong
  • Georgia Salanti
  • Sally C. Morton
  • Richard D. Riley
  • Haitao Chu
  • Stephen E. Kimmel
  • Yong Chen

Abstract

Small study effects occur when smaller studies show different, often larger, treatment effects than large ones, which may threaten the validity of systematic reviews and meta‐analyses. The most well‐known reasons for small study effects include publication bias, outcome reporting bias, and clinical heterogeneity. Methods to account for small study effects in univariate meta‐analysis have been extensively studied. However, detecting small study effects in a multivariate meta‐analysis setting remains an untouched research area. One of the complications is that different types of selection processes can be involved in the reporting of multivariate outcomes. For example, some studies may be completely unpublished while others may selectively report multiple outcomes. In this paper, we propose a score test as an overall test of small study effects in multivariate meta‐analysis. Two detailed case studies are given to demonstrate the advantage of the proposed test over various naive applications of univariate tests in practice. Through simulation studies, the proposed test is found to retain nominal Type I error rates with considerable power in moderate sample size settings. Finally, we also evaluate the concordance between the proposed tests with the naive application of univariate tests by evaluating 44 systematic reviews with multiple outcomes from the Cochrane Database.

Suggested Citation

  • Chuan Hong & Georgia Salanti & Sally C. Morton & Richard D. Riley & Haitao Chu & Stephen E. Kimmel & Yong Chen, 2020. "Testing small study effects in multivariate meta‐analysis," Biometrics, The International Biometric Society, vol. 76(4), pages 1240-1250, December.
    • Handle: RePEc:bla:biomet:v:76:y:2020:i:4:p:1240-1250
    DOI: 10.1111/biom.13342
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/biom.13342
    Download Restriction: no
    File URL: https://libkey.io/10.1111/biom.13342?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Richard E. Chandler & Steven Bate, 2007. "Inference for clustered data using the independence loglikelihood," Biometrika, Biometrika Trust, vol. 94(1), pages 167-183.
    2. Kerry Dwan & Carrol Gamble & Paula R Williamson & Jamie J Kirkham & the Reporting Bias Group, 2013. "Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias — An Updated Review," PLOS ONE, Public Library of Science, vol. 8(7), pages 1-37, July.
    3. Yong Chen & Kung-Yee Liang, 2010. "On the asymptotic behaviour of the pseudolikelihood ratio test statistic with boundary problems," Biometrika, Biometrika Trust, vol. 97(3), pages 603-620.
    4. Sue Duval & Richard Tweedie, 2000. "Trim and Fill: A Simple Funnel-Plot–Based Method of Testing and Adjusting for Publication Bias in Meta-Analysis," Biometrics, The International Biometric Society, vol. 56(2), pages 455-463, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Je-Young Lee & Minkyung Baek, 2023. "Effects of Gamification on Students’ English Language Proficiency: A Meta-Analysis on Research in South Korea," Sustainability, MDPI, vol. 15(14), pages 1-19, July.
    2. Ana-Maria Staicu & Yingxing Li & Ciprian M. Crainiceanu & David Ruppert, 2014. "Likelihood Ratio Tests for Dependent Data with Applications to Longitudinal and Functional Data Analysis," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 41(4), pages 932-949, December.
    3. Qian Li & Yan Chen & Shikun Sun & Muyuan Zhu & Jing Xue & Zihan Gao & Jinfeng Zhao & Yihe Tang, 2022. "Research on Crop Irrigation Schedules Under Deficit Irrigation—A Meta-analysis," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 36(12), pages 4799-4817, September.
    4. Bart Verkuil & Serpil Atasayi & Marc L Molendijk, 2015. "Workplace Bullying and Mental Health: A Meta-Analysis on Cross-Sectional and Longitudinal Data," PLOS ONE, Public Library of Science, vol. 10(8), pages 1-16, August.
    5. Damiano Pizzol & Mike Trott & Igor Grabovac & Mario Antunes & Anna Claudia Colangelo & Simona Ippoliti & Cristian Petre Ilie & Anne Carrie & Nicola Veronese & Lee Smith, 2021. "Laparoscopy in Low-Income Countries: 10-Year Experience and Systematic Literature Review," IJERPH, MDPI, vol. 18(11), pages 1-11, May.
    6. Wolfgang Goymann & John C. Wingfield, 2014. "Male-to-female testosterone ratios, dimorphism, and life history—what does it really tell us?," Behavioral Ecology, International Society for Behavioral Ecology, vol. 25(4), pages 685-699.
    7. Alderotti, Giammarco & Rapallini, Chiara & Traverso, Silvio, 2023. "The Big Five personality traits and earnings: A meta-analysis," Journal of Economic Psychology, Elsevier, vol. 94(C).
    8. Ünal, Zehra E. & Kartal, Gamze & Ulusoy, Serra & Ala, Aslı M. & Yilmaz, Munube & Geary, David C., 2023. "Relative contributions of g and basic domain-specific mathematics skills to complex mathematics competencies," Intelligence, Elsevier, vol. 101(C).
    9. Daniele Zago & Maria Eugênia Andrighetto Canozzi & Júlio Otávio Jardim Barcellos, 2020. "Pregnant beef cow’s nutrition and its effects on postnatal weight and carcass quality of their progeny," PLOS ONE, Public Library of Science, vol. 15(8), pages 1-20, August.
    10. Viktoria Maria Baumeister & Leonie Petra Kuen & Maike Bruckes & Gerhard Schewe, 2021. "The Relationship of Work-Related ICT Use With Well-being, Incorporating the Role of Resources and Demands: A Meta-Analysis," SAGE Open, , vol. 11(4), pages 21582440211, November.
    11. Gundula Krack, 2019. "How to make value-based health insurance designs more effective? A systematic review and meta-analysis," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 20(6), pages 841-856, August.
    12. Angélica Pigola & Priscila Rezende Costa, 2022. "In search of understanding about knowledge and learning on innovation performance," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(7), pages 3995-4022, July.
    13. Claire Godard-Sebillotte & Mélanie Le Berre & Tibor Schuster & Miguel Trottier & Isabelle Vedel, 2019. "Impact of health service interventions on acute hospital use in community-dwelling persons with dementia: A systematic literature review and meta-analysis," PLOS ONE, Public Library of Science, vol. 14(6), pages 1-18, June.
    14. Christopher Snyder & Ran Zhuo, 2018. "Sniff Tests as a Screen in the Publication Process: Throwing out the Wheat with the Chaff," NBER Working Papers 25058, National Bureau of Economic Research, Inc.
    15. Chuang Yuan & Jing Wang & Michael Ying, 2016. "Predictive Value of Carotid Distensibility Coefficient for Cardiovascular Diseases and All-Cause Mortality: A Meta-Analysis," PLOS ONE, Public Library of Science, vol. 11(4), pages 1-15, April.
    16. Francisco Javier Blanco-Encomienda & Rocío García-Cantero & María José Latorre-Medina, 2020. "Association between Work-Related Rumination, Work Environment and Employee Well-Being: A Meta-Analytic Study of Main and Moderator Effects," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 150(3), pages 887-910, August.
    17. Pedro L. Cosio & Manuel Crespo-Posadas & Álvaro Velarde-Sotres & Mireia Pelaez, 2021. "Effect of Chronic Resistance Training on Circulating Irisin: Systematic Review and Meta-Analysis of Randomized Controlled Trials," IJERPH, MDPI, vol. 18(5), pages 1-16, March.
    18. Costa, Rui J. & Wilkinson-Herbots, Hilde M., 2021. "Inference of gene flow in the process of speciation: Efficient maximum-likelihood implementation of a generalised isolation-with-migration model," Theoretical Population Biology, Elsevier, vol. 140(C), pages 1-15.
    19. Augusteijn, Hilde Elisabeth Maria & van Aert, Robbie Cornelis Maria & van Assen, Marcel A. L. M., 2021. "Posterior Probabilities of Effect Sizes and Heterogeneity in Meta-Analysis: An Intuitive Approach of Dealing with Publication Bias," OSF Preprints avkgj, Center for Open Science.
    20. Sesha Kethineni & Susan Frazier‐Kouassi & Yuki Shigemoto & Wesley Jennings & Stephanie M. Cardwell & Alex R. Piquero & Kimberly Gay & Dayanand Sundaravadivelu, 2021. "PROTOCOL: Effectiveness of parent‐engagement programs to reduce truancy and juvenile delinquency: A systematic review," Campbell Systematic Reviews, John Wiley & Sons, vol. 17(3), September.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:biomet:v:76:y:2020:i:4:p:1240-1250. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.blackwellpublishing.com/journal.asp?ref=0006-341X .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.