IDEAS home Printed from https://ideas.repec.org/a/bla/biomet/v76y2020i4p1240-1250.html
   My bibliography  Save this article

Testing small study effects in multivariate meta‐analysis

Author

Listed:
  • Chuan Hong
  • Georgia Salanti
  • Sally C. Morton
  • Richard D. Riley
  • Haitao Chu
  • Stephen E. Kimmel
  • Yong Chen

Abstract

Small study effects occur when smaller studies show different, often larger, treatment effects than large ones, which may threaten the validity of systematic reviews and meta‐analyses. The most well‐known reasons for small study effects include publication bias, outcome reporting bias, and clinical heterogeneity. Methods to account for small study effects in univariate meta‐analysis have been extensively studied. However, detecting small study effects in a multivariate meta‐analysis setting remains an untouched research area. One of the complications is that different types of selection processes can be involved in the reporting of multivariate outcomes. For example, some studies may be completely unpublished while others may selectively report multiple outcomes. In this paper, we propose a score test as an overall test of small study effects in multivariate meta‐analysis. Two detailed case studies are given to demonstrate the advantage of the proposed test over various naive applications of univariate tests in practice. Through simulation studies, the proposed test is found to retain nominal Type I error rates with considerable power in moderate sample size settings. Finally, we also evaluate the concordance between the proposed tests with the naive application of univariate tests by evaluating 44 systematic reviews with multiple outcomes from the Cochrane Database.

Suggested Citation

  • Chuan Hong & Georgia Salanti & Sally C. Morton & Richard D. Riley & Haitao Chu & Stephen E. Kimmel & Yong Chen, 2020. "Testing small study effects in multivariate meta‐analysis," Biometrics, The International Biometric Society, vol. 76(4), pages 1240-1250, December.
    • Handle: RePEc:bla:biomet:v:76:y:2020:i:4:p:1240-1250
    DOI: 10.1111/biom.13342
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/biom.13342
    Download Restriction: no
    File URL: https://libkey.io/10.1111/biom.13342?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Richard E. Chandler & Steven Bate, 2007. "Inference for clustered data using the independence loglikelihood," Biometrika, Biometrika Trust, vol. 94(1), pages 167-183.
    2. Kerry Dwan & Carrol Gamble & Paula R Williamson & Jamie J Kirkham & the Reporting Bias Group, 2013. "Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias — An Updated Review," PLOS ONE, Public Library of Science, vol. 8(7), pages 1-37, July.
    3. Yong Chen & Kung-Yee Liang, 2010. "On the asymptotic behaviour of the pseudolikelihood ratio test statistic with boundary problems," Biometrika, Biometrika Trust, vol. 97(3), pages 603-620.
    4. Sue Duval & Richard Tweedie, 2000. "Trim and Fill: A Simple Funnel-Plot–Based Method of Testing and Adjusting for Publication Bias in Meta-Analysis," Biometrics, The International Biometric Society, vol. 56(2), pages 455-463, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Je-Young Lee & Minkyung Baek, 2023. "Effects of Gamification on Students’ English Language Proficiency: A Meta-Analysis on Research in South Korea," Sustainability, MDPI, vol. 15(14), pages 1-19, July.
    2. Yucheon Kim & Songyi Lee, 2023. "A Systematic Review and Meta-Analysis of the Effectiveness of Non-Face-to-Face Coaching," Sustainability, MDPI, vol. 15(12), pages 1-15, June.
    3. Ana-Maria Staicu & Yingxing Li & Ciprian M. Crainiceanu & David Ruppert, 2014. "Likelihood Ratio Tests for Dependent Data with Applications to Longitudinal and Functional Data Analysis," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 41(4), pages 932-949, December.
    4. Qian Li & Yan Chen & Shikun Sun & Muyuan Zhu & Jing Xue & Zihan Gao & Jinfeng Zhao & Yihe Tang, 2022. "Research on Crop Irrigation Schedules Under Deficit Irrigation—A Meta-analysis," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 36(12), pages 4799-4817, September.
    5. Bart Verkuil & Serpil Atasayi & Marc L Molendijk, 2015. "Workplace Bullying and Mental Health: A Meta-Analysis on Cross-Sectional and Longitudinal Data," PLOS ONE, Public Library of Science, vol. 10(8), pages 1-16, August.
    6. Damiano Pizzol & Mike Trott & Igor Grabovac & Mario Antunes & Anna Claudia Colangelo & Simona Ippoliti & Cristian Petre Ilie & Anne Carrie & Nicola Veronese & Lee Smith, 2021. "Laparoscopy in Low-Income Countries: 10-Year Experience and Systematic Literature Review," IJERPH, MDPI, vol. 18(11), pages 1-11, May.
    7. Ferrero, Marta & Vadillo, Miguel A. & León, Samuel P., 2021. "A valid evaluation of the theory of multiple intelligences is not yet possible: Problems of methodological quality for intervention studies," Intelligence, Elsevier, vol. 88(C).
    8. Wolfgang Goymann & John C. Wingfield, 2014. "Male-to-female testosterone ratios, dimorphism, and life history—what does it really tell us?," Behavioral Ecology, International Society for Behavioral Ecology, vol. 25(4), pages 685-699.
    9. Alderotti, Giammarco & Rapallini, Chiara & Traverso, Silvio, 2023. "The Big Five personality traits and earnings: A meta-analysis," Journal of Economic Psychology, Elsevier, vol. 94(C).
    10. Zhigang Chen & Jiye Zhu & Chaoyuan Huang & Fang Lian & Guobin Wu & Yinnong Zhao, 2015. "The Association between Three Cyclooxygenase-2 Polymorphisms and Hepatocellular Carcinoma Risk: A Meta-Analysis," PLOS ONE, Public Library of Science, vol. 10(3), pages 1-12, March.
    11. Ünal, Zehra E. & Kartal, Gamze & Ulusoy, Serra & Ala, Aslı M. & Yilmaz, Munube & Geary, David C., 2023. "Relative contributions of g and basic domain-specific mathematics skills to complex mathematics competencies," Intelligence, Elsevier, vol. 101(C).
    12. Jo Smith & Xiao Qi Ang & Emma L. Giles & Gemma Traviss-Turner, 2023. "Emotional Eating Interventions for Adults Living with Overweight or Obesity: A Systematic Review and Meta-Analysis," IJERPH, MDPI, vol. 20(3), pages 1-22, February.
    13. Daniele Zago & Maria Eugênia Andrighetto Canozzi & Júlio Otávio Jardim Barcellos, 2020. "Pregnant beef cow’s nutrition and its effects on postnatal weight and carcass quality of their progeny," PLOS ONE, Public Library of Science, vol. 15(8), pages 1-20, August.
    14. Jenny Jing Wen Liu & Natalie Ein & Julia Gervasio & Mira Battaion & Kenneth Fung, 2022. "The Pursuit of Resilience: A Meta-Analysis and Systematic Review of Resilience-Promoting Interventions," Journal of Happiness Studies, Springer, vol. 23(4), pages 1771-1791, April.
    15. Viktoria Maria Baumeister & Leonie Petra Kuen & Maike Bruckes & Gerhard Schewe, 2021. "The Relationship of Work-Related ICT Use With Well-being, Incorporating the Role of Resources and Demands: A Meta-Analysis," SAGE Open, , vol. 11(4), pages 21582440211, November.
    16. Xiao Yan Chen & Camilla K. M. Lo & Ko Ling Chan & Wing Cheong Leung & Patrick Ip, 2022. "Association between Childhood Exposure to Family Violence and Telomere Length: A Meta-Analysis," IJERPH, MDPI, vol. 19(19), pages 1-18, September.
    17. Hsiu-Man Lin & Yu-Tzu Chang & Meng-Hsiang Chen & Shu-Tsen Liu & Bo-Shen Chen & Lin Li & Chiao-Yu Lee & Yu-Ru Sue & Tsai-Mei Sung & Cheuk-Kwan Sun & Pin-Yang Yeh, 2022. "Structural and Functional Neural Correlates in Individuals with Excessive Smartphone Use: A Systematic Review and Meta-Analysis," IJERPH, MDPI, vol. 19(23), pages 1-15, December.
    18. Gundula Krack, 2019. "How to make value-based health insurance designs more effective? A systematic review and meta-analysis," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 20(6), pages 841-856, August.
    19. Nyeonju Kang & James H Cauraugh, 2017. "Does non-invasive brain stimulation reduce essential tremor? A systematic review and meta-analysis," PLOS ONE, Public Library of Science, vol. 12(9), pages 1-15, September.
    20. Angélica Pigola & Priscila Rezende Costa, 2022. "In search of understanding about knowledge and learning on innovation performance," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(7), pages 3995-4022, July.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:biomet:v:76:y:2020:i:4:p:1240-1250. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.blackwellpublishing.com/journal.asp?ref=0006-341X .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.