IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0204056.html
   My bibliography  Save this article

Bias caused by sampling error in meta-analysis with small sample sizes

Author

Listed:
  • Lifeng Lin

Abstract

Background: Meta-analyses frequently include studies with small sample sizes. Researchers usually fail to account for sampling error in the reported within-study variances; they model the observed study-specific effect sizes with the within-study variances and treat these sample variances as if they were the true variances. However, this sampling error may be influential when sample sizes are small. This article illustrates that the sampling error may lead to substantial bias in meta-analysis results. Methods: We conducted extensive simulation studies to assess the bias caused by sampling error. Meta-analyses with continuous and binary outcomes were simulated with various ranges of sample size and extents of heterogeneity. We evaluated the bias and the confidence interval coverage for five commonly-used effect sizes (i.e., the mean difference, standardized mean difference, odds ratio, risk ratio, and risk difference). Results: Sampling error did not cause noticeable bias when the effect size was the mean difference, but the standardized mean difference, odds ratio, risk ratio, and risk difference suffered from this bias to different extents. The bias in the estimated overall odds ratio and risk ratio was noticeable even when each individual study had more than 50 samples under some settings. Also, Hedges’ g, which is a bias-corrected estimate of the standardized mean difference within studies, might lead to larger bias than Cohen’s d in meta-analysis results. Conclusions: Cautions are needed to perform meta-analyses with small sample sizes. The reported within-study variances may not be simply treated as the true variances, and their sampling error should be fully considered in such meta-analyses.

Suggested Citation

  • Lifeng Lin, 2018. "Bias caused by sampling error in meta-analysis with small sample sizes," PLOS ONE, Public Library of Science, vol. 13(9), pages 1-19, September.
  • Handle: RePEc:plo:pone00:0204056
    DOI: 10.1371/journal.pone.0204056
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0204056
    Download Restriction: no

    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0204056&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pone.0204056?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Lifeng Lin & Haitao Chu & James S. Hodges, 2017. "Alternative measures of between-study heterogeneity in meta-analysis: Reducing the impact of outlying studies," Biometrics, The International Biometric Society, vol. 73(1), pages 156-166, March.
    2. Jessica Gurevitch & Julia Koricheva & Shinichi Nakagawa & Gavin Stewart, 2018. "Meta-analysis and the science of research synthesis," Nature, Nature, vol. 555(7695), pages 175-182, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Samira Gonzalez-Hoelling & Carme Bertran-Noguer & Gloria Reig-Garcia & Rosa Suñer-Soler, 2021. "Effects of a Music-Based Rhythmic Auditory Stimulation on Gait and Balance in Subacute Stroke," IJERPH, MDPI, vol. 18(4), pages 1-14, February.
    2. Fahad M. Al Amer & Christopher G. Thompson & Lifeng Lin, 2021. "Bayesian Methods for Meta-Analyses of Binary Outcomes: Implementations, Examples, and Impact of Priors," IJERPH, MDPI, vol. 18(7), pages 1-14, March.
    3. Chaikumbung, Mayula, 2021. "Institutions and consumer preferences for renewable energy: A meta-regression analysis," Renewable and Sustainable Energy Reviews, Elsevier, vol. 146(C).
    4. Abbe Muller & David Vlahov & Matthew J. Akiyama & Ann Kurth, 2020. "Hepatitis C Reinfection in People Who Inject Drugs in Resource-Limited Countries: A Systematic Review and Analysis," IJERPH, MDPI, vol. 17(14), pages 1-11, July.
    5. Sean Cowlishaw & Isabella Freijah & Dzenana Kartal & Alyssa Sbisa & Ashlee Mulligan & MaryAnn Notarianni & Anne-Laure Couineau & David Forbes & Meaghan O’Donnell & Andrea Phelps & Katherine M. Iverson, 2022. "Intimate Partner Violence (IPV) in Military and Veteran Populations: A Systematic Review of Population-Based Surveys and Population Screening Studies," IJERPH, MDPI, vol. 19(14), pages 1-23, July.
    6. Laura A. Outhwaite & Erin Early & Christothea Herodotou & Jo Van Herwegen, 2023. "Can Maths Apps Add Value to Learning? A Systematic Review," CEPEO Working Paper Series 23-02, UCL Centre for Education Policy and Equalising Opportunities, revised Jan 2023.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Oded Berger-Tal & Alison L Greggor & Biljana Macura & Carrie Ann Adams & Arden Blumenthal & Amos Bouskila & Ulrika Candolin & Carolina Doran & Esteban Fernández-Juricic & Kiyoko M Gotanda & Catherine , 2019. "Systematic reviews and maps as tools for applying behavioral ecology to management and policy," Behavioral Ecology, International Society for Behavioral Ecology, vol. 30(1), pages 1-8.
    2. Phu Nguyen-Van & Anne Stenger & Tuyen Tiet, 2021. "Social incentive factors in interventions promoting sustainable behaviors: A meta-analysis," PLOS ONE, Public Library of Science, vol. 16(12), pages 1-27, December.
    3. Jinhui Li & Zhengyi Deng & Simon John Christoph Soerensen & Linda Kachuri & Andres Cardenas & Rebecca E. Graff & John T. Leppert & Marvin E. Langston & Benjamin I. Chung, 2024. "Ambient air pollution and urological cancer risk: A systematic review and meta-analysis of epidemiological evidence," Nature Communications, Nature, vol. 15(1), pages 1-17, December.
    4. Irsova, Zuzana & Doucouliagos, Hristos & Havranek, Tomas & Stanley, T. D., 2023. "Meta-Analysis of Social Science Research: A Practitioner’s Guide," EconStor Preprints 273719, ZBW - Leibniz Information Centre for Economics.
    5. Alexander L. Brown & Taisuke Imai & Ferdinand M. Vieider & Colin F. Camerer, 2024. "Meta-analysis of Empirical Estimates of Loss Aversion," Journal of Economic Literature, American Economic Association, vol. 62(2), pages 485-516, June.
    6. Ruhaimatu Abudu & Raj Bridgelall, 2024. "Autonomous Ships: A Thematic Review," World, MDPI, vol. 5(2), pages 1-17, April.
    7. Lukas Hafner & Maxime Pichon & Christophe Burucoa & Sophie H. A. Nusser & Alexandra Moura & Marc Garcia-Garcera & Marc Lecuit, 2021. "Listeria monocytogenes faecal carriage is common and depends on the gut microbiota," Nature Communications, Nature, vol. 12(1), pages 1-13, December.
    8. Kulinskaya, Elena & Mah, Eung Yaw, 2021. "Simulation results on the performance of statistical methods in cumulative meta analysis," MetaArXiv 8t4pf, Center for Open Science.
    9. Christopher Hansen & Holger Steinmetz & Jörn Block, 2022. "How to conduct a meta-analysis in eight steps: a practical guide," Management Review Quarterly, Springer, vol. 72(1), pages 1-19, February.
    10. Denis Fougère & Arthur Heim, 2019. "L'évaluation socioéconomique de l'investissement social," Working Papers hal-03456048, HAL.
    11. In-Soo Shin & Juh-Hyun Shin & Dong-Eun Jang & Jiyeon Lee, 2022. "A Methodological Quality Evaluation of Meta-Analyses on Nursing Home Research: Overview and Suggestions for Future Directions," IJERPH, MDPI, vol. 19(1), pages 1-19, January.
    12. Fabio De Felice & Ilaria Baffo & Antonella Petrillo, 2022. "Critical Infrastructures Overview: Past, Present and Future," Sustainability, MDPI, vol. 14(4), pages 1-20, February.
    13. Na Wang & Debra K. Creedy & Mingna Zhang & Hong Lu & Elizabeth Elder & Jyai Allen & Li Guo & Qian Xiao & Jenny Gamble, 2022. "Designing a Needs-Oriented Psychological Intervention for Chinese Women Undergoing an Abortion," IJERPH, MDPI, vol. 20(1), pages 1-12, December.
    14. Subroy, Vandana & Gunawardena, Asha & Polyakov, Maksym & Pandit, Ram & Pannell, David J., 2019. "The worth of wildlife: A meta-analysis of global non-market values of threatened species," Ecological Economics, Elsevier, vol. 164(C), pages 1-1.
    15. Wenqiang Xie & Xiaodong Yan, 2023. "Responses of Wheat Protein Content and Protein Yield to Future Climate Change in China during 2041–2060," Sustainability, MDPI, vol. 15(19), pages 1-22, September.
    16. Antonino Malacrinò & Victoria A Sadowski & Tvisha K Martin & Nathalia Cavichiolli de Oliveira & Ian J Brackett & James D Feller & Kristian J Harris & Orlando Combita Heredia & Rosa Vescio & Alison E B, 2020. "Biological invasions alter environmental microbiomes: A meta-analysis," PLOS ONE, Public Library of Science, vol. 15(10), pages 1-12, October.
    17. Fahui Jiang & Shangshu Huang & Yan Wu & Mahbub Ul Islam & Fangjin Dong & Zhen Cao & Guohui Chen & Yuming Guo, 2022. "A Large-Scale Dataset of Conservation and Deep Tillage in Mollisols, Northeast Plain, China," Data, MDPI, vol. 8(1), pages 1-15, December.
    18. Fátima L. Vieira & Paulo A. Vieira & Denis A. Coelho, 2019. "A Data-Driven Approach to Development of a Taxonomy Framework for Triple Bottom Line Metrics," Sustainability, MDPI, vol. 11(9), pages 1-17, May.
    19. Matthew J Page & Joanne E McKenzie & Patrick M Bossuyt & Isabelle Boutron & Tammy C Hoffmann & Cynthia D Mulrow & Larissa Shamseer & Jennifer M Tetzlaff & Elie A Akl & Sue E Brennan & Roger Chou & Jul, 2021. "The PRISMA 2020 statement: An updated guideline for reporting systematic reviews," PLOS Medicine, Public Library of Science, vol. 18(3), pages 1-15, March.
    20. Junwen Zhu & Weishu Liu, 2020. "A tale of two databases: the use of Web of Science and Scopus in academic papers," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(1), pages 321-335, April.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0204056. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.