IDEAS home Printed from https://ideas.repec.org/a/gam/jijerp/v18y2021i17p9304-d628408.html
   My bibliography  Save this article

Multiplicity Eludes Peer Review: The Case of COVID-19 Research

Author

Listed:
  • Oliver Gutiérrez-Hernández

    (Department of Geography, University of Málaga, 29010 Málaga, Spain)

  • Luis Ventura García

    (Institute of Natural Resources and Agrobiology of Seville (IRNAS), Spanish National Research Council (CSIC), 41012 Seville, Spain)

Abstract

Multiplicity arises when data analysis involves multiple simultaneous inferences, increasing the chance of spurious findings. It is a widespread problem frequently ignored by researchers. In this paper, we perform an exploratory analysis of the Web of Science database for COVID-19 observational studies. We examined 100 top-cited COVID-19 peer-reviewed articles based on p -values, including up to 7100 simultaneous tests, with 50% including >34 tests, and 20% > 100 tests. We found that the larger the number of tests performed, the larger the number of significant results (r = 0.87, p < 10 ?6 ). The number of p -values in the abstracts was not related to the number of p -values in the papers. However, the highly significant results ( p < 0.001) in the abstracts were strongly correlated (r = 0.61, p < 10 ?6 ) with the number of p < 0.001 significances in the papers. Furthermore, the abstracts included a higher proportion of significant results (0.91 vs. 0.50), and 80% reported only significant results. Only one reviewed paper addressed multiplicity-induced type I error inflation, pointing to potentially spurious results bypassing the peer-review process. We conclude the need to pay special attention to the increased chance of false discoveries in observational studies, including non-replicated striking discoveries with a potentially large social impact. We propose some easy-to-implement measures to assess and limit the effects of multiplicity.

Suggested Citation

  • Oliver Gutiérrez-Hernández & Luis Ventura García, 2021. "Multiplicity Eludes Peer Review: The Case of COVID-19 Research," IJERPH, MDPI, vol. 18(17), pages 1-10, September.
    • Handle: RePEc:gam:jijerp:v:18:y:2021:i:17:p:9304-:d:628408
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1660-4601/18/17/9304/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1660-4601/18/17/9304/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Rebecca A. Betensky, 2019. "The p-Value Requires Context, Not a Threshold," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 115-117, March.
    2. Valentin Amrhein & Sander Greenland & Blake McShane, 2019. "Scientists rise up against statistical significance," Nature, Nature, vol. 567(7748), pages 305-307, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jae H. Kim, 2022. "Moving to a world beyond p-value," Review of Managerial Science, Springer, vol. 16(8), pages 2467-2493, November.
    2. Mia Papasideris & Scott T Leatherdale & Kate Battista & Peter A Hall, 2021. "An examination of the prospective association between physical activity and academic achievement in youth at the population level," PLOS ONE, Public Library of Science, vol. 16(6), pages 1-15, June.
    3. Surhan Cam & Serap Palaz, 2023. "Mutual interests management with a purposive approach: Evidence from the Turkish shipyards for an amorphous impact model between (subjective) well‐being and performance," Industrial Relations Journal, Wiley Blackwell, vol. 54(1), pages 40-70, January.
    4. Siddharth Sareen & Andrea Saltelli & Kjetil Rommetveit, 2020. "Ethics of quantification: illumination, obfuscation and performative legitimation," Palgrave Communications, Palgrave Macmillan, vol. 6(1), pages 1-5, December.
    5. Tracy, Elizabeth M. & Billingsley, Joseph & Pollack, Jeffrey M. & Barber, Dennis & Beorchia, Ace & Carr, Jon C. & Gonzalez, Gabe & Harris, Michael L. & Michaelis, Timothy L. & Morrow, Grayson & Philli, 2021. "A behavioral insights approach to recruiting entrepreneurs for an academic study during the COVID-19 pandemic," Journal of Business Venturing Insights, Elsevier, vol. 16(C).
    6. Lars Mewes & Leonie Tuitjer & Peter Dirksmeier, 2024. "Exploring the variances of climate change opinions in Germany at a fine-grained local scale," Nature Communications, Nature, vol. 15(1), pages 1-14, December.
    7. Denis Fougère & Nicolas Jacquemet, 2020. "Policy Evaluation Using Causal Inference Methods," SciencePo Working papers Main hal-03455978, HAL.
    8. Kristin B. Dobbin & Amanda L. Fencl & Gregory Pierce & Melissa Beresford & Silvia Gonzalez & Wendy Jepson, 2023. "Understanding perceived climate risks to household water supply and their implications for adaptation: evidence from California," Climatic Change, Springer, vol. 176(4), pages 1-20, April.
    9. Abbas, Sadia & Adapa, Sujana & Sheridan, Alison & Azeem, Muhammad Masood, 2022. "Informal competition and firm level innovation in South Asia: The moderating role of innovation time off and R&D intensity," Technological Forecasting and Social Change, Elsevier, vol. 181(C).
    10. Elizabeth P.D. Koselka & Lucy C. Weidner & Arseniy Minasov & Marc G. Berman & William R. Leonard & Marianne V. Santoso & Junia N. de Brito & Zachary C. Pope & Mark A. Pereira & Teresa H. Horton, 2019. "Walking Green: Developing an Evidence Base for Nature Prescriptions," IJERPH, MDPI, vol. 16(22), pages 1-18, November.
    11. Piotr Skórka & Beata Grzywacz & Dawid Moroń & Magdalena Lenda, 2020. "The macroecology of the COVID-19 pandemic in the Anthropocene," PLOS ONE, Public Library of Science, vol. 15(7), pages 1-17, July.
    12. Susan Zief & John Deke & Paul Burkander & Andrew Langan & Subuhi Asheer, "undated". "Impacts of a Home Visiting Program Enhanced with Content on Healthy Birth Spacing," Mathematica Policy Research Reports 230745c77416490bb743dae6f, Mathematica Policy Research.
    13. Le, Kien & Nguyen, My, 2020. "Shedding light on maternal education and child health in developing countries," World Development, Elsevier, vol. 133(C).
    14. Helmut Wasserbacher & Martin Spindler, 2024. "Credit Ratings: Heterogeneous Effect on Capital Structure," Papers 2406.18936, arXiv.org.
    15. Santini, Ziggi Ivan & Thygesen, Lau Caspar & Koyanagi, Ai & Stewart-Brown, Sarah & Meilstrup, Charlotte & Nielsen, Line & Olsen, Kim Rose & Birkjær, Michael & McDaid, David & Koushede, Vibeke & Ekholm, 2022. "Economics of mental wellbeing: a prospective study estimating associated productivity costs due to sickness absence from the workplace in Denmark," LSE Research Online Documents on Economics 116690, London School of Economics and Political Science, LSE Library.
    16. Palea, Vera & Drogo, Federico, 2020. "Carbon Emissions and the Cost of Debt Financing: What Role for Policy Commitment, Firm Disclosure and Corporate Governance?," Department of Economics and Statistics Cognetti de Martiis. Working Papers 202002, University of Turin.
    17. Chenxi Zhao & Chenglei Zhao & Minmin Zhao & Lin Wang & Jiawei Guo & Longhai Zhang & Yunfeng Li & Yuliang Sun & Ling Zhang & Zheng’ao Li & Wenfei Zhu, 2022. "Effect of Exergame Training on Working Memory and Executive Function in Older Adults," Sustainability, MDPI, vol. 14(17), pages 1-11, August.
    18. Tim Bothe & Josephine Jacob & Christoph Kröger & Jochen Walker, 2020. "How expensive are post-traumatic stress disorders? Estimating incremental health care and economic costs on anonymised claims data," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 21(6), pages 917-930, August.
    19. Pütz, Peter & Kramer-Sunderbrink, Arne & Dreher, Robin Tim & Hoffmann, Leona & Werner, Robin, 2022. "A Proposed Hybrid Effect Size Plus p-Value Criterion. A Comment on Goodman et al. (The American Statistician, 2019)," Journal of Comments and Replications in Economics (JCRE), ZBW - Leibniz Information Centre for Economics, vol. 1, pages 1-15.
    20. Guillaume Coqueret, 2023. "Forking paths in financial economics," Papers 2401.08606, arXiv.org.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jijerp:v:18:y:2021:i:17:p:9304-:d:628408. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.