IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0166387.html
   My bibliography  Save this article

The Global Burden of Journal Peer Review in the Biomedical Literature: Strong Imbalance in the Collective Enterprise

Author

Listed:
  • Michail Kovanis
  • Raphaël Porcher
  • Philippe Ravaud
  • Ludovic Trinquart

Abstract

The growth in scientific production may threaten the capacity for the scientific community to handle the ever-increasing demand for peer review of scientific publications. There is little evidence regarding the sustainability of the peer-review system and how the scientific community copes with the burden it poses. We used mathematical modeling to estimate the overall quantitative annual demand for peer review and the supply in biomedical research. The modeling was informed by empirical data from various sources in the biomedical domain, including all articles indexed at MEDLINE. We found that for 2015, across a range of scenarios, the supply exceeded by 15% to 249% the demand for reviewers and reviews. However, 20% of the researchers performed 69% to 94% of the reviews. Among researchers actually contributing to peer review, 70% dedicated 1% or less of their research work-time to peer review while 5% dedicated 13% or more of it. An estimated 63.4 million hours were devoted to peer review in 2015, among which 18.9 million hours were provided by the top 5% contributing reviewers. Our results support that the system is sustainable in terms of volume but emphasizes a considerable imbalance in the distribution of the peer-review effort across the scientific community. Finally, various individual interactions between authors, editors and reviewers may reduce to some extent the number of reviewers who are available to editors at any point.

Suggested Citation

  • Michail Kovanis & Raphaël Porcher & Philippe Ravaud & Ludovic Trinquart, 2016. "The Global Burden of Journal Peer Review in the Biomedical Literature: Strong Imbalance in the Collective Enterprise," PLOS ONE, Public Library of Science, vol. 11(11), pages 1-14, November.
    • Handle: RePEc:plo:pone00:0166387
    DOI: 10.1371/journal.pone.0166387
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0166387
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0166387&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0166387?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Sascha Ott & Daniel Hebenstreit, 2014. "Apply market forces to peer review," Nature, Nature, vol. 506(7488), pages 295-295, February.
    2. Michail Kovanis & Raphaël Porcher & Philippe Ravaud & Ludovic Trinquart, 2016. "Complex systems approach to scientific publication and peer-review system: development of an agent-based model calibrated with empirical journal data," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(2), pages 695-715, February.
    3. Martijn Arns, 2014. "Open access is tiring out peer reviewers," Nature, Nature, vol. 515(7528), pages 467-467, November.
    4. Adrian Mulligan & Louise Hall & Ellen Raphael, 2013. "Peer review in a changing world: An international study measuring the attitudes of researchers," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(1), pages 132-161, January.
    5. Adrian Mulligan & Louise Hall & Ellen Raphael, 2013. "Peer review in a changing world: An international study measuring the attitudes of researchers," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(1), pages 132-161, January.
    6. Raj Chetty & Emmanuel Saez & Laszlo Sandor, 2014. "What Policies Increase Prosocial Behavior? An Experiment with Referees at the Journal of Public Economics," Journal of Economic Perspectives, American Economic Association, vol. 28(3), pages 169-188, Summer.
    7. In-Uck Park & Mike W. Peacey & Marcus R. Munafò, 2014. "Modelling the effects of subjective and objective decision making in scientific peer review," Nature, Nature, vol. 506(7486), pages 93-96, February.
    8. Robert H. S. Kraus, 2016. "Matchmaker aims to cut journal shopping," Nature, Nature, vol. 531(7595), pages 448-448, March.
    9. Kendall Powell, 2016. "Does it take too long to publish research?," Nature, Nature, vol. 530(7589), pages 148-151, February.
    10. Tim Vines & Loren Rieseberg & Harry Smith, 2010. "No crisis in supply of peer reviewers," Nature, Nature, vol. 468(7327), pages 1041-1041, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Sarvenaz Sarabipour & Humberto J Debat & Edward Emmott & Steven J Burgess & Benjamin Schwessinger & Zach Hensel, 2019. "On the value of preprints: An early career researcher perspective," PLOS Biology, Public Library of Science, vol. 17(2), pages 1-12, February.
    2. Bravo, Giangiacomo & Farjam, Mike & Grimaldo Moreno, Francisco & Birukou, Aliaksandr & Squazzoni, Flaminio, 2018. "Hidden connections: Network effects on editorial decisions in four computer science journals," Journal of Informetrics, Elsevier, vol. 12(1), pages 101-112.
    3. Lin Zhang & Yuanyuan Shang & Ying Huang & Gunnar Sivertsen, 2022. "Gender differences among active reviewers: an investigation based on publons," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(1), pages 145-179, January.
    4. Zhang, Lin & Shang, Yuanyuan & HUANG, Ying & Sivertsen, Gunnar, 2021. "Gender differences among active reviewers: an investigation based on Publons," SocArXiv 4z6w8, Center for Open Science.
    5. Alexander Maedche & Edona Elshan & Hartmut Höhle & Christiane Lehrer & Jan Recker & Ali Sunyaev & Benjamin Sturm & Oliver Werth, 2024. "Open Science," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 66(4), pages 517-532, August.
    6. Federico Bianchi & Francisco Grimaldo & Giangiacomo Bravo & Flaminio Squazzoni, 2018. "The peer review game: an agent-based model of scientists facing resource constraints and institutional pressures," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 1401-1420, September.
    7. John P A Ioannidis, 2018. "Meta-research: Why research on research matters," PLOS Biology, Public Library of Science, vol. 16(3), pages 1-6, March.
    8. Rafael D’Andrea & James P O’Dwyer, 2017. "Can editors save peer review from peer reviewers?," PLOS ONE, Public Library of Science, vol. 12(10), pages 1-14, October.
    9. Bianchi, Federico & Grimaldo, Francisco & Squazzoni, Flaminio, 2019. "The F3-index. Valuing reviewers for scholarly journals," Journal of Informetrics, Elsevier, vol. 13(1), pages 78-86.
    10. ederico Bianchi & Flaminio Squazzoni, 2022. "Can transparency undermine peer review? A simulation model of scientist behavior under open peer review [Reviewing Peer Review]," Science and Public Policy, Oxford University Press, vol. 49(5), pages 791-800.
    11. Sun, Zhuanlan & He, Dongjin & Li, Yiwei, 2024. "How the readability of manuscript before journal submission advantages peer review process: Evidence from biomedical scientific publications," Journal of Informetrics, Elsevier, vol. 18(3).
    12. Lin Zhang & Ziyi Tu & Yifei Yu & Yuanyuan Shang & Ying Huang, 2024. "Spotting potential reviewers for interdisciplinary research: insights on active reviewers from Publons," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(9), pages 5533-5556, September.
    13. Salandra, Rossella & Criscuolo, Paola & Salter, Ammon, 2021. "Directing scientists away from potentially biased publications: the role of systematic reviews in health care," Research Policy, Elsevier, vol. 50(1).
    14. Maciej J. Mrowinski & Agata Fronczak & Piotr Fronczak & Olgica Nedic & Aleksandar Dekanski, 2020. "The hurdles of academic publishing from the perspective of journal editors: a case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 115-133, October.
    15. Ana Teresa Santos & Sandro Mendonça, 2022. "Do papers (really) match journals’ “aims and scope”? A computational assessment of innovation studies," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7449-7470, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Michail Kovanis & Ludovic Trinquart & Philippe Ravaud & Raphaël Porcher, 2017. "Evaluating alternative systems of peer review: a large-scale agent-based modelling approach to scientific publication," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(1), pages 651-671, October.
    2. Michail Kovanis & Raphaël Porcher & Philippe Ravaud & Ludovic Trinquart, 2016. "Complex systems approach to scientific publication and peer-review system: development of an agent-based model calibrated with empirical journal data," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(2), pages 695-715, February.
    3. J. A. Garcia & Rosa Rodriguez-Sánchez & J. Fdez-Valdivia, 2021. "The interplay between the reviewer’s incentives and the journal’s quality standard," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 3041-3061, April.
    4. Vivian M Nguyen & Neal R Haddaway & Lee F G Gutowsky & Alexander D M Wilson & Austin J Gallagher & Michael R Donaldson & Neil Hammerschlag & Steven J Cooke, 2015. "How Long Is Too Long in Contemporary Peer Review? Perspectives from Authors Publishing in Conservation Biology Journals," PLOS ONE, Public Library of Science, vol. 10(8), pages 1-20, August.
    5. Bianchi, Federico & Grimaldo, Francisco & Squazzoni, Flaminio, 2019. "The F3-index. Valuing reviewers for scholarly journals," Journal of Informetrics, Elsevier, vol. 13(1), pages 78-86.
    6. Maciej J Mrowinski & Piotr Fronczak & Agata Fronczak & Marcel Ausloos & Olgica Nedic, 2017. "Artificial intelligence in peer review: How can evolutionary computation support journal editors?," PLOS ONE, Public Library of Science, vol. 12(9), pages 1-11, September.
    7. Maciej J. Mrowinski & Agata Fronczak & Piotr Fronczak & Olgica Nedic & Marcel Ausloos, 2016. "Review time in peer review: quantitative analysis and modelling of editorial workflows," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(1), pages 271-286, April.
    8. Rodríguez Sánchez, Isabel & Makkonen, Teemu & Williams, Allan M., 2019. "Peer review assessment of originality in tourism journals: critical perspective of key gatekeepers," Annals of Tourism Research, Elsevier, vol. 77(C), pages 1-11.
    9. Alessandro Checco & Lorenzo Bracciale & Pierpaolo Loreti & Stephen Pinfield & Giuseppe Bianchi, 2021. "AI-assisted peer review," Palgrave Communications, Palgrave Macmillan, vol. 8(1), pages 1-11, December.
    10. Kuklin, Alexander A. (Куклин, Александр) & Balyakina, Evgeniya A. (Балякина, Евгения), 2017. "Active policy as a key to success for an International Economic Periodical [Активная Политика — Залог Успеха Международного Экономического Журнала]," Ekonomicheskaya Politika / Economic Policy, Russian Presidential Academy of National Economy and Public Administration, vol. 6, pages 160-177, December.
    11. Dietmar Wolfram & Peiling Wang & Adam Hembree & Hyoungjoo Park, 2020. "Open peer review: promoting transparency in open science," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1033-1051, November.
    12. Yuetong Chen & Hao Wang & Baolong Zhang & Wei Zhang, 2022. "A method of measuring the article discriminative capacity and its distribution," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3317-3341, June.
    13. Paul Sebo & Jean Pascal Fournier & Claire Ragot & Pierre-Henri Gorioux & François R. Herrmann & Hubert Maisonneuve, 2019. "Factors associated with publication speed in general medical journals: a retrospective study of bibliometric data," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(2), pages 1037-1058, May.
    14. Qianjin Zong & Yafen Xie & Jiechun Liang, 2020. "Does open peer review improve citation count? Evidence from a propensity score matching analysis of PeerJ," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 607-623, October.
    15. Eirini Delikoura & Dimitrios Kouis, 2021. "Open Research Data and Open Peer Review: Perceptions of a Medical and Health Sciences Community in Greece," Publications, MDPI, vol. 9(2), pages 1-19, March.
    16. Narjes Vara & Mahdieh Mirzabeigi & Hajar Sotudeh & Seyed Mostafa Fakhrahmad, 2022. "Application of k-means clustering algorithm to improve effectiveness of the results recommended by journal recommender system," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3237-3252, June.
    17. Simone Righi & Károly Takács, 2017. "The miracle of peer review and development in science: an agent-based model," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(1), pages 587-607, October.
    18. Katarina Krapež, 2022. "Advancing Self-Evaluative and Self-Regulatory Mechanisms of Scholarly Journals: Editors’ Perspectives on What Needs to Be Improved in the Editorial Process," Publications, MDPI, vol. 10(1), pages 1-18, March.
    19. Pengfei Jia & Weixi Xie & Guangyao Zhang & Xianwen Wang, 2023. "Do reviewers get their deserved acknowledgments from the authors of manuscripts?," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(10), pages 5687-5703, October.
    20. Jens Peter Andersen & Serge P. J. M. Horbach & Tony Ross-Hellauer, 2024. "Through the secret gate: a study of member-contributed submissions in PNAS," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(9), pages 5673-5687, September.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0166387. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.