IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0011344.html
   My bibliography  Save this article

The Usefulness of Peer Review for Selecting Manuscripts for Publication: A Utility Analysis Taking as an Example a High-Impact Journal

Author

Listed:
  • Lutz Bornmann
  • Hans-Dieter Daniel

Abstract

Background: High predictive validity – that is, a strong association between the outcome of peer review (usually, reviewers' ratings) and the scientific quality of a manuscript submitted to a journal (measured as citations of the later published paper) – does not as a rule suffice to demonstrate the usefulness of peer review for the selection of manuscripts. To assess usefulness, it is important to include in addition the base rate (proportion of submissions that are fundamentally suitable for publication) and the selection rate (the proportion of submissions accepted). Methodology/Principal Findings: Taking the example of the high-impact journal Angewandte Chemie International Edition (AC-IE), we present a general approach for determining the usefulness of peer reviews for the selection of manuscripts for publication. The results of our study show that peer review is useful: 78% of the submissions accepted by AC-IE are correctly accepted for publication when the editor's decision is based on one review, 69% of the submissions are correctly accepted for publication when the editor's decision is based on two reviews, and 65% of the submissions are correctly accepted for publication when the editor's decision is based on three reviews. Conclusions/Significance: The paper points out through what changes in the selection rate, base rate or validity coefficient a higher success rate (utility) in the AC-IE selection process could be achieved.

Suggested Citation

  • Lutz Bornmann & Hans-Dieter Daniel, 2010. "The Usefulness of Peer Review for Selecting Manuscripts for Publication: A Utility Analysis Taking as an Example a High-Impact Journal," PLOS ONE, Public Library of Science, vol. 5(6), pages 1-9, June.
    • Handle: RePEc:plo:pone00:0011344
    DOI: 10.1371/journal.pone.0011344
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0011344
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0011344&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0011344?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Thed N van Leeuwen, 2007. "Modelling of bibliometric approaches and importance of output verification in research performance assessment," Research Evaluation, Oxford University Press, vol. 16(2), pages 93-105, June.
    2. Bornmann, Lutz & Daniel, Hans-Dieter, 2009. "Extent of type I and type II errors in editorial decisions: A case study on Angewandte Chemie International Edition," Journal of Informetrics, Elsevier, vol. 3(4), pages 348-352.
    3. Martin, Ben R. & Irvine, John, 1993. "Assessing basic research : Some partial indicators of scientific progress in radio astronomy," Research Policy, Elsevier, vol. 22(2), pages 106-106, April.
    4. Christoph Neuhaus & Hans-Dieter Daniel, 2009. "A new reference standard for citation analysis in chemistry and related fields based on the sections of Chemical Abstracts," Scientometrics, Springer;Akadémiai Kiadó, vol. 78(2), pages 219-229, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jeffrey L Jackson & Malathi Srinivasan & Joanna Rea & Kathlyn E Fletcher & Richard L Kravitz, 2011. "The Validity of Peer Review in a General Medicine Journal," PLOS ONE, Public Library of Science, vol. 6(7), pages 1-8, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bornmann, Lutz & Schier, Hermann & Marx, Werner & Daniel, Hans-Dieter, 2012. "What factors determine citation counts of publications in chemistry besides their quality?," Journal of Informetrics, Elsevier, vol. 6(1), pages 11-18.
    2. Maaike Verbree & Edwin Horlings & Peter Groenewegen & Inge Weijden & Peter Besselaar, 2015. "Organizational factors influencing scholarly performance: a multivariate study of biomedical research groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(1), pages 25-49, January.
    3. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    4. Thed N. Leeuwen & Erik Wijk & Paul F. Wouters, 2016. "Bibliometric analysis of output and impact based on CRIS data: a case study on the registered output of a Dutch university," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(1), pages 1-16, January.
    5. Christoph Neuhaus & Hans-Dieter Daniel, 2009. "A new reference standard for citation analysis in chemistry and related fields based on the sections of Chemical Abstracts," Scientometrics, Springer;Akadémiai Kiadó, vol. 78(2), pages 219-229, February.
    6. Selçuk Bilir & Ersin Göğüş & Özgecan Önal & Nazlı Derya Öztürkmen & Talar Yontan, 2013. "Research performance of Turkish astronomers in the period of 1980–2010," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(2), pages 477-489, November.
    7. Henk F. Moed, 2000. "Bibliometric Indicators Reflect Publication and Management Strategies," Scientometrics, Springer;Akadémiai Kiadó, vol. 47(2), pages 323-346, February.
    8. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Di Costa, Flavia, 2021. "The scholarly impact of private sector research: A multivariate analysis," Journal of Informetrics, Elsevier, vol. 15(3).
    9. Rafols, Ismael & Leydesdorff, Loet & O’Hare, Alice & Nightingale, Paul & Stirling, Andy, 2012. "How journal rankings can suppress interdisciplinary research: A comparison between Innovation Studies and Business & Management," Research Policy, Elsevier, vol. 41(7), pages 1262-1282.
    10. T. J. Phelan, 1999. "A compendium of issues for citation analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 45(1), pages 117-136, May.
    11. Giovanni Abramo & Ciriaco Andrea D'Angelo & Flavia Di Costa, 2020. "The relative impact of private research on scientific advancement," Papers 2012.04908, arXiv.org.
    12. Wolfgang Polt & Manfred Paier & Andreas Schibany & Helmut Gassler & Gernot Hutschenreiter & Norbert Knoll & Hannes Leo & Michael Peneder, 1999. "Österreichischer Technologiebericht 1999," WIFO Studies, WIFO, number 8332.
    13. Aksnes, Dag W. & Rip, Arie, 2009. "Researchers' perceptions of citations," Research Policy, Elsevier, vol. 38(6), pages 895-905, July.
    14. Perianes-Rodriguez, Antonio & Ruiz-Castillo, Javier, 2017. "A comparison of the Web of Science and publication-level classification systems of science," Journal of Informetrics, Elsevier, vol. 11(1), pages 32-45.
    15. Bornmann, Lutz & Leydesdorff, Loet, 2013. "The validation of (advanced) bibliometric indicators through peer assessments: A comparative study using data from InCites and F1000," Journal of Informetrics, Elsevier, vol. 7(2), pages 286-291.
    16. van Raan, A. F. J. & van Leeuwen, Th. N., 2002. "Assessment of the scientific basis of interdisciplinary, applied research: Application of bibliometric methods in Nutrition and Food Research," Research Policy, Elsevier, vol. 31(4), pages 611-632, May.
    17. Thed Leeuwen & Rodrigo Costas & Clara Calero-Medina & Martijn Visser, 2013. "The role of editorial material in bibliometric research performance assessments," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(2), pages 817-828, May.
    18. Lutz Bornmann & Werner Marx, 2014. "How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 487-509, January.
    19. Wildgaard, Lorna, 2016. "A critical cluster analysis of 44 indicators of author-level performance," Journal of Informetrics, Elsevier, vol. 10(4), pages 1055-1078.
    20. Sabrina Petersohn & Thomas Heinze, 2018. "Professionalization of bibliometric research assessment. Insights from the history of the Leiden Centre for Science and Technology Studies (CWTS)," Science and Public Policy, Oxford University Press, vol. 45(4), pages 565-578.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0011344. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.