IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0068258.html
   My bibliography  Save this article

Reviewers’ Ratings and Bibliometric Indicators: Hand in Hand When Assessing Over Research Proposals?

Author

Listed:
  • Álvaro Cabezas-Clavijo
  • Nicolás Robinson-García
  • Manuel Escabias
  • Evaristo Jiménez-Contreras

Abstract

Background: The peer review system has been traditionally challenged due to its many limitations especially for allocating funding. Bibliometric indicators may well present themselves as a complement. Objective: We analyze the relationship between peers’ ratings and bibliometric indicators for Spanish researchers in the 2007 National R&D Plan for 23 research fields. Methods and Materials: We analyze peers’ ratings for 2333 applications. We also gathered principal investigators’ research output and impact and studied the differences between accepted and rejected applications. We used the Web of Science database and focused on the 2002-2006 period. First, we analyzed the distribution of granted and rejected proposals considering a given set of bibliometric indicators to test if there are significant differences. Then, we applied a multiple logistic regression analysis to determine if bibliometric indicators can explain by themselves the concession of grant proposals. Results: 63.4% of the applications were funded. Bibliometric indicators for accepted proposals showed a better previous performance than for those rejected; however the correlation between peer review and bibliometric indicators is very heterogeneous among most areas. The logistic regression analysis showed that the main bibliometric indicators that explain the granting of research proposals in most cases are the output (number of published articles) and the number of papers published in journals that belong to the first quartile ranking of the Journal Citations Report. Discussion: Bibliometric indicators predict the concession of grant proposals at least as well as peer ratings. Social Sciences and Education are the only areas where no relation was found, although this may be due to the limitations of the Web of Science’s coverage. These findings encourage the use of bibliometric indicators as a complement to peer review in most of the analyzed areas.

Suggested Citation

  • Álvaro Cabezas-Clavijo & Nicolás Robinson-García & Manuel Escabias & Evaristo Jiménez-Contreras, 2013. "Reviewers’ Ratings and Bibliometric Indicators: Hand in Hand When Assessing Over Research Proposals?," PLOS ONE, Public Library of Science, vol. 8(6), pages 1-1, June.
    • Handle: RePEc:plo:pone00:0068258
    DOI: 10.1371/journal.pone.0068258
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0068258
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0068258&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0068258?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Rodrigo Costas & María Bordons, 2005. "Bibliometric indicators at the micro-level: some results in the area of natural resources at the Spanish CSIC," Research Evaluation, Oxford University Press, vol. 14(2), pages 110-120, August.
    2. Hendy Abdoul & Christophe Perrey & Philippe Amiel & Florence Tubach & Serge Gottot & Isabelle Durand-Zaleski & Corinne Alberti, 2012. "Peer Review of Grant Applications: Criteria Used and Qualitative Study of Reviewer Practices," PLOS ONE, Public Library of Science, vol. 7(9), pages 1-15, September.
    3. Liz Allen & Ceri Jones & Kevin Dolby & David Lynn & Mark Walport, 2009. "Looking for Landmarks: The Role of Expert Review and Bibliometric Analysis in Evaluating Scientific Publication Outputs," PLOS ONE, Public Library of Science, vol. 4(6), pages 1-8, June.
    4. Henk F. Moed, 2008. "UK Research Assessment Exercises: Informed judgments on research quality or quantity?," Scientometrics, Springer;Akadémiai Kiadó, vol. 74(1), pages 153-161, January.
    5. Per O. Seglen, 1992. "The skewness of science," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 43(9), pages 628-638, October.
    6. Thed N. van Leeuwen & Henk F. Moed, 2012. "Funding decisions, peer review, and scientific excellence in physical sciences, chemistry, and geosciences," Research Evaluation, Oxford University Press, vol. 21(3), pages 189-198, August.
    7. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Di Costa, 2011. "National research assessment exercises: a comparison of peer review and bibliometrics rankings," Scientometrics, Springer;Akadémiai Kiadó, vol. 89(3), pages 929-941, December.
    8. Juan Miguel Campanario, 1996. "Have referees rejected some of the most‐cited articles of all times?," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 47(4), pages 302-310, April.
    9. Vincent Larivière & Benoit Macaluso & Éric Archambault & Yves Gingras, 2010. "Which scientific elites? On the concentration of research funds, publications and citations," Research Evaluation, Oxford University Press, vol. 19(1), pages 45-53, March.
    10. Lutz Bornmann & Hans-Dieter Daniel, 2005. "Selection of research fellowship recipients by committee peer review. Reliability, fairness and predictive validity of Board of Trustees' decisions," Scientometrics, Springer;Akadémiai Kiadó, vol. 63(2), pages 297-320, April.
    11. Alonso, S. & Cabrerizo, F.J. & Herrera-Viedma, E. & Herrera, F., 2009. "h-Index: A review focused in its variants, computation and standardization for different scientific fields," Journal of Informetrics, Elsevier, vol. 3(4), pages 273-289.
    12. Dag W Aksnes & Randi Elisabeth Taxt, 2004. "Peer reviews and bibliometric indicators: a comparative study at a Norwegian university," Research Evaluation, Oxford University Press, vol. 13(1), pages 33-41, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Christopher W Belter, 2014. "Measuring the Value of Research Data: A Citation Analysis of Oceanographic Data Sets," PLOS ONE, Public Library of Science, vol. 9(3), pages 1-9, March.
    2. Mingers, John & Leydesdorff, Loet, 2015. "A review of theory and practice in scientometrics," European Journal of Operational Research, Elsevier, vol. 246(1), pages 1-19.
    3. Franceschet, Massimo & Costantini, Antonio, 2011. "The first Italian research assessment exercise: A bibliometric perspective," Journal of Informetrics, Elsevier, vol. 5(2), pages 275-291.
    4. Juan A Crespo & Ignacio Ortuño-Ortín & Javier Ruiz-Castillo, 2012. "The Citation Merit of Scientific Publications," PLOS ONE, Public Library of Science, vol. 7(11), pages 1-9, November.
    5. Huang, Ding-wei, 2018. "Optimal distribution of science funding," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 502(C), pages 613-618.
    6. Huang, Ding-wei, 2021. "A basic model for empirical funding distributions," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 566(C).
    7. J. A. García & Rosa Rodriguez-Sánchez & J. Fdez-Valdivia & Nicolas Robinson-García & Daniel Torres-Salinas, 2013. "Benchmarking research performance at the university level with information theoretic measures," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(1), pages 435-452, April.
    8. Lorna Wildgaard & Jesper W. Schneider & Birger Larsen, 2014. "A review of the characteristics of 108 author-level bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 125-158, October.
    9. Mingers, John & Yang, Liying, 2017. "Evaluating journal quality: A review of journal citation indicators and ranking in business and management," European Journal of Operational Research, Elsevier, vol. 257(1), pages 323-337.
    10. Farideh Osare & Mariam Keshvari, 2024. "Evaluation of a Scientific Productivity Model among World Highly Cited Authors: a Study Based on Experts’ Opinions," Journal of the Knowledge Economy, Springer;Portland International Center for Management of Engineering and Technology (PICMET), vol. 15(3), pages 14452-14485, September.
    11. Bar-Ilan, Judit, 2008. "Informetrics at the beginning of the 21st century—A review," Journal of Informetrics, Elsevier, vol. 2(1), pages 1-52.
    12. Robert A. Buckle & John Creedy, 2022. "Methods to evaluate institutional responses to performance‐based research funding systems," Australian Economic Papers, Wiley Blackwell, vol. 61(3), pages 615-634, September.
    13. Jacques Wainer & Paula Vieira, 2013. "Correlations between bibliometrics and peer evaluation for all disciplines: the evaluation of Brazilian scientists," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(2), pages 395-410, August.
    14. Giovanni Abramo & Ciriaco Andrea D’Angelo & Emanuela Reale, 2019. "Peer review versus bibliometrics: Which method better predicts the scholarly impact of publications?," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 537-554, October.
    15. Kevin W. Boyack & Caleb Smith & Richard Klavans, 2018. "Toward predicting research proposal success," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(2), pages 449-461, February.
    16. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    17. Albarrán, Pedro & Ortuño, Ignacio & Ruiz-Castillo, Javier, 2011. "The measurement of low- and high-impact in citation distributions: Technical results," Journal of Informetrics, Elsevier, vol. 5(1), pages 48-63.
    18. Abramo, Giovanni, 2018. "Revisiting the scientometric conceptualization of impact and its measurement," Journal of Informetrics, Elsevier, vol. 12(3), pages 590-597.
    19. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    20. Juan Gorraiz & Christian Gumpenberger, 2015. "A flexible bibliometric approach for the assessment of professorial appointments," Scientometrics, Springer;Akadémiai Kiadó, vol. 105(3), pages 1699-1719, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0068258. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.