IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v88y2011i3d10.1007_s11192-011-0425-7.html
   My bibliography  Save this article

On the correlation between bibliometric indicators and peer review: reply to Opthof and Leydesdorff

Author

Listed:
  • Ludo Waltman

    (Leiden University)

  • Nees Jan Eck

    (Leiden University)

  • Thed N. Leeuwen

    (Leiden University)

  • Martijn S. Visser

    (Leiden University)

  • Anthony F. J. Raan

    (Leiden University)

Abstract

Opthof and Leydesdorff (Scientometrics, 2011) reanalyze data reported by Van Raan (Scientometrics 67(3):491–502, 2006) and conclude that there is no significant correlation between on the one hand average citation scores measured using the CPP/FCSm indicator and on the other hand the quality judgment of peers. We point out that Opthof and Leydesdorff draw their conclusions based on a very limited amount of data. We also criticize the statistical methodology used by Opthof and Leydesdorff. Using a larger amount of data and a more appropriate statistical methodology, we do find a significant correlation between the CPP/FCSm indicator and peer judgment.

Suggested Citation

  • Ludo Waltman & Nees Jan Eck & Thed N. Leeuwen & Martijn S. Visser & Anthony F. J. Raan, 2011. "On the correlation between bibliometric indicators and peer review: reply to Opthof and Leydesdorff," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(3), pages 1017-1022, September.
    • Handle: RePEc:spr:scient:v:88:y:2011:i:3:d:10.1007_s11192-011-0425-7
    DOI: 10.1007/s11192-011-0425-7
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-011-0425-7
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-011-0425-7?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Rinia, E. J. & van Leeuwen, Th. N. & van Vuren, H. G. & van Raan, A. F. J., 1998. "Comparative analysis of a set of bibliometric indicators and central peer review criteria: Evaluation of condensed matter physics in the Netherlands," Research Policy, Elsevier, vol. 27(1), pages 95-107, May.
    2. Tobias Opthof & Loet Leydesdorff, 2011. "A comment to the paper by Waltman et al., Scientometrics, 87, 467–481, 2011," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(3), pages 1011-1016, September.
    3. Anthony F. J. Raan, 2006. "Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 67(3), pages 491-502, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Schneider, Jesper W., 2013. "Caveats for using statistical significance tests in research assessments," Journal of Informetrics, Elsevier, vol. 7(1), pages 50-62.
    2. Daisuke Sakai, 2019. "Who is peer reviewed? Comparing publication patterns of peer-reviewed and non-peer-reviewed papers in Japanese political science," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 65-80, October.
    3. Mariana-Daniela González-Zamar & Emilio Abad-Segura, 2021. "Emotional Creativity in Art Education: An Exploratory Analysis and Research Trends," IJERPH, MDPI, vol. 18(12), pages 1-20, June.
    4. Ehsan Mohammadi & Mike Thelwall, 2013. "Assessing non-standard article impact using F1000 labels," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(2), pages 383-395, November.
    5. Gangan Prathap, 2012. "A comment to the papers by Opthof and Leydesdorff, Scientometrics, 88, 1011–1016, 2011 and Waltman et al., Scientometrics, 88, 1017–1022, 2011," Scientometrics, Springer;Akadémiai Kiadó, vol. 90(2), pages 737-743, February.
    6. Jacques Wainer & Paula Vieira, 2013. "Correlations between bibliometrics and peer evaluation for all disciplines: the evaluation of Brazilian scientists," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(2), pages 395-410, August.
    7. Daniela Filippo & Rafael Aleixandre-Benavent & Elías Sanz-Casado, 2020. "Toward a classification of Spanish scholarly journals in social sciences and humanities considering their impact and visibility," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1709-1732, November.
    8. Adriana Bin & Sergio Salles-Filho & Luiza Maria Capanema & Fernando Antonio Basile Colugnati, 2015. "What difference does it make? Impact of peer-reviewed scholarships on scientific production," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(2), pages 1167-1188, February.
    9. Bouyssou, Denis & Marchant, Thierry, 2016. "Ranking authors using fractional counting of citations: An axiomatic approach," Journal of Informetrics, Elsevier, vol. 10(1), pages 183-199.
    10. José Luis Gallego Ortega & Antonio Rodríguez Fuentes & Antonio García Guzmán, 2021. "Application of Mathematical Methods to the Study of Special-Needs Education in Spanish Journals," Mathematics, MDPI, vol. 9(6), pages 1-17, March.
    11. Mingliang Yue & Hongbo Tang & Fan Liu & Tingcan Ma, 2021. "Consistency index: measuring the performances of scholar journal reviewers," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(8), pages 7183-7195, August.
    12. Gangan Prathap, 2012. "Energy indicators and percentile ranking normalization," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(3), pages 997-1003, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Takanori Ida & Naomi Fukuzawa, 2013. "Effects of large-scale research funding programs: a Japanese case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(3), pages 1253-1273, March.
    2. Gangan Prathap, 2012. "Energy indicators and percentile ranking normalization," Scientometrics, Springer;Akadémiai Kiadó, vol. 91(3), pages 997-1003, June.
    3. Alberto Baccini & Giuseppe De Nicolao, 2016. "Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(3), pages 1651-1671, September.
    4. Muzammil Tahira & Rose Alinda Alias & Aryati Bakri, 2013. "Scientometric assessment of engineering in Malaysians universities," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(3), pages 865-879, September.
    5. Sigifredo Laengle & José M. Merigó & Nikunja Mohan Modak & Jian-Bo Yang, 2020. "Bibliometrics in operations research and management science: a university analysis," Annals of Operations Research, Springer, vol. 294(1), pages 769-813, November.
    6. Jacques Wainer & Paula Vieira, 2013. "Correlations between bibliometrics and peer evaluation for all disciplines: the evaluation of Brazilian scientists," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(2), pages 395-410, August.
    7. Giovanni Abramo & Ciriaco Andrea D’Angelo & Emanuela Reale, 2019. "Peer review versus bibliometrics: Which method better predicts the scholarly impact of publications?," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 537-554, October.
    8. Daisuke Sakai, 2019. "Who is peer reviewed? Comparing publication patterns of peer-reviewed and non-peer-reviewed papers in Japanese political science," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 65-80, October.
    9. Ehsan Mohammadi & Mike Thelwall, 2013. "Assessing non-standard article impact using F1000 labels," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(2), pages 383-395, November.
    10. Abramo, Giovanni, 2018. "Revisiting the scientometric conceptualization of impact and its measurement," Journal of Informetrics, Elsevier, vol. 12(3), pages 590-597.
    11. Primož Južnič & Stojan Pečlin & Matjaž Žaucer & Tilen Mandelj & Miro Pušnik & Franci Demšar, 2010. "Scientometric indicators: peer-review, bibliometric methods and conflict of interests," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(2), pages 429-441, November.
    12. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    13. Wu, Jiang, 2013. "Investigating the universal distributions of normalized indicators and developing field-independent index," Journal of Informetrics, Elsevier, vol. 7(1), pages 63-71.
    14. Giovanni Abramo & Tindaro Cicero & Ciriaco Andrea D’Angelo, 2013. "National peer-review research assessment exercises for the hard sciences can be a complete waste of money: the Italian case," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(1), pages 311-324, April.
    15. Franceschet, Massimo & Costantini, Antonio, 2011. "The first Italian research assessment exercise: A bibliometric perspective," Journal of Informetrics, Elsevier, vol. 5(2), pages 275-291.
    16. Vieira, Elizabeth S. & Cabral, José A.S. & Gomes, José A.N.F., 2014. "How good is a model based on bibliometric indicators in predicting the final decisions made by peers?," Journal of Informetrics, Elsevier, vol. 8(2), pages 390-405.
    17. Fedderke, J.W. & Goldschmidt, M., 2015. "Does massive funding support of researchers work?: Evaluating the impact of the South African research chair funding initiative," Research Policy, Elsevier, vol. 44(2), pages 467-482.
    18. Gangan Prathap, 2012. "A comment to the papers by Opthof and Leydesdorff, Scientometrics, 88, 1011–1016, 2011 and Waltman et al., Scientometrics, 88, 1017–1022, 2011," Scientometrics, Springer;Akadémiai Kiadó, vol. 90(2), pages 737-743, February.
    19. Li, Jiang & Sanderson, Mark & Willett, Peter & Norris, Michael & Oppenheim, Charles, 2010. "Ranking of library and information science researchers: Comparison of data sources for correlating citation data, and expert judgments," Journal of Informetrics, Elsevier, vol. 4(4), pages 554-563.
    20. Giovanni Abramo & Ciriaco Andrea D'Angelo, 2015. "The VQR, Italy's second national research assessment: Methodological failures and ranking distortions," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(11), pages 2202-2214, November.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:88:y:2011:i:3:d:10.1007_s11192-011-0425-7. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.