IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v4y2010i4p554-563.html
   My bibliography  Save this article

Ranking of library and information science researchers: Comparison of data sources for correlating citation data, and expert judgments

Author

Listed:
  • Li, Jiang
  • Sanderson, Mark
  • Willett, Peter
  • Norris, Michael
  • Oppenheim, Charles

Abstract

This paper studies the correlations between peer review and citation indicators when evaluating research quality in library and information science (LIS). Forty-two LIS experts provided judgments on a 5-point scale of the quality of research published by 101 scholars; the median rankings resulting from these judgments were then correlated with h-, g- and H-index values computed using three different sources of citation data: Web of Science (WoS), Scopus and Google Scholar (GS). The two variants of the basic h-index correlated more strongly with peer judgment than did the h-index itself; citation data from Scopus was more strongly correlated with the expert judgments than was data from GS, which in turn was more strongly correlated than data from WoS; correlations from a carefully cleaned version of GS data were little different from those obtained using swiftly gathered GS data; the indices from the citation databases resulted in broadly similar rankings of the LIS academics; GS disadvantaged researchers in bibliometrics compared to the other two citation database while WoS disadvantaged researchers in the more technical aspects of information retrieval; and experts from the UK and other European countries rated UK academics with higher scores than did experts from the USA.

Suggested Citation

  • Li, Jiang & Sanderson, Mark & Willett, Peter & Norris, Michael & Oppenheim, Charles, 2010. "Ranking of library and information science researchers: Comparison of data sources for correlating citation data, and expert judgments," Journal of Informetrics, Elsevier, vol. 4(4), pages 554-563.
    • Handle: RePEc:eee:infome:v:4:y:2010:i:4:p:554-563
    DOI: 10.1016/j.joi.2010.06.005
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157710000593
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2010.06.005?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Norris, Michael & Oppenheim, Charles, 2010. "Peer review and the h-index: Two studies," Journal of Informetrics, Elsevier, vol. 4(3), pages 221-232.
    2. Leo Egghe, 2006. "Theory and practise of the g-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 69(1), pages 131-152, October.
    3. Charles Oppenheim, 2007. "Using the h‐index to rank influential British researchers in information science and librarianship," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 58(2), pages 297-301, January.
    4. Gunther Maier, 2006. "Impact factors and peer judgment: The case of regional science journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 69(3), pages 651-667, December.
    5. Norris, Michael & Oppenheim, Charles, 2007. "Comparing alternatives to the Web of Science for coverage of the social sciences’ literature," Journal of Informetrics, Elsevier, vol. 1(2), pages 161-169.
    6. Michael Schreiber, 2010. "Revisiting the g-index: The average number of citations in the g-core," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(1), pages 169-174, January.
    7. Rinia, E. J. & van Leeuwen, Th. N. & van Vuren, H. G. & van Raan, A. F. J., 1998. "Comparative analysis of a set of bibliometric indicators and central peer review criteria: Evaluation of condensed matter physics in the Netherlands," Research Policy, Elsevier, vol. 27(1), pages 95-107, May.
    8. Lokman I. Meho & Kiduk Yang, 2007. "Impact of data sources on citation counts and rankings of LIS faculty: Web of science versus scopus and google scholar," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 58(13), pages 2105-2125, November.
    9. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Caprasecca, Alessandro, 2009. "Allocative efficiency in public research funding: Can bibliometrics help?," Research Policy, Elsevier, vol. 38(1), pages 206-215, February.
    10. Michael S. Patterson & Simon Harris, 2009. "The relationship between reviewers’ quality-scores and number of citations for papers published in the journal Physics in Medicine and Biology from 2003–2005," Scientometrics, Springer;Akadémiai Kiadó, vol. 80(2), pages 343-349, August.
    11. Anthony F. J. Raan, 2006. "Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 67(3), pages 491-502, June.
    12. Alonso, S. & Cabrerizo, F.J. & Herrera-Viedma, E. & Herrera, F., 2009. "h-Index: A review focused in its variants, computation and standardization for different scientific fields," Journal of Informetrics, Elsevier, vol. 3(4), pages 273-289.
    13. Dag W Aksnes & Randi Elisabeth Taxt, 2004. "Peer reviews and bibliometric indicators: a comparative study at a Norwegian university," Research Evaluation, Oxford University Press, vol. 13(1), pages 33-41, April.
    14. Judit Bar-Ilan, 2008. "Which h-index? — A comparison of WoS, Scopus and Google Scholar," Scientometrics, Springer;Akadémiai Kiadó, vol. 74(2), pages 257-271, February.
    15. Mark Sanderson, 2008. "Revisiting h measured on UK LIS and IR academics," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(7), pages 1184-1190, May.
    16. Lutz Bornmann & Hans-Dieter Daniel, 2006. "Selecting scientific excellence through committee peer review - A citation analysis of publications previously published to approval or rejection of post-doctoral research fellowship applicants," Scientometrics, Springer;Akadémiai Kiadó, vol. 68(3), pages 427-440, September.
    17. Emanuela Reale & Anna Barbara & Antonio Costantini, 2007. "Peer review for the evaluation of academic research: lessons from the Italian experience," Research Evaluation, Oxford University Press, vol. 16(3), pages 216-228, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. William H. Walters & Esther Isabelle Wilder, 2015. "Worldwide contributors to the literature of library and information science: top authors, 2007–2012," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(1), pages 301-327, April.
    2. Giovanni Abramo & Ciriaco D’Angelo, 2015. "An assessment of the first “scientific habilitation” for university appointments in Italy," Economia Politica: Journal of Analytical and Institutional Economics, Springer;Fondazione Edison, vol. 32(3), pages 329-357, December.
    3. H. Kent Baker & Satish Kumar & Nitesh Pandey, 2020. "A bibliometric analysis of European Financial Managementʼs first 25 years," European Financial Management, European Financial Management Association, vol. 26(5), pages 1224-1260, November.
    4. Pin Li & Guoli Yang & Chuanqi Wang, 2019. "Visual topical analysis of library and information science," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(3), pages 1753-1791, December.
    5. Jacques Wainer & Paula Vieira, 2013. "Correlations between bibliometrics and peer evaluation for all disciplines: the evaluation of Brazilian scientists," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(2), pages 395-410, August.
    6. Shen, Hongquan & Xie, Juan & Ao, Weiyi & Cheng, Ying, 2022. "The continuity and citation impact of scientific collaboration with different gender composition," Journal of Informetrics, Elsevier, vol. 16(1).
    7. Jiang Li & Yueting Li, 2015. "Patterns and evolution of coauthorship in China’s humanities and social sciences," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(3), pages 1997-2010, March.
    8. Satish Kumar & Filomena Maggino & Raj V. Mahto & Riya Sureka & Leonardo Salvatore Alaimo & Weng Marc Lim, 2022. "Social Indicators Research: A Retrospective Using Bibliometric Analysis," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 162(1), pages 413-448, July.
    9. Juan Xie & Kaile Gong & Jiang Li & Qing Ke & Hyonchol Kang & Ying Cheng, 2019. "A probe into 66 factors which are possibly associated with the number of citations an article received," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(3), pages 1429-1454, June.
    10. Vanclay, Jerome K., 2012. "Publication patterns of award-winning forest scientists and implications for the Australian ERA journal ranking," Journal of Informetrics, Elsevier, vol. 6(1), pages 19-26.
    11. Satish Kumar & Weng Marc Lim & Nitesh Pandey & J. Christopher Westland, 2021. "20 years of Electronic Commerce Research," Electronic Commerce Research, Springer, vol. 21(1), pages 1-40, March.
    12. Wen-Yau Cathy Lin, 2012. "Research status and characteristics of library and information science in Taiwan: a bibliometric analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(1), pages 7-21, July.
    13. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jacques Wainer & Paula Vieira, 2013. "Correlations between bibliometrics and peer evaluation for all disciplines: the evaluation of Brazilian scientists," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(2), pages 395-410, August.
    2. Filippo Radicchi & Claudio Castellano, 2013. "Analysis of bibliometric indicators for individual scholars in a large data set," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(3), pages 627-637, December.
    3. Franceschet, Massimo & Costantini, Antonio, 2011. "The first Italian research assessment exercise: A bibliometric perspective," Journal of Informetrics, Elsevier, vol. 5(2), pages 275-291.
    4. Vieira, Elizabeth S. & Cabral, José A.S. & Gomes, José A.N.F., 2014. "How good is a model based on bibliometric indicators in predicting the final decisions made by peers?," Journal of Informetrics, Elsevier, vol. 8(2), pages 390-405.
    5. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    6. Zhang, Lin & Thijs, Bart & Glänzel, Wolfgang, 2011. "The diffusion of H-related literature," Journal of Informetrics, Elsevier, vol. 5(4), pages 583-593.
    7. Lorna Wildgaard, 2015. "A comparison of 17 author-level bibliometric indicators for researchers in Astronomy, Environmental Science, Philosophy and Public Health in Web of Science and Google Scholar," Scientometrics, Springer;Akadémiai Kiadó, vol. 104(3), pages 873-906, September.
    8. Takanori Ida & Naomi Fukuzawa, 2013. "Effects of large-scale research funding programs: a Japanese case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(3), pages 1253-1273, March.
    9. Fiorenzo Franceschini & Domenico Maisano, 2011. "Bibliometric positioning of scientific manufacturing journals: a comparative analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 86(2), pages 463-485, February.
    10. Bar-Ilan, Judit, 2008. "Informetrics at the beginning of the 21st century—A review," Journal of Informetrics, Elsevier, vol. 2(1), pages 1-52.
    11. Mingers, John & Leydesdorff, Loet, 2015. "A review of theory and practice in scientometrics," European Journal of Operational Research, Elsevier, vol. 246(1), pages 1-19.
    12. Maor Weinberger & Maayan Zhitomirsky-Geffet, 2021. "Diversity of success: measuring the scholarly performance diversity of tenured professors in the Israeli academia," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 2931-2970, April.
    13. Giovanni Abramo & Ciriaco Andrea D’Angelo & Emanuela Reale, 2019. "Peer review versus bibliometrics: Which method better predicts the scholarly impact of publications?," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 537-554, October.
    14. Bornmann, Lutz & Mutz, Rüdiger & Hug, Sven E. & Daniel, Hans-Dieter, 2011. "A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants," Journal of Informetrics, Elsevier, vol. 5(3), pages 346-359.
    15. Ehsan Mohammadi & Mike Thelwall, 2013. "Assessing non-standard article impact using F1000 labels," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(2), pages 383-395, November.
    16. Primož Južnič & Stojan Pečlin & Matjaž Žaucer & Tilen Mandelj & Miro Pušnik & Franci Demšar, 2010. "Scientometric indicators: peer-review, bibliometric methods and conflict of interests," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(2), pages 429-441, November.
    17. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    18. Massimo Franceschet, 2010. "A comparison of bibliometric indicators for computer science scholars and journals on Web of Science and Google Scholar," Scientometrics, Springer;Akadémiai Kiadó, vol. 83(1), pages 243-258, April.
    19. Gaby Haddow & Paul Genoni, 2010. "Citation analysis and peer ranking of Australian social science journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(2), pages 471-487, November.
    20. Rosenstreich, Daniela & Wooliscroft, Ben, 2009. "Measuring the impact of accounting journals using Google Scholar and the g-index," The British Accounting Review, Elsevier, vol. 41(4), pages 227-239.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:4:y:2010:i:4:p:554-563. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.