IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0006022.html
   My bibliography  Save this article

A Principal Component Analysis of 39 Scientific Impact Measures

Author

Listed:
  • Johan Bollen
  • Herbert Van de Sompel
  • Aric Hagberg
  • Ryan Chute

Abstract

Background: The impact of scientific publications has traditionally been expressed in terms of citation counts. However, scientific activity has moved online over the past decade. To better capture scientific impact in the digital era, a variety of new impact measures has been proposed on the basis of social network analysis and usage log data. Here we investigate how these new measures relate to each other, and how accurately and completely they express scientific impact. Methodology: We performed a principal component analysis of the rankings produced by 39 existing and proposed measures of scholarly impact that were calculated on the basis of both citation and usage log data. Conclusions: Our results indicate that the notion of scientific impact is a multi-dimensional construct that can not be adequately measured by any single indicator, although some measures are more suitable than others. The commonly used citation Impact Factor is not positioned at the core of this construct, but at its periphery, and should thus be used with caution.

Suggested Citation

  • Johan Bollen & Herbert Van de Sompel & Aric Hagberg & Ryan Chute, 2009. "A Principal Component Analysis of 39 Scientific Impact Measures," PLOS ONE, Public Library of Science, vol. 4(6), pages 1-11, June.
    • Handle: RePEc:plo:pone00:0006022
    DOI: 10.1371/journal.pone.0006022
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0006022
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0006022&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0006022?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Leo Egghe, 2006. "Theory and practise of the g-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 69(1), pages 131-152, October.
    2. Johan Bollen & Herbert Van de Sompel & Aric Hagberg & Luis Bettencourt & Ryan Chute & Marko A Rodriguez & Lyudmila Balakireva, 2009. "Clickstream Data Yields High-Resolution Maps of Science," PLOS ONE, Public Library of Science, vol. 4(3), pages 1-11, March.
    3. Li, Baibing & Martin, Elaine B. & Morris, A. Julian, 2002. "On principal component analysis in L1," Computational Statistics & Data Analysis, Elsevier, vol. 40(3), pages 471-474, September.
    4. Johan Bollen & Herbert Van de Sompel, 2008. "Usage impact factor: The effects of sample characteristics on usage‐based impact metrics," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(1), pages 136-149, January.
    5. Stephen P. Harter & Thomas E. Nisonger, 1997. "ISI's impact factor as misnomer: A proposed new measure to assess journal impact," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 48(12), pages 1146-1148, December.
    6. Chen, P. & Xie, H. & Maslov, S. & Redner, S., 2007. "Finding scientific gems with Google’s PageRank algorithm," Journal of Informetrics, Elsevier, vol. 1(1), pages 8-15.
    7. Ronald Rousseau, 2005. "Median and percentile impact factors: A set of new indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 63(3), pages 431-441, June.
    8. Tim Brody & Stevan Harnad & Leslie Carr, 2006. "Earlier Web usage statistics as predictors of later citation impact," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 57(8), pages 1060-1072, June.
    9. Loet Leydesdorff, 2007. "Betweenness centrality as an indicator of the interdisciplinarity of scientific journals," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 58(9), pages 1303-1319, July.
    10. Maria Bordons & M. T. Fernández & Isabel Gómez, 2002. "Advantages and limitations in the use of impact factor measures for the assessment of research performance," Scientometrics, Springer;Akadémiai Kiadó, vol. 53(2), pages 195-206, February.
    11. Loet Leydesdorff, 2007. "Visualization of the citation impact environments of scientific journals: An online mapping exercise," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 58(1), pages 25-38, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ana Teresa Santos & Sandro Mendonça, 2022. "Do papers (really) match journals’ “aims and scope”? A computational assessment of innovation studies," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7449-7470, December.
    2. Xipeng Liu & Xinmiao Li, 2024. "Unbiased evaluation of ranking algorithms applied to the Chinese green patents citation network," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(6), pages 2999-3021, June.
    3. Tingcan Ma & Gui-Fang Wang & Ke Dong & Mukun Cao, 2012. "The Journal’s Integrated Impact Index: a new indicator for journal evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 90(2), pages 649-658, February.
    4. Mike Thelwall, 2012. "Journal impact evaluation: a webometric perspective," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 429-441, August.
    5. Lorna Wildgaard & Jesper W. Schneider & Birger Larsen, 2014. "A review of the characteristics of 108 author-level bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 125-158, October.
    6. Cameron Neylon & Shirley Wu, 2009. "Article-Level Metrics and the Evolution of Scientific Impact," PLOS Biology, Public Library of Science, vol. 7(11), pages 1-6, November.
    7. Damien Besancenot & Abdelghani Maddi, 2019. "Should citations be weighted to assess the influence of an academic article?," Economics Bulletin, AccessEcon, vol. 435(1), pages 435-445.
    8. Fuli Zhang, 2017. "Evaluating journal impact based on weighted citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(2), pages 1155-1169, November.
    9. Bar-Ilan, Judit, 2008. "Informetrics at the beginning of the 21st century—A review," Journal of Informetrics, Elsevier, vol. 2(1), pages 1-52.
    10. Bai, Xiaomei & Zhang, Fuli & Liu, Jiaying & Xia, Feng, 2023. "Quantifying the impact of scientific collaboration and papers via motif-based heterogeneous networks," Journal of Informetrics, Elsevier, vol. 17(2).
    11. Eleni Fragkiadaki & Georgios Evangelidis, 2016. "Three novel indirect indicators for the assessment of papers and authors based on generations of citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(2), pages 657-694, February.
    12. Christian Schlögl & Juan Gorraiz & Christian Gumpenberger & Kris Jack & Peter Kraker, 2014. "Comparison of downloads, citations and readership data for two information systems journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(2), pages 1113-1128, November.
    13. Dinesh Pradhan & Partha Sarathi Paul & Umesh Maheswari & Subrata Nandi & Tanmoy Chakraborty, 2017. "$$C^3$$ C 3 -index: a PageRank based multi-faceted metric for authors’ performance measurement," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(1), pages 253-273, January.
    14. Christian Schloegl & Juan Gorraiz, 2010. "Comparison of citation and usage indicators: the case of oncology journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 82(3), pages 567-580, March.
    15. Tehmina Amjad & Ying Ding & Ali Daud & Jian Xu & Vincent Malic, 2015. "Topic-based heterogeneous rank," Scientometrics, Springer;Akadémiai Kiadó, vol. 104(1), pages 313-334, July.
    16. Koon-Kiu Yan & Mark Gerstein, 2011. "The Spread of Scientific Information: Insights from the Web Usage Statistics in PLoS Article-Level Metrics," PLOS ONE, Public Library of Science, vol. 6(5), pages 1-7, May.
    17. Hao Wang & Hua-Wei Shen & Xue-Qi Cheng, 2016. "Scientific credit diffusion: Researcher level or paper level?," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(2), pages 827-837, November.
    18. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    19. Liwen Vaughan & Juan Tang & Rongbin Yang, 2017. "Investigating disciplinary differences in the relationships between citations and downloads," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(3), pages 1533-1545, June.
    20. Ismael Rafols & Alan Porter & Loet Leydesdorff, 2009. "Overlay Maps of Science: a New Tool for Research Policy," SPRU Working Paper Series 179, SPRU - Science Policy Research Unit, University of Sussex Business School.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0006022. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.