IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v122y2020i3d10.1007_s11192-019-03338-y.html
   My bibliography  Save this article

Should highly cited items be excluded in impact factor calculation? The effect of review articles on journal impact factor

Author

Listed:
  • Lei Lei

    (Huazhong University of Science and Technology)

  • Yunmei Sun

    (Huazhong University of Science and Technology)

Abstract

The use of the journal impact factor for evaluation purposes has been widely challenged because of the simplistic algorithm involved in impact factor calculation. As a result, a journal's impact factor may be affected by many variables, such as highly cited review articles. In the present study, we aimed to investigate whether and to what extent review articles affect the impact factors of journals. We examined the citations of original research articles and review articles in the field of physics, and compared the impact factors of the journals Science, Nature, and Cell with and without citation counts of review articles included. The results showed that review articles consistently received higher numbers of citations than did original research articles. In addition, review articles accounted for 3–10% of the impact factors of these top journals. Based on our findings, we suggest that sophisticated methods, in addition to the impact factor, be employed for research evaluation purposes.

Suggested Citation

  • Lei Lei & Yunmei Sun, 2020. "Should highly cited items be excluded in impact factor calculation? The effect of review articles on journal impact factor," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(3), pages 1697-1706, March.
    • Handle: RePEc:spr:scient:v:122:y:2020:i:3:d:10.1007_s11192-019-03338-y
    DOI: 10.1007/s11192-019-03338-y
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-019-03338-y
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-019-03338-y?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Erwin Krauskopf, 2013. "Deceiving the research community through manipulation of the impact factor," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(11), pages 2403-2403, November.
    2. Erwin Krauskopf, 2013. "Deceiving the research community through manipulation of the impact factor," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(11), pages 2403-2403, November.
    3. Paula Stephan & Reinhilde Veugelers & Jian Wang, 2017. "Reviewers are blinkered by bibliometrics," Nature, Nature, vol. 544(7651), pages 411-412, April.
    4. Dong-Hui Yang & Xin Li & Xiaoxia Sun & Jie Wan, 2016. "Detecting impact factor manipulation with data mining techniques," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1989-2005, December.
    5. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    6. Bornmann, Lutz & Marx, Werner, 2018. "Critical rationalism and the search for standard (field-normalized) indicators in bibliometrics," Journal of Informetrics, Elsevier, vol. 12(3), pages 598-604.
    7. Guang Yu & Dong-Hui Yang & Wang Liang, 2010. "Reliability-based citation impact factor and the manipulation of impact factor," Scientometrics, Springer;Akadémiai Kiadó, vol. 83(1), pages 259-270, April.
    8. Dongbo Shi & Ronald Rousseau & Liu Yang & Jiang Li, 2017. "A journal's impact factor is influenced by changes in publication delays of citing journals," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(3), pages 780-789, March.
    9. Lei Lei & Sheng Yan, 2016. "Readability and citations in information science: evidence from abstracts and articles of four journals (2003–2012)," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(3), pages 1155-1169, September.
    10. Diana Hicks & Paul Wouters & Ludo Waltman & Sarah de Rijcke & Ismael Rafols, 2015. "Bibliometrics: The Leiden Manifesto for research metrics," Nature, Nature, vol. 520(7548), pages 429-431, April.
    11. Weishu Liu & Guangyuan Hu & Mengdi Gu, 2016. "The probability of publishing in first-quartile journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(3), pages 1273-1276, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Tian-Yuan Huang & Liying Yang, 2022. "Superior identification index: Quantifying the capability of academic journals to recognize good research," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(7), pages 4023-4043, July.
    2. Thelwall, Mike & Kousha, Kayvan & Stuart, Emma & Makita, Meiko & Abdoli, Mahshid & Wilson, Paul & Levitt, Jonathan, 2023. "Do bibliometrics introduce gender, institutional or interdisciplinary biases into research evaluations?," Research Policy, Elsevier, vol. 52(8).
    3. Raminta Pranckutė, 2021. "Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World," Publications, MDPI, vol. 9(1), pages 1-59, March.
    4. Guofeng Wang & Rui Shi & Wei Cheng & Lihua Gao & Xiankai Huang, 2023. "Bibliometric Analysis for Carbon Neutrality with Hotspots, Frontiers, and Emerging Trends between 1991 and 2022," IJERPH, MDPI, vol. 20(2), pages 1-23, January.
    5. Bahuleyan, Athira & Chavan, Meena & Krzeminska, Anna & Chirico, Francesco, 2024. "Process and variance research: Integrating research on university spinoff evolution," Technovation, Elsevier, vol. 130(C).
    6. Yves Fassin, 2021. "The impact of review articles in management and economics journal rankings and metrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(12), pages 9623-9632, December.
    7. Linda C. Smith, 2024. "Reviews and Reviewing: Approaches to Research Synthesis. An Annual Review of Information Science and Technology (ARIST) paper," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 75(3), pages 245-267, March.
    8. Clemens Blümel & Alexander Schniedermann, 2020. "Studying review articles in scientometrics and beyond: a research agenda," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(1), pages 711-728, July.
    9. Salim Moussa, 2021. "Are FT50 journals really leading? A comment on Fassin," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(12), pages 9613-9622, December.
    10. Sepideh Fahimifar & Khadijeh Mousavi & Fatemeh Mozaffari & Marcel Ausloos, 2023. "Identification of the most important external features of highly cited scholarly papers through 3 (i.e., Ridge, Lasso, and Boruta) feature selection data mining methods," Quality & Quantity: International Journal of Methodology, Springer, vol. 57(4), pages 3685-3712, August.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lutz Bornmann & Julian N. Marewski, 2019. "Heuristics as conceptual lens for understanding and studying the usage of bibliometrics in research evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(2), pages 419-459, August.
    2. Raminta Pranckutė, 2021. "Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World," Publications, MDPI, vol. 9(1), pages 1-59, March.
    3. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Costa, 2023. "Correlating article citedness and journal impact: an empirical investigation by field on a large-scale dataset," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(3), pages 1877-1894, March.
    4. Lanu Kim & Jason H. Portenoy & Jevin D. West & Katherine W. Stovel, 2020. "Scientific journals still matter in the era of academic search engines and preprint archives," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 71(10), pages 1218-1226, October.
    5. Sergio Copiello, 2019. "Research Interest: another undisclosed (and redundant) algorithm by ResearchGate," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(1), pages 351-360, July.
    6. Milojević, Staša & Radicchi, Filippo & Bar-Ilan, Judit, 2017. "Citation success index − An intuitive pair-wise journal comparison metric," Journal of Informetrics, Elsevier, vol. 11(1), pages 223-231.
    7. Lutz Bornmann & Alexander Tekles & Loet Leydesdorff, 2019. "How well does I3 perform for impact measurement compared to other bibliometric indicators? The convergent validity of several (field-normalized) indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(2), pages 1187-1205, May.
    8. Xu, Shuqi & Mariani, Manuel Sebastian & Lü, Linyuan & Medo, Matúš, 2020. "Unbiased evaluation of ranking metrics reveals consistent performance in science and technology citation data," Journal of Informetrics, Elsevier, vol. 14(1).
    9. Bornmann, Lutz & Haunschild, Robin & Mutz, Rüdiger, 2020. "Should citations be field-normalized in evaluative bibliometrics? An empirical analysis based on propensity score matching," Journal of Informetrics, Elsevier, vol. 14(4).
    10. Aleskerov, F. & Badgaeva, D. & Pislyakov, V. & Sterligov, I. & Shvydun, S., 2016. "An Importance of Russian and International Economic Journals: a Network Approach," Journal of the New Economic Association, New Economic Association, vol. 30(2), pages 193-205.
    11. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    12. Alona Zharova & Wolfgang K. Härdle & Stefan Lessmann, 2017. "Is Scientific Performance a Function of Funds?," SFB 649 Discussion Papers SFB649DP2017-028, Sonderforschungsbereich 649, Humboldt University, Berlin, Germany.
    13. Antonoyiannakis, Manolis, 2018. "Impact Factors and the Central Limit Theorem: Why citation averages are scale dependent," Journal of Informetrics, Elsevier, vol. 12(4), pages 1072-1088.
    14. Zhaoping Yan & Kaiyu Fan, 2024. "An integrated indicator for evaluating scientific papers: considering academic impact and novelty," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(11), pages 6909-6929, November.
    15. Sebnem Cansun & Engin Arik, 2018. "Political science publications about Turkey," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(1), pages 169-188, April.
    16. Tang, Li & Yang, Defang & Wang, Mingxing & Guo, Ying, 2024. "The mediating impact of citation scope: Evidence from China's ESI publications," Journal of Informetrics, Elsevier, vol. 18(3).
    17. Mark Bukowski & Sandra Geisler & Thomas Schmitz-Rode & Robert Farkas, 2020. "Feasibility of activity-based expert profiling using text mining of scientific publications and patents," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(2), pages 579-620, May.
    18. Haunschild, Robin & Daniels, Angela D. & Bornmann, Lutz, 2022. "Scores of a specific field-normalized indicator calculated with different approaches of field-categorization: Are the scores different or similar?," Journal of Informetrics, Elsevier, vol. 16(1).
    19. Ruben Miranda & Esther Garcia-Carpintero, 2019. "Comparison of the share of documents and citations from different quartile journals in 25 research areas," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 479-501, October.
    20. Lutz Bornmann & Klaus Wohlrabe, 2019. "Normalisation of citation impact in economics," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(2), pages 841-884, August.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:122:y:2020:i:3:d:10.1007_s11192-019-03338-y. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.