IDEAS home Printed from https://ideas.repec.org/p/ehl/lserod/117286.html
   My bibliography  Save this paper

Defining, measuring, and rewarding scholarly impact: mind the level of analysis

Author

Listed:
  • Ramani, Ravi S.
  • Aguinis, Herman
  • Coyle-Shapiro, Jacqueline A.M.

Abstract

We address the grossly incorrect inferences that result from using journal impact factor (JIF) as a proxy to assess individual researcher and article scholarly impact. This invalid practice occurs because of confusion about the definition and measurement of impact at different levels of analysis. Specifically, JIF is a journal-level measure of impact, computed by aggregating citations of individual articles (i.e., upward effect), and is therefore inappropriate when measuring impact at lower levels of analysis, such as that of individual researchers, or of individual articles published in a particular journal (i.e., downward effect). We illustrate the severity of the errors that occur when using JIF to evaluate individual scholarly impact, and advocate for an immediate moratorium on the exclusive use of JIF and other journal-level (i.e., higher level of analysis) measures when assessing the impact of individual researchers and individual articles (i.e., lower level of analysis). Given the importance and interest in assessing the scholarly impact of researchers and articles, we delineate level-appropriate and readily available measures.

Suggested Citation

  • Ramani, Ravi S. & Aguinis, Herman & Coyle-Shapiro, Jacqueline A.M., 2022. "Defining, measuring, and rewarding scholarly impact: mind the level of analysis," LSE Research Online Documents on Economics 117286, London School of Economics and Political Science, LSE Library.
    • Handle: RePEc:ehl:lserod:117286
    as

    Download full text from publisher

    File URL: http://eprints.lse.ac.uk/117286/
    File Function: Open access version.
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Klein, Julie Thompson & Falk-Krzesinski, Holly J., 2017. "Interdisciplinary and collaborative work: Framing promotion and tenure practices and policies," Research Policy, Elsevier, vol. 46(6), pages 1055-1061.
    2. Ashkan Ebadi & Andrea Schiffauerova, 2015. "How to Receive More Funding for Your Research? Get Connected to the Right People!," PLOS ONE, Public Library of Science, vol. 10(7), pages 1-19, July.
    3. Fei Shu & Wei Quan & Bikun Chen & Junping Qiu & Cassidy R. Sugimoto & Vincent Larivière, 2020. "The role of Web of Science publications in China’s tenure system," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(3), pages 1683-1695, March.
    4. Millar, Morgan M., 2013. "Interdisciplinary research and the early career: The effect of interdisciplinary dissertation research on career placement and publication productivity of doctoral graduates in the sciences," Research Policy, Elsevier, vol. 42(5), pages 1152-1164.
    5. Akella, Akhil Pandey & Alhoori, Hamed & Kondamudi, Pavan Ravikanth & Freeman, Cole & Zhou, Haiming, 2021. "Early indicators of scientific impact: Predicting citations with altmetrics," Journal of Informetrics, Elsevier, vol. 15(2).
    6. Anne-Wil Harzing & Satu Alakangas, 2016. "Google Scholar, Scopus and the Web of Science: a longitudinal and cross-disciplinary comparison," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(2), pages 787-804, February.
    7. Cassidy R. Sugimoto & Sam Work & Vincent Larivière & Stefanie Haustein, 2017. "Scholarly use of social media and altmetrics: A review of the literature," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(9), pages 2037-2062, September.
    8. Melinda Harm Benson & Christopher D. Lippitt & Ryan Morrison & Barbara Cosens & Jan Boll & Brian C. Chaffin & Alexander K. Fremier & Robert Heinse & Derek Kauneckis & Timothy E. Link & Caroline E. Scr, 2016. "Five ways to support interdisciplinary work before tenure," Journal of Environmental Studies and Sciences, Springer;Association of Environmental Studies and Sciences, vol. 6(2), pages 260-267, June.
    9. Isabelle Régner & Catherine Thinus-Blanc & Agnès Netter & Toni Schmader & Pascal Huguet, 2019. "Committees with implicit biases promote fewer women when they do not believe gender bias exists," Nature Human Behaviour, Nature, vol. 3(11), pages 1171-1179, November.
    10. Chris Woolston, 2021. "Impact factor abandoned by Dutch university in hiring and promotion decisions," Nature, Nature, vol. 595(7867), pages 462-462, July.
    11. Kathleen L. McGinn & Katherine L. Milkman, 2013. "Looking Up and Looking Out: Career Mobility Effects of Demographic Similarity Among Professionals," Organization Science, INFORMS, vol. 24(4), pages 1041-1060, August.
    12. Antonis Sidiropoulos & Dimitrios Katsaros & Yannis Manolopoulos, 2007. "Generalized Hirsch h-index for disclosing latent facts in citation networks," Scientometrics, Springer;Akadémiai Kiadó, vol. 72(2), pages 253-280, August.
    13. José Luis Ortega, 2020. "Blogs and news sources coverage in altmetrics data providers: a comparative analysis by country, language, and subject," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(1), pages 555-572, January.
    14. Ryazanova, Olga & McNamara, Peter & Aguinis, Herman, 2017. "Research performance as a quality signal in international labor markets: Visibility of business schools worldwide through a global research performance system," Journal of World Business, Elsevier, vol. 52(6), pages 831-841.
    15. Jessica L. Cundiff & Cinnamon L. Danube & Matthew J. Zawadzki & Stephanie A. Shields, 2018. "Testing an Intervention for Recognizing and Reporting Subtle Gender Bias in Promotion and Tenure Decisions," The Journal of Higher Education, Taylor & Francis Journals, vol. 89(5), pages 611-636, September.
    16. Powdthavee, Nattavudh & Riyanto, Yohanes E. & Knetsch, Jack L., 2018. "Lower-rated publications do lower academics’ judgments of publication lists: Evidence from a survey experiment of economists," Journal of Economic Psychology, Elsevier, vol. 66(C), pages 33-44.
    17. Chun-Ting Zhang, 2009. "The e-Index, Complementing the h-Index for Excess Citations," PLOS ONE, Public Library of Science, vol. 4(5), pages 1-4, May.
    18. Robin Haunschild & Lutz Bornmann & Jonathan Adams, 2019. "R package for producing beamplots as a preferred alternative to the h index when assessing single researchers (based on downloads from Web of Science)," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(2), pages 925-927, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Kickul, Jill & Griffiths, Mark & Brännback, Malin & Robb, Colleen C., 2023. "Don't always judge an article by its cover: An examination of proxies for journal impact and citations in entrepreneurship," Journal of Business Venturing Insights, Elsevier, vol. 20(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wildgaard, Lorna, 2016. "A critical cluster analysis of 44 indicators of author-level performance," Journal of Informetrics, Elsevier, vol. 10(4), pages 1055-1078.
    2. Arnold, Austin & Cafer, Anne & Green, John & Haines, Seena & Mann, Georgianna & Rosenthal, Meagen, 2021. "“Perspective: Promoting and fostering multidisciplinary research in universities”," Research Policy, Elsevier, vol. 50(9).
    3. Vîiu, Gabriel-Alexandru, 2016. "A theoretical evaluation of Hirsch-type bibliometric indicators confronted with extreme self-citation," Journal of Informetrics, Elsevier, vol. 10(2), pages 552-566.
    4. Zhenbin Yan & Qiang Wu & Xingchen Li, 2016. "Do Hirsch-type indices behave the same in assessing single publications? An empirical study of 29 bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1815-1833, December.
    5. Perc, Matjaž, 2010. "Zipf’s law and log-normal distributions in measures of scientific output across fields and institutions: 40 years of Slovenia’s research as an example," Journal of Informetrics, Elsevier, vol. 4(3), pages 358-364.
    6. James C. Ryan, 2016. "A validation of the individual annual h-index (hIa): application of the hIa to a qualitatively and quantitatively different sample," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(1), pages 577-590, October.
    7. Yu Liu & Wei Zuo & Ying Gao & Yanhong Qiao, 2013. "Comprehensive geometrical interpretation of h-type indices," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(2), pages 605-615, August.
    8. Lorna Wildgaard & Jesper W. Schneider & Birger Larsen, 2014. "A review of the characteristics of 108 author-level bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 125-158, October.
    9. Shahzad, Murtuza & Alhoori, Hamed & Freedman, Reva & Rahman, Shaikh Abdul, 2022. "Quantifying the online long-term interest in research," Journal of Informetrics, Elsevier, vol. 16(2).
    10. Amodio, Pierluigi & Brugnano, Luigi & Scarselli, Filippo, 2021. "Implementation of the PaperRank and AuthorRank indices in the Scopus database," Journal of Informetrics, Elsevier, vol. 15(4).
    11. Antonia Gogoglou & Antonis Sidiropoulos & Dimitrios Katsaros & Yannis Manolopoulos, 2017. "The fractal dimension of a citation curve: quantifying an individual’s scientific output using the geometry of the entire curve," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(3), pages 1751-1774, June.
    12. Madiha Ameer & Muhammad Tanvir Afzal, 2019. "Evaluation of h-index and its qualitative and quantitative variants in Neuroscience," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(2), pages 653-673, November.
    13. Camil Demetrescu & Irene Finocchi & Andrea Ribichini & Marco Schaerf, 2020. "On bibliometrics in academic promotions: a case study in computer science and engineering in Italy," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(3), pages 2207-2228, September.
    14. Muhammad Usman & Ghulam Mustafa & Muhammad Tanvir Afzal, 2021. "Ranking of author assessment parameters using Logistic Regression," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(1), pages 335-353, January.
    15. Vitus Püttmann & Jens Ruhose & Stephan L. Thomsen, 2023. "Academics’ Attitudes Toward Engaging in Public Discussions: Experimental Evidence on the Impact of Engagement Conditions," Research in Higher Education, Springer;Association for Institutional Research, vol. 64(5), pages 765-788, August.
    16. Brandão, Luana Carneiro & Soares de Mello, João Carlos Correia Baptista, 2019. "A multi-criteria approach to the h-index," European Journal of Operational Research, Elsevier, vol. 276(1), pages 357-363.
    17. Sidiropoulos, A. & Gogoglou, A. & Katsaros, D. & Manolopoulos, Y., 2016. "Gazing at the skyline for star scientists," Journal of Informetrics, Elsevier, vol. 10(3), pages 789-813.
    18. Abdul Rahman Shaikh & Hamed Alhoori & Maoyuan Sun, 2023. "YouTube and science: models for research impact," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(2), pages 933-955, February.
    19. Qurat-ul Ain & Hira Riaz & Muhammad Tanvir Afzal, 2019. "Evaluation of h-index and its citation intensity based variants in the field of mathematics," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(1), pages 187-211, April.
    20. Zhang, Lin & Thijs, Bart & Glänzel, Wolfgang, 2011. "The diffusion of H-related literature," Journal of Informetrics, Elsevier, vol. 5(4), pages 583-593.

    More about this item

    JEL classification:

    • J50 - Labor and Demographic Economics - - Labor-Management Relations, Trade Unions, and Collective Bargaining - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ehl:lserod:117286. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: LSERO Manager (email available below). General contact details of provider: https://edirc.repec.org/data/lsepsuk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.