IDEAS home Printed from https://ideas.repec.org/a/eee/respol/v51y2022i10s0048733322001317.html
   My bibliography  Save this article

Who games metrics and rankings? Institutional niches and journal impact factor inflation

Author

Listed:
  • Siler, Kyle
  • Larivière, Vincent

Abstract

Ratings and rankings are omnipresent and influential in contemporary society. Individuals and organizations strategically respond to incentives set by rating systems. We use academic publishing as a case study to examine organizational variation in responses to influential metrics. The Journal Impact Factor (JIF) is a prominent metric linked to the value of academic journals, as well as career prospects of researchers. Since scholars, institutions, and publishers alike all have strong interests in affiliating with high JIF journals, strategic behaviors to ‘game’ the JIF metric are prevalent. Strategic self-citation is a common tactic employed to inflate JIF values. Based on empirical analyses of academic journals indexed in the Web of Science, we examine institutional characteristics conducive to strategic self-citation for JIF inflation. Journals disseminated by for-profit publishers, with lower JIFs, published in academically peripheral countries and with more recent founding dates were more likely to exhibit JIF-inflating self-citation patterns. Findings reveal the importance of status and institutional logics in influencing metrics gaming behaviors, as well as how metrics can affect work outcomes in different types of institutions. While quantitative rating systems affect many who are being evaluated, certain types of people and organizations are more prone to being influenced by rating systems than others.

Suggested Citation

  • Siler, Kyle & Larivière, Vincent, 2022. "Who games metrics and rankings? Institutional niches and journal impact factor inflation," Research Policy, Elsevier, vol. 51(10).
    • Handle: RePEc:eee:respol:v:51:y:2022:i:10:s0048733322001317
    DOI: 10.1016/j.respol.2022.104608
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0048733322001317
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.respol.2022.104608?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Petersen, Alexander M., 2019. "Megajournal mismanagement: Manuscript decision bias and anomalous editor activity at PLOS ONE," Journal of Informetrics, Elsevier, vol. 13(4).
    2. Hall, Jeremy & Martin, Ben R., 2019. "Towards a taxonomy of research misconduct: The case of business school research," Research Policy, Elsevier, vol. 48(2), pages 414-427.
    3. repec:hal:spmain:info:hdl:2441/6cbt691h0h8o9q5rf0apko0pda is not listed on IDEAS
    4. Wilhite, Allen & Fong, Eric A. & Wilhite, Seth, 2019. "The influence of editorial decisions and the academic network on self-citations and journal impact factors," Research Policy, Elsevier, vol. 48(6), pages 1513-1522.
    5. Dina Mayzlin & Yaniv Dover & Judith Chevalier, 2014. "Promotional Reviews: An Empirical Investigation of Online Review Manipulation," American Economic Review, American Economic Association, vol. 104(8), pages 2421-2455, August.
    6. Theodore C. Bergstrom, 2001. "Free Labour for Costly Journals?," Journal of Economic Perspectives, American Economic Association, vol. 15(4), pages 183-198, Fall.
    7. Li Tang & Guangyuan Hu, 2018. "Evaluation woes: Metrics can help beat bias," Nature, Nature, vol. 559(7714), pages 331-331, July.
    8. Wang, Jian & Veugelers, Reinhilde & Stephan, Paula, 2017. "Bias against novelty in science: A cautionary tale for users of bibliometric indicators," Research Policy, Elsevier, vol. 46(8), pages 1416-1436.
    9. Seeber, Marco & Cattaneo, Mattia & Meoli, Michele & Malighetti, Paolo, 2019. "Self-citations as strategic response to the use of metrics for career decisions," Research Policy, Elsevier, vol. 48(2), pages 478-491.
    10. Shubhanshu Mishra & Brent D Fegley & Jana Diesner & Vetle I Torvik, 2018. "Self-citation is the hallmark of productive authors, of any gender," PLOS ONE, Public Library of Science, vol. 13(9), pages 1-21, September.
    11. Benjamin M. Althouse & Jevin D. West & Carl T. Bergstrom & Theodore Bergstrom, 2009. "Differences in impact factor across fields and over time," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(1), pages 27-34, January.
    12. Vincent Larivière & Stefanie Haustein & Philippe Mongeon, 2015. "The Oligopoly of Academic Publishers in the Digital Era," PLOS ONE, Public Library of Science, vol. 10(6), pages 1-15, June.
    13. Matthew S. Bothner & Joel M. Podolny & Edward Bishop Smith, 2011. "Organizing Contests for Status: The Matthew Effect vs. the Mark Effect," Management Science, INFORMS, vol. 57(3), pages 439-457, March.
    14. Vincent Larivière & Yves Gingras, 2010. "The impact factor's Matthew Effect: A natural experiment in bibliometrics," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(2), pages 424-427, February.
    15. Mario Biagioli, 2016. "Watch out for cheats in citation game," Nature, Nature, vol. 535(7611), pages 201-201, July.
    16. Jingfeng Xia & Jennifer L. Harmon & Kevin G. Connolly & Ryan M. Donnelly & Mary R. Anderson & Heather A. Howard, 2015. "Who publishes in “predatory” journals?," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(7), pages 1406-1417, July.
    17. Kyle Siler, 2020. "Demarcating spectrums of predatory publishing: Economic and institutional sources of academic legitimacy," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 71(11), pages 1386-1401, November.
    18. David A. King, 2004. "The scientific impact of nations," Nature, Nature, vol. 430(6997), pages 311-316, July.
    19. Berggren, Christian & Karabag, Solmaz Filiz, 2019. "Scientific misconduct at an elite medical institute: The role of competing institutional logics and fragmented control," Research Policy, Elsevier, vol. 48(2), pages 428-443.
    20. Badi H. Baltagi, 2021. "Econometric Analysis of Panel Data," Springer Texts in Business and Economics, Springer, edition 6, number 978-3-030-53953-5, June.
    21. Chirikov, Igor, 2021. "Does Conflict of Interest Distort Global University Rankings?," University of California at Berkeley, Center for Studies in Higher Education qt8hk672nh, Center for Studies in Higher Education, UC Berkeley.
    22. Julien Jacqmin, 2021. "Do ads influence rankings? Evidence from the higher education sector," Education Economics, Taylor & Francis Journals, vol. 29(5), pages 509-526, September.
    23. Kyle Siler & Philippe Vincent-Lamarre & Cassidy R. Sugimoto & Vincent Larivière, 2021. "Predatory publishers’ latest scam: bootlegged and rebranded papers," Nature, Nature, vol. 598(7882), pages 563-565, October.
    24. Martin Szomszor & David A. Pendlebury & Jonathan Adams, 2020. "How much is too much? The difference between research influence and self-citation excess," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(2), pages 1119-1147, May.
    25. Jeremy Hall & Ben R. Martin, 2019. "Towards a Taxonomy of Academic Misconduct: The Case of Business School Research," SPRU Working Paper Series 2019-02, SPRU - Science Policy Research Unit, University of Sussex Business School.
    26. Vincent Larivière & Yves Gingras, 2010. "The impact factor's Matthew Effect: A natural experiment in bibliometrics," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 61(2), pages 424-427, February.
    27. Amandine Ody‐Brasier & Amanda Sharkey, 2019. "Under pressure: Reputation, ratings, and inaccurate self‐reporting in the nursing home industry," Strategic Management Journal, Wiley Blackwell, vol. 40(10), pages 1517-1544, October.
    28. Ruben Miranda & Esther Garcia-Carpintero, 2019. "Comparison of the share of documents and citations from different quartile journals in 25 research areas," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 479-501, October.
    29. Diana Hicks & Paul Wouters & Ludo Waltman & Sarah de Rijcke & Ismael Rafols, 2015. "Bibliometrics: The Leiden Manifesto for research metrics," Nature, Nature, vol. 520(7548), pages 429-431, April.
    30. Howard D. White, 2001. "Authors as citers over time," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 52(2), pages 87-108.
    31. Charles F Hickman & Eric A Fong & Allen W Wilhite & Yeolan Lee, 2019. "Academic misconduct and criminal liability: Manipulating academic journal impact factors," Science and Public Policy, Oxford University Press, vol. 46(5), pages 661-667.
    32. Fourcade, Marion & Healy, Kieran, 2013. "Classification situations: Life-chances in the neoliberal era," Accounting, Organizations and Society, Elsevier, vol. 38(8), pages 559-572.
    33. Marion Fourcade & Kieran Healy, 2013. "Classification situations: Life-chances in the neoliberal era," Post-Print hal-03470535, HAL.
    34. Henry Small, 2004. "On the shoulders of Robert Merton: Towards a normative theory of citation," Scientometrics, Springer;Akadémiai Kiadó, vol. 60(1), pages 71-79, May.
    35. Campbell, Donald T., 1979. "Assessing the impact of planned social change," Evaluation and Program Planning, Elsevier, vol. 2(1), pages 67-90, January.
    36. Michael Luca & Georgios Zervas, 2016. "Fake It Till You Make It: Reputation, Competition, and Yelp Review Fraud," Management Science, INFORMS, vol. 62(12), pages 3412-3427, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Kosyakov, Denis & Pislyakov, Vladimir, 2024. "“I'd like to publish in Q1, but there's no Q1 to be found”: Study of journal quartile distributions across subject categories and topics," Journal of Informetrics, Elsevier, vol. 18(1).
    2. Hunter Bennett & Ben Singh & Flynn Slattery, 2024. "Journal self-citations trends in sport sciences: an analysis of disciplinary journals from 2013 to 2022," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(8), pages 4741-4750, August.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Yangping Zhou, 2021. "Self-citation and citation of top journal publishers and their interpretation in the journal-discipline context," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 6013-6040, July.
    2. Lu Liu & Benjamin F. Jones & Brian Uzzi & Dashun Wang, 2023. "Data, measurement and empirical methods in the science of science," Nature Human Behaviour, Nature, vol. 7(7), pages 1046-1058, July.
    3. Groen-Xu, Moqi & Bös, Gregor & Teixeira, Pedro A. & Voigt, Thomas & Knapp, Bernhard, 2023. "Short-term incentives of research evaluations: Evidence from the UK Research Excellence Framework," Research Policy, Elsevier, vol. 52(6).
    4. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Grilli, Leonardo, 2021. "The effects of citation-based research evaluation schemes on self-citation behavior," Journal of Informetrics, Elsevier, vol. 15(4).
    5. Osterloh, Margit & Frey, Bruno S., 2020. "How to avoid borrowed plumes in academia," Research Policy, Elsevier, vol. 49(1).
    6. Martin Szomszor & David A. Pendlebury & Jonathan Adams, 2020. "How much is too much? The difference between research influence and self-citation excess," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(2), pages 1119-1147, May.
    7. Hui Li & Weishu Liu, 2020. "Same same but different: self-citations identified through Scopus and Web of Science Core Collection," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(3), pages 2723-2732, September.
    8. Lanu Kim & Jason H. Portenoy & Jevin D. West & Katherine W. Stovel, 2020. "Scientific journals still matter in the era of academic search engines and preprint archives," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 71(10), pages 1218-1226, October.
    9. Apostolos Filippas & John J. Horton & Joseph M. Golden, 2022. "Reputation Inflation," Marketing Science, INFORMS, vol. 41(4), pages 733-745, July.
    10. Milojević, Staša & Radicchi, Filippo & Bar-Ilan, Judit, 2017. "Citation success index − An intuitive pair-wise journal comparison metric," Journal of Informetrics, Elsevier, vol. 11(1), pages 223-231.
    11. Tobias Kiesslich & Marlena Beyreis & Georg Zimmermann & Andreas Traweger, 2021. "Citation inequality and the Journal Impact Factor: median, mean, (does it) matter?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1249-1269, February.
    12. Bornmann, Lutz & Haunschild, Robin & Mutz, Rüdiger, 2020. "Should citations be field-normalized in evaluative bibliometrics? An empirical analysis based on propensity score matching," Journal of Informetrics, Elsevier, vol. 14(4).
    13. Glenna, Leland & Bruce, Analena, 2021. "Suborning science for profit: Monsanto, glyphosate, and private science research misconduct," Research Policy, Elsevier, vol. 50(7).
    14. Tang, Li & Yang, Defang & Wang, Mingxing & Guo, Ying, 2024. "The mediating impact of citation scope: Evidence from China's ESI publications," Journal of Informetrics, Elsevier, vol. 18(3).
    15. Abdelghani Maddi & Yves Gingras, 2021. "Gender Diversity In Research Teams And Citation Impact In Economics And Management," Journal of Economic Surveys, Wiley Blackwell, vol. 35(5), pages 1381-1404, December.
    16. Xiomara S. Q. Chacon & Thiago C. Silva & Diego R. Amancio, 2020. "Comparing the impact of subfields in scientific journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 625-639, October.
    17. Laura Bowering Mullen, 2024. "Open Access, Scholarly Communication, and Open Science in Psychology: An Overview for Researchers," SAGE Open, , vol. 14(1_suppl), pages 21582440231, April.
    18. Taşkın, Zehra & Doğan, Güleda & Kulczycki, Emanuel & Zuccala, Alesia Ann, 2021. "Self-Citation Patterns of Journals Indexed in the Journal Citation Reports," Journal of Informetrics, Elsevier, vol. 15(4).
    19. Marek Kwiek & Wojciech Roszka, 2022. "Are female scientists less inclined to publish alone? The gender solo research gap," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(4), pages 1697-1735, April.
    20. Bruns, Stephan B. & Asanov, Igor & Bode, Rasmus & Dunger, Melanie & Funk, Christoph & Hassan, Sherif M. & Hauschildt, Julia & Heinisch, Dominik & Kempa, Karol & König, Johannes & Lips, Johannes & Verb, 2019. "Reporting errors and biases in published empirical findings: Evidence from innovation research," Research Policy, Elsevier, vol. 48(9), pages 1-1.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:respol:v:51:y:2022:i:10:s0048733322001317. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/respol .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.