IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0221212.html
   My bibliography  Save this article

Citation gaming induced by bibliometric evaluation: A country-level comparative analysis

Author

Listed:
  • Alberto Baccini
  • Giuseppe De Nicolao
  • Eugenio Petrovich

Abstract

It is several years since national research evaluation systems around the globe started making use of quantitative indicators to measure the performance of researchers. Nevertheless, the effects on these systems on the behavior of the evaluated researchers are still largely unknown. For investigating this topic, we propose a new inwardness indicator able to gauge the degree of scientific self-referentiality of a country. Inwardness is defined as the proportion of citations coming from the country over the total number of citations gathered by the country. A comparative analysis of the trends for the G10 countries in the years 2000-2016 reveals a net increase of the Italian inwardness. Italy became, both globally and for a large majority of the research fields, the country with the highest inwardness and the lowest rate of international collaborations. The change in the Italian trend occurs in the years following the introduction in 2011 of national regulations in which key passages of professional careers are governed by bibliometric indicators. A most likely explanation of the peculiar Italian trend is a generalized strategic use of citations in the Italian scientific community, both in the form of strategic author self-citations and of citation clubs. We argue that the Italian case offers crucial insights on the constitutive effects of evaluation systems. As such, it could become a paradigmatic case in the debate about the use of indicators in science-policy contexts.

Suggested Citation

  • Alberto Baccini & Giuseppe De Nicolao & Eugenio Petrovich, 2019. "Citation gaming induced by bibliometric evaluation: A country-level comparative analysis," PLOS ONE, Public Library of Science, vol. 14(9), pages 1-16, September.
    • Handle: RePEc:plo:pone00:0221212
    DOI: 10.1371/journal.pone.0221212
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0221212
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0221212&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0221212?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. P. Pichappan & S. Sarasvady, 2002. "The other side of the coin: The intricacies of author self-citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 54(2), pages 285-290, June.
    2. Stefanie Haustein & Vincent Larivière, 2015. "The Use of Bibliometrics for Assessing Research: Possibilities, Limitations and Adverse Effects," Springer Books, in: Isabell M. Welpe & Jutta Wollersheim & Stefanie Ringelhan & Margit Osterloh (ed.), Incentives and Performance, edition 127, pages 121-139, Springer.
    3. Peter Ingwersen & Birger Larsen, 2014. "Influence of a performance indicator on Danish research production and citation impact 2000–12," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(2), pages 1325-1344, November.
    4. Hendrik P. van Dalen & Kène Henkens, 2012. "Intended and unintended consequences of a publish‐or‐perish culture: A worldwide survey," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 63(7), pages 1282-1293, July.
    5. Alberto Baccini & Lucio Barabesi, 2010. "Interlocking editorship. A network analysis of the links between economic journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 82(2), pages 365-389, February.
    6. Norman Kaplan, 1965. "The norms of citation behavior: Prolegomena to the footnote," American Documentation, Wiley Blackwell, vol. 16(3), pages 179-184, July.
    7. Frederic S. Lee & Xuan Pham & Gyun Gu, 2013. "The UK Research Assessment Exercise and the narrowing of UK economics," Cambridge Journal of Economics, Cambridge Political Economy Society, vol. 37(4), pages 693-717.
    8. Loet Leydesdorff, 2008. "Caveats for the use of citation indicators in research and journal evaluations," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(2), pages 278-287, January.
    9. Ruth Müller & Sarah de Rijcke, 2017. "Thinking with Indicators. Exploring the Epistemic Impacts of Academic Performance Indicators in the Life Sciences," Research Evaluation, Oxford University Press, vol. 26(4), pages 361-361.
    10. Linda Butler, 2003. "Modifying publication practices in response to funding formulas," Research Evaluation, Oxford University Press, vol. 12(1), pages 39-46, April.
    11. Laudel, Grit & Gläser, Jochen, 2014. "Beyond breakthrough research: Epistemic properties of research and their consequences for research funding," Research Policy, Elsevier, vol. 43(7), pages 1204-1216.
    12. Peter Dahler-Larsen, 2014. "Constitutive Effects of Performance Indicators: Getting beyond unintended consequences," Public Management Review, Taylor & Francis Journals, vol. 16(7), pages 969-986, October.
    13. Loet Leydesdorff, 2007. "Visualization of the citation impact environments of scientific journals: An online mapping exercise," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 58(1), pages 25-38, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ying Zhang & Cornelia Lawson & Liangping Ding, 2023. "Can scientists remain internationally visible after the return to their home country? A study of Chinese scientists," MIOIR Working Paper Series 2023-01, The Manchester Institute of Innovation Research (MIoIR), The University of Manchester.
    2. Nakajima, Kazuki & Liu, Ruodan & Shudo, Kazuyuki & Masuda, Naoki, 2023. "Quantifying gender imbalance in East Asian academia: Research career and citation practice," Journal of Informetrics, Elsevier, vol. 17(4).
    3. Alejandro Agafonow & Marybel Perez, 2024. "When an A Is NOT an A in Academic Research, or How A-Journal List Metrics Inhibit Exploratory Behaviour in Academia," Journal of Interdisciplinary Economics, , vol. 36(1), pages 105-121, January.
    4. Timur Gareev & Irina Peker, 2023. "Quantity versus quality in publication activity: knowledge production at the regional level," Papers 2311.08830, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Grilli, Leonardo, 2021. "The effects of citation-based research evaluation schemes on self-citation behavior," Journal of Informetrics, Elsevier, vol. 15(4).
    2. Julia Heuritsch, 2021. "Reflexive Behaviour: How Publication Pressure Affects Research Quality in Astronomy," Publications, MDPI, vol. 9(4), pages 1-23, November.
    3. Christian Schneijderberg & Nicolai Götze & Lars Müller, 2022. "A study of 25 years of publication outputs in the German academic profession," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(1), pages 1-28, January.
    4. Рубинштейн Александр Яковлевич, "undated". "Ранжирование Российских Экономических Журналов: Научный Метод Или «Игра В Цыфирь»? [Ran Ranking of Russian Economic Journals: The Scientific Method or “Numbers Game”?]," Working papers a:pru175:ye:2016:1, Institute of Economics.
    5. Soo Jeung Lee & Christian Schneijderberg & Yangson Kim & Isabel Steinhardt, 2021. "Have Academics’ Citation Patterns Changed in Response to the Rise of World University Rankings? A Test Using First-Citation Speeds," Sustainability, MDPI, vol. 13(17), pages 1-19, August.
    6. Julia Heuritsch, 2023. "The Evaluation Gap in Astronomy—Explained through a Rational Choice Framework," Publications, MDPI, vol. 11(2), pages 1-26, June.
    7. Giovanni Abramo & Ciriaco Andrea D’Angelo & Myroslava Hladchenko, 2023. "Assessing the effects of publication requirements for professorship on research performance and publishing behaviour of Ukrainian academics," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4589-4609, August.
    8. Tony Ross-Hellauer & Thomas Klebel & Petr Knoth & Nancy Pontika, 2024. "Value dissonance in research(er) assessment: individual and perceived institutional priorities in review, promotion, and tenure," Science and Public Policy, Oxford University Press, vol. 51(3), pages 337-351.
    9. Rodrigo Costas & Thed N. Leeuwen & María Bordons, 2010. "Self-citations at the meso and individual levels: effects of different calculation methods," Scientometrics, Springer;Akadémiai Kiadó, vol. 82(3), pages 517-537, March.
    10. Aleskerov, F. & Badgaeva, D. & Pislyakov, V. & Sterligov, I. & Shvydun, S., 2016. "An Importance of Russian and International Economic Journals: a Network Approach," Journal of the New Economic Association, New Economic Association, vol. 30(2), pages 193-205.
    11. Johansson, Dan & Karlsson, Johan & Malm, Arvid, 2020. "Family business—A missing link in economics?," Journal of Family Business Strategy, Elsevier, vol. 11(1).
    12. Lucas Brunet & Ruth Müller, 2022. "Making the cut: How panel reviewers use evaluation devices to select applications at the European Research Council," Research Evaluation, Oxford University Press, vol. 31(4), pages 486-497.
    13. Martin Szomszor & David A. Pendlebury & Jonathan Adams, 2020. "How much is too much? The difference between research influence and self-citation excess," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(2), pages 1119-1147, May.
    14. Groen-Xu, Moqi & Bös, Gregor & Teixeira, Pedro A. & Voigt, Thomas & Knapp, Bernhard, 2023. "Short-term incentives of research evaluations: Evidence from the UK Research Excellence Framework," Research Policy, Elsevier, vol. 52(6).
    15. Vîiu, Gabriel-Alexandru, 2016. "A theoretical evaluation of Hirsch-type bibliometric indicators confronted with extreme self-citation," Journal of Informetrics, Elsevier, vol. 10(2), pages 552-566.
    16. Michaela Strinzel & Josh Brown & Wolfgang Kaltenbrunner & Sarah Rijcke & Michael Hill, 2021. "Ten ways to improve academic CVs for fairer research assessment," Palgrave Communications, Palgrave Macmillan, vol. 8(1), pages 1-4, December.
    17. Eduardo Kunzel Teixeira & Mirian Oliveira, 2018. "Editorial board interlocking in knowledge management and intellectual capital research field," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(3), pages 1853-1869, December.
    18. Stephan Puehringer & Johanna Rath & Teresa Griesebner, 2021. "The political economy of academic publishing: On the commodification of a public good," PLOS ONE, Public Library of Science, vol. 16(6), pages 1-21, June.
    19. Aleskerov, F. & Kazachinskaya, A. & Karabekyan, D. & Semina, A. & Yakuba, V., 2021. "Economic journals of Russia, their characteristics and network analysis," Journal of the New Economic Association, New Economic Association, vol. 50(2), pages 170-182.
    20. Ramón A. Feenstra & Emilio Delgado López-Cózar, 2022. "Philosophers’ appraisals of bibliometric indicators and their use in evaluation: from recognition to knee-jerk rejection," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(4), pages 2085-2103, April.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0221212. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.