IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v127y2022i7d10.1007_s11192-022-04421-7.html
   My bibliography  Save this article

Measuring the influence of non-scientific features on citations

Author

Listed:
  • Stefano Mammola

    (University of Helsinki
    National Research Council of Italy (CNR))

  • Elena Piano

    (University of Turin)

  • Alberto Doretto

    (University of Piemonte Orientale)

  • Enrico Caprio

    (University of Turin)

  • Dan Chamberlain

    (University of Turin)

Abstract

Citations remain a prime, yet controversial, measure of academic performance. Ideally, how often a paper is cited should solely depend on the quality of the science reported therein. However, non-scientific factors, including structural elements (e.g., length of abstract, number of references) or attributes of authors (e.g., prestige and gender), may all influence citation outcomes. Knowing the predicted effect of these features on citations might make it possible to ‘game the system’ of citation counts when writing a paper. We conducted a meta-analysis to build a quantitative understanding of the effect of similar non-scientific features on the impact of scientific articles in terms of citations. We showed that article length, number of authors, author experience and their collaboration network, Impact Factors, availability as open access, online sharing, different referencing practice, and number of figures all exerted a positive influence on citations. These patterns were consistent across most disciplines. We also documented temporal trends towards a recent increase in the effect of journal Impact Factor and number of authors on citations. We suggest that our approach can be used as a benchmark to monitor the influence of these effects over time, minimising the influence of non-scientific features as a means to game the system of citation counts, and thus enhancing their usefulness as a measure of scientific quality.

Suggested Citation

  • Stefano Mammola & Elena Piano & Alberto Doretto & Enrico Caprio & Dan Chamberlain, 2022. "Measuring the influence of non-scientific features on citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(7), pages 4123-4137, July.
    • Handle: RePEc:spr:scient:v:127:y:2022:i:7:d:10.1007_s11192-022-04421-7
    DOI: 10.1007/s11192-022-04421-7
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-022-04421-7
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-022-04421-7?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Benjamin Freeling & Zoë A. Doubleday & Sean D. Connell, 2019. "Opinion: How can we boost the impact of publications? Try better writing," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 116(2), pages 341-343, January.
    2. Letchford, Adrian & Preis, Tobias & Moat, Helen Susannah, 2016. "The advantage of simple paper abstracts," Journal of Informetrics, Elsevier, vol. 10(1), pages 1-8.
    3. Diana Kwon, 2022. "The rise of citational justice: how scholars are making references fairer," Nature, Nature, vol. 603(7902), pages 568-571, March.
    4. Juan Xie & Kaile Gong & Ying Cheng & Qing Ke, 2019. "The correlation between paper length and citations: a meta-analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(3), pages 763-786, March.
    5. Maite Barrios & Anna Villarroya & Ángel Borrego, 2013. "Scientific production in psychology: a gender analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(1), pages 15-23, April.
    6. Shubhanshu Mishra & Brent D Fegley & Jana Diesner & Vetle I Torvik, 2018. "Self-citation is the hallmark of productive authors, of any gender," PLOS ONE, Public Library of Science, vol. 13(9), pages 1-21, September.
    7. Viechtbauer, Wolfgang, 2010. "Conducting Meta-Analyses in R with the metafor Package," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 36(i03).
    8. Peter A. Todd & Richard J. Ladle, 2008. "Citations: poor practices by authors reduce their value," Nature, Nature, vol. 451(7176), pages 244-244, January.
    9. Abramo, Giovanni & Aksnes, Dag W. & D’Angelo, Ciriaco Andrea, 2021. "Gender differences in research performance within and between countries: Italy vs Norway," Journal of Informetrics, Elsevier, vol. 15(2).
    10. Bedoor AlShebli & Kinga Makovi & Talal Rahwan, 2020. "Retraction Note: The association between early career informal mentorship in academic collaborations and junior author performance," Nature Communications, Nature, vol. 11(1), pages 1-1, December.
    11. Clément Bosquet & Pierre-Philippe Combes, 2013. "Are academics who publish more also more cited? Individual determinants of publication and citation records," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(3), pages 831-857, December.
    12. James H. Fowler & Dag W. Aksnes, 2007. "Does self-citation pay?," Scientometrics, Springer;Akadémiai Kiadó, vol. 72(3), pages 427-437, September.
    13. Hongquan Shen & Juan Xie & Jiang Li & Ying Cheng, 2021. "The correlation between scientific collaboration and citation count at the paper level: a meta-analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 3443-3470, April.
    14. David Moher & Alessandro Liberati & Jennifer Tetzlaff & Douglas G Altman & The PRISMA Group, 2009. "Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement," PLOS Medicine, Public Library of Science, vol. 6(7), pages 1-6, July.
    15. Iman Tahamtan & Lutz Bornmann, 2019. "What do citation counts measure? An updated review of studies on citations in scientific documents published between 2006 and 2018," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(3), pages 1635-1684, December.
    16. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    17. Iman Tahamtan & Askar Safipour Afshar & Khadijeh Ahamdzadeh, 2016. "Factors affecting number of citations: a comprehensive review of the literature," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(3), pages 1195-1225, June.
    18. Sarah W Davies & Hollie M Putnam & Tracy Ainsworth & Julia K Baum & Colleen B Bove & Sarah C Crosby & Isabelle M Côté & Anne Duplouy & Robinson W Fulweiler & Alyssa J Griffin & Torrance C Hanley & Tes, 2021. "Promoting inclusive metrics of success and impact to dismantle a discriminatory reward system in science," PLOS Biology, Public Library of Science, vol. 19(6), pages 1-15, June.
    19. Philip Ball, 2008. "A longer paper gathers more citations," Nature, Nature, vol. 455(7211), pages 274-274, September.
    20. Stefano Mammola & Diego Fontaneto & Alejandro Martínez & Filipe Chichorro, 2021. "Impact of the reference list features on the number of citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(1), pages 785-799, January.
    21. Clément Bosquet & Pierre-Philippe Combes, 2013. "Are academics who publish more also more cited? Individual determinants of publication and citation records," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(3), pages 831-857, December.
    22. Bernhard Knapp & Rémi Bardenet & Miguel O Bernabeu & Rafel Bordas & Maria Bruna & Ben Calderhead & Jonathan Cooper & Alexander G Fletcher & Derek Groen & Bram Kuijper & Joanna Lewis & Greg McInerny & , 2015. "Ten Simple Rules for a Successful Cross-Disciplinary Collaboration," PLOS Computational Biology, Public Library of Science, vol. 11(4), pages 1-7, April.
    23. Wolfgang Glänzel & Bart Thijs, 2004. "The influence of author self-citations on bibliometric macro indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 59(3), pages 281-310, March.
    24. Luke Holman & Claire Morandin, 2019. "Researchers collaborate with same-gendered colleagues more often than expected across the life sciences," PLOS ONE, Public Library of Science, vol. 14(4), pages 1-19, April.
    25. W James Jacob, 2015. "Interdisciplinary trends in higher education," Palgrave Communications, Palgrave Macmillan, vol. 1(palcomms2), pages 15001-15001, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Evelyn Eika & Frode Eika Sandnes, 2022. "Starstruck by journal prestige and citation counts? On students’ bias and perceptions of trustworthiness according to clues in publication references," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(11), pages 6363-6390, November.
    2. Giovanni Abramo & Ciriaco Andrea D’Angelo & Leonardo Grilli, 2024. "The role of non-scientific factors vis-à-vis the quality of publications in determining their scholarly impact," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(8), pages 5003-5019, August.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Martorell Cunil, Onofre & Otero González, Luis & Durán Santomil, Pablo & Mulet Forteza, Carlos, 2023. "How to accomplish a highly cited paper in the tourism, leisure and hospitality field," Journal of Business Research, Elsevier, vol. 157(C).
    2. Giovanni Abramo & Ciriaco Andrea D’Angelo & Leonardo Grilli, 2024. "The role of non-scientific factors vis-à-vis the quality of publications in determining their scholarly impact," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(8), pages 5003-5019, August.
    3. Stefano Mammola & Diego Fontaneto & Alejandro Martínez & Filipe Chichorro, 2021. "Impact of the reference list features on the number of citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(1), pages 785-799, January.
    4. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Grilli, Leonardo, 2021. "The effects of citation-based research evaluation schemes on self-citation behavior," Journal of Informetrics, Elsevier, vol. 15(4).
    5. Shen, Hongquan & Xie, Juan & Ao, Weiyi & Cheng, Ying, 2022. "The continuity and citation impact of scientific collaboration with different gender composition," Journal of Informetrics, Elsevier, vol. 16(1).
    6. Martin Szomszor & David A. Pendlebury & Jonathan Adams, 2020. "How much is too much? The difference between research influence and self-citation excess," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(2), pages 1119-1147, May.
    7. Mingyang Wang & Zhenyu Wang & Guangsheng Chen, 2019. "Which can better predict the future success of articles? Bibliometric indices or alternative metrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(3), pages 1575-1595, June.
    8. Shen, Hongquan & Cheng, Ying & Ju, Xiufang & Xie, Juan, 2022. "Rethinking the effect of inter-gender collaboration on research performance for scholars," Journal of Informetrics, Elsevier, vol. 16(4).
    9. Paulo Henrique Santos Gonçalves & Thiago Gonçalves-Souza & Ulysses Paulino Albuquerque, 2020. "Chronic anthropogenic disturbances in ecology: a bibliometric approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(2), pages 1103-1117, May.
    10. Lin Zhang & Gunnar Sivertsen & Huiying Du & Ying Huang & Wolfgang Glänzel, 2021. "Gender differences in the aims and impacts of research," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(11), pages 8861-8886, November.
    11. Manuel Marques-Cruz & Daniel Martinho Dias & João A. Fonseca & Bernardo Sousa-Pinto, 2024. "Ten year citation prediction model for systematic reviews using early years citation data," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(8), pages 4847-4862, August.
    12. Copiello, Sergio, 2019. "Peer and neighborhood effects: Citation analysis using a spatial autoregressive model and pseudo-spatial data," Journal of Informetrics, Elsevier, vol. 13(1), pages 238-254.
    13. Raminta Pranckutė, 2021. "Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World," Publications, MDPI, vol. 9(1), pages 1-59, March.
    14. Zhang, Lin & Sivertsen, Gunnar & Du, Huiying & HUANG, Ying & Glänzel, Wolfgang, 2021. "Gender differences in the aims and impacts of research," SocArXiv 9n347, Center for Open Science.
    15. Libo Sheng & Dongqing Lyu & Xuanmin Ruan & Hongquan Shen & Ying Cheng, 2023. "The association between prior knowledge and the disruption of an article," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4731-4751, August.
    16. Vogel, Rick & Hattke, Fabian & Petersen, Jessica, 2017. "Journal rankings in management and business studies: What rules do we play by?," Research Policy, Elsevier, vol. 46(10), pages 1707-1722.
    17. Rayanne Barros Setubal & Daniel Silva Farias & Clarice Casa Nova & Anna Carolina Fornero Aguiar & Tauany Aparecida Silva Santa Rosa Rodrigues & Rafael Teixeira Santos Lira & Anderson Luiz Vargas Ferre, 2022. "Microwave effect: analyzing citations from classic theories and their reinventions—a case study from a classic paper in aquatic ecology—Brooks & Dodson, 1965," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4751-4767, August.
    18. Ha, Taehyun, 2022. "An explainable artificial-intelligence-based approach to investigating factors that influence the citation of papers," Technological Forecasting and Social Change, Elsevier, vol. 184(C).
    19. Wenxuan Shi & Renli Wu, 2024. "Women’s strength in science: exploring the influence of female participation on research impact and innovation," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(7), pages 4529-4551, July.
    20. Giuseppe La Torre & Remigio Bova & Rosario Andrea Cocchiara & Cristina Sestili & Anna Tagliaferri & Simona Maggiacomo & Camilla Foschi & William Zomparelli & Maria Vittoria Manai & David Shaholli & Va, 2023. "What Are the Determinants of the Quality of Systematic Reviews in the International Journals of Occupational Medicine? A Methodological Study Review of Published Literature," IJERPH, MDPI, vol. 20(2), pages 1-12, January.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:127:y:2022:i:7:d:10.1007_s11192-022-04421-7. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.