IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v129y2024i8d10.1007_s11192-024-05106-z.html
   My bibliography  Save this article

The role of non-scientific factors vis-à-vis the quality of publications in determining their scholarly impact

Author

Listed:
  • Giovanni Abramo

    (Universitas Mercatorum)

  • Ciriaco Andrea D’Angelo

    (University of Rome “Tor Vergata”)

  • Leonardo Grilli

    (University of Florence, Applications “G. Parenti”)

Abstract

In the evaluation of scientific publications’ impact, the interplay between intrinsic quality and non-scientific factors remains a subject of debate. While peer review traditionally assesses quality, bibliometric techniques gauge scholarly impact. This study investigates the role of non-scientific attributes alongside quality scores from peer review in determining scholarly impact. Leveraging data from the first Italian Research Assessment Exercise (VTR 2001–2003) and Web of Science citations, we analyse the relationship between quality scores, non-scientific factors, and publication short- and long-term impact. Our findings shed light on the significance of non-scientific elements overlooked in peer review, offering policymakers and research management insights in choosing evaluation methodologies. Sections delve into the debate, identify non-scientific influences, detail methodologies, present results, and discuss implications.

Suggested Citation

  • Giovanni Abramo & Ciriaco Andrea D’Angelo & Leonardo Grilli, 2024. "The role of non-scientific factors vis-à-vis the quality of publications in determining their scholarly impact," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(8), pages 5003-5019, August.
    • Handle: RePEc:spr:scient:v:129:y:2024:i:8:d:10.1007_s11192-024-05106-z
    DOI: 10.1007/s11192-024-05106-z
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-024-05106-z
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-024-05106-z?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Bornmann, Lutz & Leydesdorff, Loet, 2015. "Does quality and content matter for citedness? A comparison with para-textual factors and over time," Journal of Informetrics, Elsevier, vol. 9(3), pages 419-429.
    2. Bertocchi, Graziella & Gambardella, Alfonso & Jappelli, Tullio & Nappi, Carmela A. & Peracchi, Franco, 2015. "Bibliometric evaluation vs. informed peer review: Evidence from Italy," Research Policy, Elsevier, vol. 44(2), pages 451-466.
    3. Vincent Larivière & Yves Gingras, 2010. "On the relationship between interdisciplinarity and scientific impact," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(1), pages 126-131, January.
    4. Vincent Larivière & Etienne Vignola-Gagné & Christian Villeneuve & Pascal Gélinas & Yves Gingras, 2011. "Sex differences in research funding, productivity and impact: an analysis of Québec university professors," Scientometrics, Springer;Akadémiai Kiadó, vol. 87(3), pages 483-498, June.
    5. Giovanni Abramo & Ciriaco Andrea D’Angelo & Fulvio Viel, 2010. "Peer review research assessment: a sensitivity analysis of performance rankings to the share of research product evaluated," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(3), pages 705-720, December.
    6. Jian Wang & Bart Thijs & Wolfgang Glänzel, 2015. "Interdisciplinarity and Impact: Distinct Effects of Variety, Balance, and Disparity," PLOS ONE, Public Library of Science, vol. 10(5), pages 1-18, May.
    7. Éric Archambault & Étienne Vignola-Gagné & Grégoire Côté & Vincent Larivière & Yves Gingrasb, 2006. "Benchmarking scientific output in the social sciences and humanities: The limits of existing databases," Scientometrics, Springer;Akadémiai Kiadó, vol. 68(3), pages 329-342, September.
    8. Stegehuis, Clara & Litvak, Nelly & Waltman, Ludo, 2015. "Predicting the long-term citation impact of recent publications," Journal of Informetrics, Elsevier, vol. 9(3), pages 642-657.
    9. P. Pichappan & S. Sarasvady, 2002. "The other side of the coin: The intricacies of author self-citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 54(2), pages 285-290, June.
    10. Tai-Quan Peng & Jonathan J.H. Zhu, 2012. "Where you publish matters most: A multilevel analysis of factors affecting citations of internet studies," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(9), pages 1789-1803, September.
    11. Chen, Shiji & Arsenault, Clément & Larivière, Vincent, 2015. "Are top-cited papers more interdisciplinary?," Journal of Informetrics, Elsevier, vol. 9(4), pages 1034-1046.
    12. Juan Xie & Kaile Gong & Ying Cheng & Qing Ke, 2019. "The correlation between paper length and citations: a meta-analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(3), pages 763-786, March.
    13. Glenn D. Walters, 2006. "Predicting subsequent citations to articles published in twelve crime-psychology journals: Author impact versus journal impact," Scientometrics, Springer;Akadémiai Kiadó, vol. 69(3), pages 499-510, December.
    14. Dag W. Aksnes & Kristoffer Rorstad & Fredrik Piro & Gunnar Sivertsen, 2011. "Are female researchers less cited? A large-scale study of Norwegian scientists," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(4), pages 628-636, April.
    15. Juan Xie & Kaile Gong & Jiang Li & Qing Ke & Hyonchol Kang & Ying Cheng, 2019. "A probe into 66 factors which are possibly associated with the number of citations an article received," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(3), pages 1429-1454, June.
    16. Mike Thelwall & Kayvan Kousha & Mahshid Abdoli & Emma Stuart & Meiko Makita & Paul Wilson & Jonathan Levitt, 2023. "Why are coauthored academic articles more cited: Higher quality or larger audience?," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 74(7), pages 791-810, July.
    17. Wolfgang Glänzel & Cornelius de Lange, 2002. "A distributional approach to multinationality measures of international scientific collaboration," Scientometrics, Springer;Akadémiai Kiadó, vol. 54(1), pages 75-89, April.
    18. Terrence A. Brooks, 1985. "Private acts and public objects: An investigation of citer motivations," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 36(4), pages 223-229, July.
    19. Xianwen Wang & Chen Liu & Wenli Mao & Zhichao Fang, 2015. "Erratum to: The open access advantage considering citation, article usage and social media attention," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(3), pages 1149-1149, June.
    20. John Rigby, 2013. "Looking for the impact of peer review: does count of funding acknowledgements really predict research impact?," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(1), pages 57-73, January.
    21. Alfredo Yegros-Yegros & Ismael Rafols & Pablo D’Este, 2015. "Does Interdisciplinary Research Lead to Higher Citation Impact? The Different Effect of Proximal and Distal Interdisciplinarity," PLOS ONE, Public Library of Science, vol. 10(8), pages 1-21, August.
    22. Bornmann, Lutz & Leydesdorff, Loet, 2013. "The validation of (advanced) bibliometric indicators through peer assessments: A comparative study using data from InCites and F1000," Journal of Informetrics, Elsevier, vol. 7(2), pages 286-291.
    23. Stremersch, Stefan & Camacho, Nuno & Vanneste, Sofie & Verniers, Isabel, 2015. "Unraveling scientific impact: Citation types in marketing journals," International Journal of Research in Marketing, Elsevier, vol. 32(1), pages 64-77.
    24. Matthew RE Symonds & Neil J Gemmell & Tamsin L Braisher & Kylie L Gorringe & Mark A Elgar, 2006. "Gender Differences in Publication Output: Towards an Unbiased Metric of Research Performance," PLOS ONE, Public Library of Science, vol. 1(1), pages 1-5, December.
    25. Abramo, Giovanni & Aksnes, Dag W. & D’Angelo, Ciriaco Andrea, 2021. "Gender differences in research performance within and between countries: Italy vs Norway," Journal of Informetrics, Elsevier, vol. 15(2).
    26. Daniel Torres-Salinas & Wenceslao Arroyo-Machado & Nicolas Robinson-Garcia, 2023. "Bibliometric denialism," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(9), pages 5357-5359, September.
    27. Dag W. Aksnes & Kristoffer Rorstad & Fredrik Piro & Gunnar Sivertsen, 2011. "Are female researchers less cited? A large‐scale study of Norwegian scientists," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 62(4), pages 628-636, April.
    28. Stefano Mammola & Elena Piano & Alberto Doretto & Enrico Caprio & Dan Chamberlain, 2022. "Measuring the influence of non-scientific features on citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(7), pages 4123-4137, July.
    29. Vincent Larivière & Yves Gingras, 2010. "On the relationship between interdisciplinarity and scientific impact," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 61(1), pages 126-131, January.
    30. Tahamtan, Iman & Bornmann, Lutz, 2018. "Core elements in the process of citing publications: Conceptual overview of the literature," Journal of Informetrics, Elsevier, vol. 12(1), pages 203-216.
    31. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Zhang, Lin, 2018. "A comparison of two approaches for measuring interdisciplinary research output: The disciplinary diversity of authors vs the disciplinary diversity of the reference list," Journal of Informetrics, Elsevier, vol. 12(4), pages 1182-1193.
    32. Abramo, Giovanni & D’Angelo, Ciriaco Andrea, 2015. "The relationship between the number of authors of a publication, its citations and the impact factor of the publishing journal: Evidence from Italy," Journal of Informetrics, Elsevier, vol. 9(4), pages 746-761.
    33. Michael S. Patterson & Simon Harris, 2009. "The relationship between reviewers’ quality-scores and number of citations for papers published in the journal Physics in Medicine and Biology from 2003–2005," Scientometrics, Springer;Akadémiai Kiadó, vol. 80(2), pages 343-349, August.
    34. Didegah, Fereshteh & Thelwall, Mike, 2013. "Which factors help authors produce the highest impact research? Collaboration, journal and document properties," Journal of Informetrics, Elsevier, vol. 7(4), pages 861-873.
    35. Giovanni Abramo & Ciriaco Andrea D’Angelo & Emanuela Reale, 2019. "Peer review versus bibliometrics: Which method better predicts the scholarly impact of publications?," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 537-554, October.
    36. Nicole Heßler & Andreas Ziegler, 2022. "Evidence-based recommendations for increasing the citation frequency of original articles," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3367-3381, June.
    37. Franceschet, Massimo & Costantini, Antonio, 2011. "The first Italian research assessment exercise: A bibliometric perspective," Journal of Informetrics, Elsevier, vol. 5(2), pages 275-291.
    38. Iman Tahamtan & Askar Safipour Afshar & Khadijeh Ahamdzadeh, 2016. "Factors affecting number of citations: a comprehensive review of the literature," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(3), pages 1195-1225, June.
    39. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Di Costa, 2014. "Inefficiency in selecting products for submission to national research assessment exercises," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(3), pages 2069-2086, March.
    40. Xianwen Wang & Chen Liu & Wenli Mao & Zhichao Fang, 2015. "The open access advantage considering citation, article usage and social media attention," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(2), pages 555-564, May.
    41. Wanjun Xia & Tianrui Li & Chongshou Li, 2023. "A review of scientific impact prediction: tasks, features and methods," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(1), pages 543-585, January.
    42. Tai‐Quan Peng & Jonathan J.H. Zhu, 2012. "Where you publish matters most: A multilevel analysis of factors affecting citations of internet studies," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 63(9), pages 1789-1803, September.
    43. Philip Ball, 2008. "A longer paper gathers more citations," Nature, Nature, vol. 455(7211), pages 274-274, September.
    44. Jonathan M. Levitt & Mike Thelwall, 2008. "Is multidisciplinary research more highly cited? A macrolevel study," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 59(12), pages 1973-1984, October.
    45. Ante, Lennart, 2022. "The relationship between readability and scientific impact: Evidence from emerging technology discourses," Journal of Informetrics, Elsevier, vol. 16(1).
    46. Elizabeth S Vieira & José A N F Gomes, 2018. "The peer-review process: The most valued dimensions according to the researcher’s scientific career," Research Evaluation, Oxford University Press, vol. 27(3), pages 246-261.
    47. Stefano Mammola & Diego Fontaneto & Alejandro Martínez & Filipe Chichorro, 2021. "Impact of the reference list features on the number of citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(1), pages 785-799, January.
    48. Jordi Duch & Xiao Han T Zeng & Marta Sales-Pardo & Filippo Radicchi & Shayna Otis & Teresa K Woodruff & Luís A Nunes Amaral, 2012. "The Possible Role of Resource Requirements and Academic Career-Choice Risk on Gender Differences in Publication Rate and Impact," PLOS ONE, Public Library of Science, vol. 7(12), pages 1-11, December.
    49. Yassine Gargouri & Chawki Hajjem & Vincent Larivière & Yves Gingras & Les Carr & Tim Brody & Stevan Harnad, 2010. "Self-Selected or Mandated, Open Access Increases Citation Impact for Higher Quality Research," PLOS ONE, Public Library of Science, vol. 5(10), pages 1-12, October.
    50. Liu, Jialin & Chen, Hongkan & Liu, Zhibo & Bu, Yi & Gu, Weiye, 2022. "Non-linearity between referencing behavior and citation impact: A large-scale, discipline-level analysis," Journal of Informetrics, Elsevier, vol. 16(3).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Martorell Cunil, Onofre & Otero González, Luis & Durán Santomil, Pablo & Mulet Forteza, Carlos, 2023. "How to accomplish a highly cited paper in the tourism, leisure and hospitality field," Journal of Business Research, Elsevier, vol. 157(C).
    2. Mingyue Sun & Tingcan Ma & Lewei Zhou & Mingliang Yue, 2023. "Analysis of the relationships among paper citation and its influencing factors: a Bayesian network-based approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(5), pages 3017-3033, May.
    3. Liu, Jialin & Chen, Hongkan & Liu, Zhibo & Bu, Yi & Gu, Weiye, 2022. "Non-linearity between referencing behavior and citation impact: A large-scale, discipline-level analysis," Journal of Informetrics, Elsevier, vol. 16(3).
    4. Fan, Lingxu & Guo, Lei & Wang, Xinhua & Xu, Liancheng & Liu, Fangai, 2022. "Does the author’s collaboration mode lead to papers’ different citation impacts? An empirical analysis based on propensity score matching," Journal of Informetrics, Elsevier, vol. 16(4).
    5. Giulio Giacomo Cantone, 2024. "How to measure interdisciplinary research? A systemic design for the model of measurement," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(8), pages 4937-4982, August.
    6. Meijun Liu & Dongbo Shi & Jiang Li, 2017. "Double-edged sword of interdisciplinary knowledge flow from hard sciences to humanities and social sciences: Evidence from China," PLOS ONE, Public Library of Science, vol. 12(9), pages 1-16, September.
    7. Meijun Liu & Sijie Yang & Yi Bu & Ning Zhang, 2023. "Female early-career scientists have conducted less interdisciplinary research in the past six decades: evidence from doctoral theses," Palgrave Communications, Palgrave Macmillan, vol. 10(1), pages 1-16, December.
    8. Shiji Chen & Yanhui Song & Fei Shu & Vincent Larivière, 2022. "Interdisciplinarity and impact: the effects of the citation time window," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(5), pages 2621-2642, May.
    9. Lina Xu & Steven Dellaportas & Jin Wang, 2022. "A study of interdisciplinary accounting research: analysing the diversity of cited references," Accounting and Finance, Accounting and Finance Association of Australia and New Zealand, vol. 62(2), pages 2131-2162, June.
    10. Shang, Yuanyuan & Sivertsen, Gunnar & Cao, Zhe & Zhang, Lin, 2021. "Gender differences in research focused on the Sustainable Development Goal of Gender Equality," SocArXiv 3fapz, Center for Open Science.
    11. Fei Shu & Jesse David Dinneen & Shiji Chen, 2022. "Measuring the disparity among scientific disciplines using Library of Congress Subject Headings," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3613-3628, June.
    12. Chen, Shiji & Qiu, Junping & Arsenault, Clément & Larivière, Vincent, 2021. "Exploring the interdisciplinarity patterns of highly cited papers," Journal of Informetrics, Elsevier, vol. 15(1).
    13. Iman Tahamtan & Askar Safipour Afshar & Khadijeh Ahamdzadeh, 2016. "Factors affecting number of citations: a comprehensive review of the literature," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(3), pages 1195-1225, June.
    14. Abramo, Giovanni, 2018. "Revisiting the scientometric conceptualization of impact and its measurement," Journal of Informetrics, Elsevier, vol. 12(3), pages 590-597.
    15. Elmira Janavi & Mohammad Javad Mansourzadeh & Mojgan Samandar Ali Eshtehardi, 2020. "A methodology for developing scientific diversification strategy of countries," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2229-2264, December.
    16. Juan Xie & Kaile Gong & Jiang Li & Qing Ke & Hyonchol Kang & Ying Cheng, 2019. "A probe into 66 factors which are possibly associated with the number of citations an article received," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(3), pages 1429-1454, June.
    17. Raminta Pranckutė, 2021. "Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World," Publications, MDPI, vol. 9(1), pages 1-59, March.
    18. Zhang, Mengya & Zhang, Gupeng & Liu, Yun & Zhai, Xiaorong & Han, Xinying, 2020. "Scientists’ genders and international academic collaboration: An empirical study of Chinese universities and research institutes," Journal of Informetrics, Elsevier, vol. 14(4).
    19. Elizabeth S. Vieira, 2023. "The influence of research collaboration on citation impact: the countries in the European Innovation Scoreboard," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(6), pages 3555-3579, June.
    20. Yuanyuan Shang & Gunnar Sivertsen & Zhe Cao & Lin Zhang, 2022. "Gender differences among first authors in research focused on the Sustainable Development Goal of Gender Equality," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4769-4796, August.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:129:y:2024:i:8:d:10.1007_s11192-024-05106-z. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.