IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v128y2023i1d10.1007_s11192-022-04581-6.html
   My bibliography  Save this article

Do open citations give insights on the qualitative peer-review evaluation in research assessments? An analysis of the Italian National Scientific Qualification

Author

Listed:
  • Federica Bologna

    (Cornell University)

  • Angelo Iorio

    (University of Bologna)

  • Silvio Peroni

    (University of Bologna)

  • Francesco Poggi

    (Italian National Research Council (CNR))

Abstract

In the past, several works have investigated ways for combining quantitative and qualitative methods in research assessment exercises. Indeed, the Italian National Scientific Qualification (NSQ), i.e. the national assessment exercise which aims at deciding whether a scholar can apply to professorial academic positions as Associate Professor and Full Professor, adopts a quantitative and qualitative evaluation process: it makes use of bibliometrics followed by a peer-review process of candidates’ CVs. The NSQ divides academic disciplines into two categories, i.e. citation-based disciplines (CDs) and non-citation-based disciplines (NDs), a division that affects the metrics used for assessing the candidates of that discipline in the first part of the process, which is based on bibliometrics. In this work, we aim at exploring whether citation-based metrics, calculated only considering open bibliographic and citation data, can support the human peer-review of NDs and yield insights on how it is conducted. To understand if and what citation-based (and, possibly, other) metrics provide relevant information, we created a series of machine learning models to replicate the decisions of the NSQ committees. As one of the main outcomes of our study, we noticed that the strength of the citational relationship between the candidate and the commission in charge of assessing his/her CV seems to play a role in the peer-review phase of the NSQ of NDs.

Suggested Citation

  • Federica Bologna & Angelo Iorio & Silvio Peroni & Francesco Poggi, 2023. "Do open citations give insights on the qualitative peer-review evaluation in research assessments? An analysis of the Italian National Scientific Qualification," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(1), pages 19-53, January.
  • Handle: RePEc:spr:scient:v:128:y:2023:i:1:d:10.1007_s11192-022-04581-6
    DOI: 10.1007/s11192-022-04581-6
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-022-04581-6
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL: https://libkey.io/10.1007/s11192-022-04581-6?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Raffaele Miniaci & Michele Pezzoni, 2020. "Social connections and editorship in economics," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 53(3), pages 1292-1317, August.
    2. Andrea Giovanni Nuzzolese & Paolo Ciancarini & Aldo Gangemi & Silvio Peroni & Francesco Poggi & Valentina Presutti, 2019. "Do altmetrics work for assessing research quality?," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(2), pages 539-562, February.
    3. Erjia Yan & Cassidy R. Sugimoto, 2011. "Institutional interactions: Exploring social, cognitive, and geographic relationships between institutions as demonstrated through citation networks," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 62(8), pages 1498-1514, August.
    4. Kevin W. Boyack & Richard Klavans, 2010. "Co‐citation analysis, bibliographic coupling, and direct citation: Which citation approach represents the research front most accurately?," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 61(12), pages 2389-2404, December.
    5. Ashkan Ebadi & Andrea Schiffauerova, 2015. "How to Receive More Funding for Your Research? Get Connected to the Right People!," PLOS ONE, Public Library of Science, vol. 10(7), pages 1-19, July.
    6. Natalia Zinovyeva & Manuel Bagues, 2015. "The Role of Connections in Academic Promotions," American Economic Journal: Applied Economics, American Economic Association, vol. 7(2), pages 264-292, April.
    7. Milojević, Staša & Sugimoto, Cassidy R. & Larivière, Vincent & Thelwall, Mike & Ding, Ying, 2014. "The role of handbooks in knowledge creation and diffusion: A case of science and technology studies," Journal of Informetrics, Elsevier, vol. 8(3), pages 693-709.
    8. Saeed-Ul Hassan & Iqra Safder & Anam Akram & Faisal Kamiran, 2018. "A novel machine-learning approach to measuring scientific knowledge flows using citation context analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(2), pages 973-996, August.
    9. Holly Else, 2018. "How Unpaywall is transforming open science," Nature, Nature, vol. 560(7718), pages 290-291, August.
    10. Erjia Yan & Ying Ding, 2012. "Scholarly network similarities: How bibliographic coupling networks, citation networks, cocitation networks, topical networks, coauthorship networks, and coword networks relate to each other," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 63(7), pages 1313-1326, July.
    11. Erjia Yan & Ying Ding, 2012. "Scholarly network similarities: How bibliographic coupling networks, citation networks, cocitation networks, topical networks, coauthorship networks, and coword networks relate to each other," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 63(7), pages 1313-1326, July.
    12. Bagues, Manuel & Sylos-Labini, Mauro & Zinovyeva, Natalia, 2019. "Connections in scientific committees and applicants’ self-selection: Evidence from a natural randomized experiment," Labour Economics, Elsevier, vol. 58(C), pages 81-97.
    13. Lawrence D. Fu & Yindalon Aphinyanaphongs & Constantin F. Aliferis, 2013. "Computer models for identifying instrumental citations in the biomedical literature," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(3), pages 871-882, December.
    14. Yu-Wei Chang & Mu-Hsuan Huang & Chiao-Wen Lin, 2015. "Evolution of research subjects in library and information science based on keyword, bibliographical coupling, and co-citation analyses," Scientometrics, Springer;Akadémiai Kiadó, vol. 105(3), pages 2071-2087, December.
    15. Nees Jan Eck & Ludo Waltman, 2010. "Software survey: VOSviewer, a computer program for bibliometric mapping," Scientometrics, Springer;Akadémiai Kiadó, vol. 84(2), pages 523-538, August.
    16. Ivan Heibi & Silvio Peroni & David Shotton, 2019. "Software review: COCI, the OpenCitations Index of Crossref open DOI-to-DOI citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(2), pages 1213-1228, November.
    17. Guijie Zhang & Luning Liu & Yuqiang Feng & Zhen Shao & Yongli Li, 2014. "Cext-N index: a network node centrality measure for collaborative relationship distribution," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 291-307, October.
    18. Kevin W. Boyack & Richard Klavans, 2010. "Co-citation analysis, bibliographic coupling, and direct citation: Which citation approach represents the research front most accurately?," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(12), pages 2389-2404, December.
    19. Howard D. White, 2004. "Reward, persuasion, and the Sokal Hoax: A study in citation identities," Scientometrics, Springer;Akadémiai Kiadó, vol. 60(1), pages 93-120, May.
    20. Tommaso Colussi, 2018. "Social Ties in Academia: A Friend Is a Treasure," The Review of Economics and Statistics, MIT Press, vol. 100(1), pages 45-50, March.
    21. Giovanni Abramo & Ciriaco Andrea D’Angelo & Francesco Rosati, 2015. "The determinants of academic career advancement: Evidence from Italy," Science and Public Policy, Oxford University Press, vol. 42(6), pages 761-774.
    22. Dmytro Cheberkus & Serhii Nazarovets, 2019. "Ukrainian open index maps local citations," Nature, Nature, vol. 575(7784), pages 596-596, November.
    23. Erjia Yan & Cassidy R. Sugimoto, 2011. "Institutional interactions: Exploring social, cognitive, and geographic relationships between institutions as demonstrated through citation networks," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(8), pages 1498-1514, August.
    24. Liwei Cai & Jiahao Tian & Jiaying Liu & Xiaomei Bai & Ivan Lee & Xiangjie Kong & Feng Xia, 2019. "Scholarly impact assessment: a survey of citation weighting solutions," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(2), pages 453-478, February.
    25. Lawrence D. Fu & Constantin F. Aliferis, 2010. "Using content-based and bibliometric features for machine learning models to predict citation counts in the biomedical literature," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(1), pages 257-270, October.
    26. Brogaard, Jonathan & Engelberg, Joseph & Parsons, Christopher A., 2014. "Networks and productivity: Causal evidence from editor rotations," Journal of Financial Economics, Elsevier, vol. 111(1), pages 251-270.
    27. Henry Small, 2004. "On the shoulders of Robert Merton: Towards a normative theory of citation," Scientometrics, Springer;Akadémiai Kiadó, vol. 60(1), pages 71-79, May.
    28. Maria Cristiana Martini & Elvira Pelle & Francesco Poggi & Andrea Sciandra, 2022. "The role of citation networks to explain academic promotions: an empirical analysis of the Italian national scientific qualification," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(10), pages 5633-5659, October.
    29. Teplitskiy, Misha & Acuna, Daniel & Elamrani-Raoult, Aïda & Körding, Konrad & Evans, James, 2018. "The sociology of scientific validity: How professional networks shape judgement in peer review," Research Policy, Elsevier, vol. 47(9), pages 1825-1841.
    30. Silvio Peroni & Paolo Ciancarini & Aldo Gangemi & Andrea Giovanni Nuzzolese & Francesco Poggi & Valentina Presutti, 2020. "The practice of self-citations: a longitudinal study," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(1), pages 253-282, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jun-Ping Qiu & Ke Dong & Hou-Qiang Yu, 2014. "Comparative study on structure and correlation among author co-occurrence networks in bibliometrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(2), pages 1345-1360, November.
    2. Dorsa Alipour & Hussein Dia, 2023. "A Systematic Review of the Role of Land Use, Transport, and Energy-Environment Integration in Shaping Sustainable Cities," Sustainability, MDPI, vol. 15(8), pages 1-29, April.
    3. Song Yanhui & Wu Lijuan & Qiu Junping, 2021. "A comparative study of first and all-author bibliographic coupling analysis based on Scientometrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1125-1147, February.
    4. Guan-Can Yang & Gang Li & Chun-Ya Li & Yun-Hua Zhao & Jing Zhang & Tong Liu & Dar-Zen Chen & Mu-Hsuan Huang, 2015. "Using the comprehensive patent citation network (CPC) to evaluate patent value," Scientometrics, Springer;Akadémiai Kiadó, vol. 105(3), pages 1319-1346, December.
    5. Yu-Wei Chang & Mu-Hsuan Huang & Chiao-Wen Lin, 2015. "Evolution of research subjects in library and information science based on keyword, bibliographical coupling, and co-citation analyses," Scientometrics, Springer;Akadémiai Kiadó, vol. 105(3), pages 2071-2087, December.
    6. Yang, Siluo & Han, Ruizhen & Wolfram, Dietmar & Zhao, Yuehua, 2016. "Visualizing the intellectual structure of information science (2006–2015): Introducing author keyword coupling analysis," Journal of Informetrics, Elsevier, vol. 10(1), pages 132-150.
    7. van der Have, Robert P. & Rubalcaba, Luis, 2016. "Social innovation research: An emerging area of innovation studies?," Research Policy, Elsevier, vol. 45(9), pages 1923-1935.
    8. Ductor, Lorenzo & Visser, Bauke, 2022. "When a coauthor joins an editorial board," Journal of Economic Behavior & Organization, Elsevier, vol. 200(C), pages 576-595.
    9. Yang, Siluo & Wang, Feifei, 2015. "Visualizing information science: Author direct citation analysis in China and around the world," Journal of Informetrics, Elsevier, vol. 9(1), pages 208-225.
    10. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Di Costa, Flavia, 2020. "The role of geographical proximity in knowledge diffusion, measured by citations to scientific literature," Journal of Informetrics, Elsevier, vol. 14(1).
    11. Ali Gazni & Fereshteh Didegah, 2016. "The relationship between authors’ bibliographic coupling and citation exchange: analyzing disciplinary differences," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(2), pages 609-626, May.
    12. Wang, Feifei & Jia, Chenran & Wang, Xiaohan & Liu, Junwan & Xu, Shuo & Liu, Yang & Yang, Chenyuyan, 2019. "Exploring all-author tripartite citation networks: A case study of gene editing," Journal of Informetrics, Elsevier, vol. 13(3), pages 856-873.
    13. Raffaele Miniaci & Michele Pezzoni, 2020. "Social connections and editorship in economics," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 53(3), pages 1292-1317, August.
    14. Ba, Zhichao & Liang, Zhentao, 2021. "A novel approach to measuring science-technology linkage: From the perspective of knowledge network coupling," Journal of Informetrics, Elsevier, vol. 15(3).
    15. Muh-Chyun Tang & Yun Jen Cheng & Kuang Hua Chen, 2017. "A longitudinal study of intellectual cohesion in digital humanities using bibliometric analyses," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(2), pages 985-1008, November.
    16. Pournader, Mehrdokht & Ghaderi, Hadi & Hassanzadegan, Amir & Fahimnia, Behnam, 2021. "Artificial intelligence applications in supply chain management," International Journal of Production Economics, Elsevier, vol. 241(C).
    17. Hervas Oliver,Jose Luis & Gonzalez,Gregorio & Caja,Pedro, 2014. "Clusters and industrial districts: where is the literature going? Identifying emerging sub-fields of research," INGENIO (CSIC-UPV) Working Paper Series 201409, INGENIO (CSIC-UPV).
    18. Bu, Yi & Ni, Shaokang & Huang, Win-bin, 2017. "Combining multiple scholarly relationships with author cocitation analysis: A preliminary exploration on improving knowledge domain mappings," Journal of Informetrics, Elsevier, vol. 11(3), pages 810-822.
    19. Piñeiro-Chousa, Juan & López-Cabarcos, M. Ángeles & Romero-Castro, Noelia María & Pérez-Pico, Ada María, 2020. "Innovation, entrepreneurship and knowledge in the business scientific field: Mapping the research front," Journal of Business Research, Elsevier, vol. 115(C), pages 475-485.
    20. Yucheng Zhang & Zhiling Wang & Lin Xiao & Lijun Wang & Pei Huang, 2023. "Discovering the evolution of online reviews: A bibliometric review," Electronic Markets, Springer;IIM University of St. Gallen, vol. 33(1), pages 1-22, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:128:y:2023:i:1:d:10.1007_s11192-022-04581-6. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.