IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v10y2016i2p408-438.html
   My bibliography  Save this article

Assessing evaluation procedures for individual researchers: The case of the Italian National Scientific Qualification

Author

Listed:
  • Marzolla, Moreno

Abstract

The Italian National Scientific Qualification (ASN) was introduced as a prerequisite for applying for tenured associate or full professor positions at state-recognized universities. The ASN is meant to attest that an individual has reached a suitable level of scientific maturity to apply for professorship positions. A five member panel, appointed for each scientific discipline, is in charge of evaluating applicants by means of quantitative indicators of impact and productivity, and through an assessment of their research profile. Many concerns were raised on the appropriateness of the evaluation criteria, and in particular on the use of bibliometrics for the evaluation of individual researchers. Additional concerns were related to the perceived poor quality of the final evaluation reports. In this paper we assess the ASN in terms of appropriateness of the applied methodology, and the quality of the feedback provided to the applicants. We argue that the ASN is not fully compliant with the best practices for the use of bibliometric indicators for the evaluation of individual researchers; moreover, the quality of final reports varies considerably across the panels, suggesting that measures should be put in place to prevent sloppy practices in future ASN rounds.

Suggested Citation

  • Marzolla, Moreno, 2016. "Assessing evaluation procedures for individual researchers: The case of the Italian National Scientific Qualification," Journal of Informetrics, Elsevier, vol. 10(2), pages 408-438.
    • Handle: RePEc:eee:infome:v:10:y:2016:i:2:p:408-438
    DOI: 10.1016/j.joi.2016.01.009
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157715300961
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2016.01.009?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Abbasi, Alireza & Hossain, Liaquat & Leydesdorff, Loet, 2012. "Betweenness centrality as a driver of preferential attachment in the evolution of research collaboration networks," Journal of Informetrics, Elsevier, vol. 6(3), pages 403-412.
    2. Pedro Albarrán & Juan A. Crespo & Ignacio Ortuño & Javier Ruiz-Castillo, 2011. "The skewness of science in 219 sub-fields and a number of aggregates," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(2), pages 385-397, August.
    3. Antonis Sidiropoulos & Dimitrios Katsaros & Yannis Manolopoulos, 2007. "Generalized Hirsch h-index for disclosing latent facts in citation networks," Scientometrics, Springer;Akadémiai Kiadó, vol. 72(2), pages 253-280, August.
    4. Vanclay, Jerome K., 2011. "An evaluation of the Australian Research Council's journal ranking," Journal of Informetrics, Elsevier, vol. 5(2), pages 265-274.
    5. Giovanni Abramo & Ciriaco D’Angelo, 2015. "An assessment of the first “scientific habilitation” for university appointments in Italy," Economia Politica: Journal of Analytical and Institutional Economics, Springer;Fondazione Edison, vol. 32(3), pages 329-357, December.
    6. Yoshiko Okubo, 1997. "Bibliometric Indicators and Analysis of Research Systems: Methods and Examples," OECD Science, Technology and Industry Working Papers 1997/1, OECD Publishing.
    7. Marzolla, Moreno, 2015. "Quantitative analysis of the Italian National Scientific Qualification," Journal of Informetrics, Elsevier, vol. 9(2), pages 285-316.
    8. Marco Pautasso, 2015. "The Italian University Habilitation and the Challenge of Increasing the Representation of Women in Academia," Challenges, MDPI, vol. 6(1), pages 1-16, March.
    9. van Rijnsoever, Frank J. & Hessels, Laurens K., 2011. "Factors associated with disciplinary and interdisciplinary research collaboration," Research Policy, Elsevier, vol. 40(3), pages 463-472, April.
    10. Diana Hicks & Paul Wouters & Ludo Waltman & Sarah de Rijcke & Ismael Rafols, 2015. "Bibliometrics: The Leiden Manifesto for research metrics," Nature, Nature, vol. 520(7548), pages 429-431, April.
    11. Bennion, Alice & Locke, William, 2010. "The Early Career Paths and Employment Conditions of the Academic Profession in 17 Countries," European Review, Cambridge University Press, vol. 18(S1), pages 7-33, May.
    12. Wagner, Caroline S. & Roessner, J. David & Bobb, Kamau & Klein, Julie Thompson & Boyack, Kevin W. & Keyton, Joann & Rafols, Ismael & Börner, Katy, 2011. "Approaches to understanding and measuring interdisciplinary scientific research (IDR): A review of the literature," Journal of Informetrics, Elsevier, vol. 5(1), pages 14-26.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Andersen, Jens Peter & Nielsen, Mathias Wullum, 2018. "Google Scholar and Web of Science: Examining gender differences in citation coverage across five scientific disciplines," Journal of Informetrics, Elsevier, vol. 12(3), pages 950-959.
    2. Giulio Marini & Viviana Meschitti, 2018. "The trench warfare of gender discrimination: evidence from academic promotions to full professor in Italy," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(2), pages 989-1006, May.
    3. Casamonti, Matilde & Zinovyeva, Natalia, 2024. "Gendered Language in Academic Evaluations: Evidence from the Italian University System," CAGE Online Working Paper Series 720, Competitive Advantage in the Global Economy (CAGE).
    4. Zbigniew Koza & Robert Lew & Emanuel Kulczycki & Piotr Stec, 2023. "Who Controls the National Academic Promotion System: An Analysis of Power Distribution in Poland," SAGE Open, , vol. 13(2), pages 21582440231, May.
    5. Aurora A. C. Teixeira & Pedro Cosme Vieira & Ana Patrícia Abreu, 2017. "Sleeping Beauties and their princes in innovation studies," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(2), pages 541-580, February.
    6. Silvio Peroni & Paolo Ciancarini & Aldo Gangemi & Andrea Giovanni Nuzzolese & Francesco Poggi & Valentina Presutti, 2020. "The practice of self-citations: a longitudinal study," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(1), pages 253-282, April.
    7. Maria Cristiana Martini & Elvira Pelle & Francesco Poggi & Andrea Sciandra, 2022. "The role of citation networks to explain academic promotions: an empirical analysis of the Italian national scientific qualification," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(10), pages 5633-5659, October.
    8. Camil Demetrescu & Irene Finocchi & Andrea Ribichini & Marco Schaerf, 2020. "On bibliometrics in academic promotions: a case study in computer science and engineering in Italy," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(3), pages 2207-2228, September.
    9. Federico Scarpa & Vincenzo Bianco & Luca A. Tagliafico, 2018. "The impact of the national assessment exercises on self-citation rate and publication venue: an empirical investigation on the engineering academic sector in Italy," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(2), pages 997-1022, November.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Camil Demetrescu & Irene Finocchi & Andrea Ribichini & Marco Schaerf, 2020. "On bibliometrics in academic promotions: a case study in computer science and engineering in Italy," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(3), pages 2207-2228, September.
    2. Giovanni Abramo & Ciriaco D’Angelo, 2015. "An assessment of the first “scientific habilitation” for university appointments in Italy," Economia Politica: Journal of Analytical and Institutional Economics, Springer;Fondazione Edison, vol. 32(3), pages 329-357, December.
    3. Zuo, Zhiya & Zhao, Kang, 2018. "The more multidisciplinary the better? – The prevalence and interdisciplinarity of research collaborations in multidisciplinary institutions," Journal of Informetrics, Elsevier, vol. 12(3), pages 736-756.
    4. Rafols, Ismael & Leydesdorff, Loet & O’Hare, Alice & Nightingale, Paul & Stirling, Andy, 2012. "How journal rankings can suppress interdisciplinary research: A comparison between Innovation Studies and Business & Management," Research Policy, Elsevier, vol. 41(7), pages 1262-1282.
    5. Wildgaard, Lorna, 2016. "A critical cluster analysis of 44 indicators of author-level performance," Journal of Informetrics, Elsevier, vol. 10(4), pages 1055-1078.
    6. Tobias Kiesslich & Marlena Beyreis & Georg Zimmermann & Andreas Traweger, 2021. "Citation inequality and the Journal Impact Factor: median, mean, (does it) matter?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1249-1269, February.
    7. Kaur, Jasleen & Ferrara, Emilio & Menczer, Filippo & Flammini, Alessandro & Radicchi, Filippo, 2015. "Quality versus quantity in scientific impact," Journal of Informetrics, Elsevier, vol. 9(4), pages 800-808.
    8. Jiang Wu & Miao Jin & Xiu-Hao Ding, 2015. "Diversity of individual research disciplines in scientific funding," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(2), pages 669-686, May.
    9. Bornmann, Lutz & Leydesdorff, Loet, 2017. "Skewness of citation impact data and covariates of citation distributions: A large-scale empirical analysis based on Web of Science data," Journal of Informetrics, Elsevier, vol. 11(1), pages 164-175.
    10. Adena R. Rissman & Lori Barrow, 2019. "Characteristics of collaborative, interdisciplinary, and engaged research among graduate students in environmental conservation," Journal of Environmental Studies and Sciences, Springer;Association of Environmental Studies and Sciences, vol. 9(3), pages 297-310, September.
    11. Lin Zhang & Beibei Sun & Zaida Chinchilla-Rodríguez & Lixin Chen & Ying Huang, 2018. "Interdisciplinarity and collaboration: on the relationship between disciplinary diversity in departmental affiliations and reference lists," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(1), pages 271-291, October.
    12. Shunshun Shi & Wenyu Zhang & Shuai Zhang & Jie Chen, 2018. "Does prestige dimension influence the interdisciplinary performance of scientific entities in knowledge flow? Evidence from the e-government field," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(2), pages 1237-1264, November.
    13. Lungeanu, Alina & Huang, Yun & Contractor, Noshir S., 2014. "Understanding the assembly of interdisciplinary teams and its impact on performance," Journal of Informetrics, Elsevier, vol. 8(1), pages 59-70.
    14. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Zhang, Lin, 2018. "A comparison of two approaches for measuring interdisciplinary research output: The disciplinary diversity of authors vs the disciplinary diversity of the reference list," Journal of Informetrics, Elsevier, vol. 12(4), pages 1182-1193.
    15. Marzolla, Moreno, 2015. "Quantitative analysis of the Italian National Scientific Qualification," Journal of Informetrics, Elsevier, vol. 9(2), pages 285-316.
    16. Lu Liu & Benjamin F. Jones & Brian Uzzi & Dashun Wang, 2023. "Data, measurement and empirical methods in the science of science," Nature Human Behaviour, Nature, vol. 7(7), pages 1046-1058, July.
    17. Eli Rudinow Saetnan & Richard Philip Kipling, 2016. "Evaluating a European knowledge hub on climate change in agriculture: Are we building a better connected community?," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(2), pages 1057-1074, November.
    18. Shengli Deng & Sudi Xia, 2020. "Mapping the interdisciplinarity in information behavior research: a quantitative study using diversity measure and co-occurrence analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(1), pages 489-513, July.
    19. Thelwall, Mike & Kousha, Kayvan & Stuart, Emma & Makita, Meiko & Abdoli, Mahshid & Wilson, Paul & Levitt, Jonathan, 2023. "Do bibliometrics introduce gender, institutional or interdisciplinary biases into research evaluations?," Research Policy, Elsevier, vol. 52(8).
    20. Carr, Gemma & Loucks, Daniel P. & Blöschl, Günter, 2018. "Gaining insight into interdisciplinary research and education programmes: A framework for evaluation," Research Policy, Elsevier, vol. 47(1), pages 35-48.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:10:y:2016:i:2:p:408-438. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.