IDEAS home Printed from https://ideas.repec.org/p/csc/cerisp/200615.html
   My bibliography  Save this paper

Peer review for the evaluation of the academic research: the Italian experience

Author

Abstract

Peer review, that is the evaluation process based on judgments formulated by independent experts, is generally used for different goals: the allocation of research funding, the review of the research results submitted for publication in scientific journals, and the assessment of the quality of research conducted by Universities and university-related Institutes. The paper deals with the latter type of peer review. The aim is to understand how the characteristics of the Italian experience provide useful lessons for improving peer review effectiveness for evaluating the academic research. More specifically, the paper investigates the peer review process developed within the Three-Year Research Assessment Exercise (VTR) in Italy. Our analysis covers four disciplinary sectors: chemistry, biology, humanities and economics. Thus, the choice includes two “hard science” sectors, which have similar type of research output submitted for the three-year evaluation process, and two sectors with different types of output. The results provide evidences, which highlight the important role played by peer review for judging the quality of the academic research in different fields of science, and for comparing different institutions’ performance. Moreover, some basic features of the evaluation process are discussed, in order to understand their usefulness for reinforcing the effectiveness of the peers’ final outcome.

Suggested Citation

  • Emanuela Reale & Anna Barbara & Antonio Costantini, 2006. "Peer review for the evaluation of the academic research: the Italian experience," CERIS Working Paper 200615, CNR-IRCrES Research Institute on Sustainable Economic Growth - Torino (TO) ITALY - former Institute for Economic Research on Firms and Growth - Moncalieri (TO) ITALY.
    • Handle: RePEc:csc:cerisp:200615
    as

    Download full text from publisher

    File URL: https://www.byterfly.eu/islandora/object/librib:352057
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Christine Wennerås & Agnes Wold, 1997. "Nepotism and sexism in peer-review," Nature, Nature, vol. 387(6631), pages 341-343, May.
    2. Per O Seglen, 1992. "How representative is the journal impact factor?," Research Evaluation, Oxford University Press, vol. 2(3), pages 143-149, December.
    3. Mario de Marchi & Maurizio Rocchi, 2001. "The editorial policies of scientific journals: Testing an impact factor model," Scientometrics, Springer;Akadémiai Kiadó, vol. 51(2), pages 395-404, June.
    4. Peter Weingart, 2005. "Impact of bibliometrics upon the science system: Inadvertent consequences?," Scientometrics, Springer;Akadémiai Kiadó, vol. 62(1), pages 117-131, January.
    5. Martin, Ben R. & Irvine, John, 1993. "Assessing basic research : Some partial indicators of scientific progress in radio astronomy," Research Policy, Elsevier, vol. 22(2), pages 106-106, April.
    6. Rinia, E. J. & van Leeuwen, Th. N. & van Vuren, H. G. & van Raan, A. F. J., 1998. "Comparative analysis of a set of bibliometric indicators and central peer review criteria: Evaluation of condensed matter physics in the Netherlands," Research Policy, Elsevier, vol. 27(1), pages 95-107, May.
    7. Rinia, E. J. & van Leeuwen, Th. N. & van Vuren, H. G. & van Raan, A. F. J., 2001. "Influence of interdisciplinarity on peer-review and bibliometric evaluations in physics research," Research Policy, Elsevier, vol. 30(3), pages 357-361, March.
    8. Dag W Aksnes & Randi Elisabeth Taxt, 2004. "Peer reviews and bibliometric indicators: a comparative study at a Norwegian university," Research Evaluation, Oxford University Press, vol. 13(1), pages 33-41, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2011. "Assessing the varying level of impact measurement accuracy as a function of the citation window length," Journal of Informetrics, Elsevier, vol. 5(4), pages 659-667.
    2. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Caprasecca, Alessandro, 2009. "Allocative efficiency in public research funding: Can bibliometrics help?," Research Policy, Elsevier, vol. 38(1), pages 206-215, February.
    3. Abramo, Giovanni, 2018. "Revisiting the scientometric conceptualization of impact and its measurement," Journal of Informetrics, Elsevier, vol. 12(3), pages 590-597.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Primož Južnič & Stojan Pečlin & Matjaž Žaucer & Tilen Mandelj & Miro Pušnik & Franci Demšar, 2010. "Scientometric indicators: peer-review, bibliometric methods and conflict of interests," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(2), pages 429-441, November.
    2. Rafols, Ismael & Leydesdorff, Loet & O’Hare, Alice & Nightingale, Paul & Stirling, Andy, 2012. "How journal rankings can suppress interdisciplinary research: A comparison between Innovation Studies and Business & Management," Research Policy, Elsevier, vol. 41(7), pages 1262-1282.
    3. van Raan, A. F. J. & van Leeuwen, Th. N., 2002. "Assessment of the scientific basis of interdisciplinary, applied research: Application of bibliometric methods in Nutrition and Food Research," Research Policy, Elsevier, vol. 31(4), pages 611-632, May.
    4. Bar-Ilan, Judit, 2008. "Informetrics at the beginning of the 21st century—A review," Journal of Informetrics, Elsevier, vol. 2(1), pages 1-52.
    5. Ulrich Schmoch & Torben Schubert, 2008. "Are international co-publications an indicator for quality of scientific research?," Scientometrics, Springer;Akadémiai Kiadó, vol. 74(3), pages 361-377, March.
    6. Giovanni Abramo & Ciriaco Andrea D’Angelo & Emanuela Reale, 2019. "Peer review versus bibliometrics: Which method better predicts the scholarly impact of publications?," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 537-554, October.
    7. Abramo, Giovanni, 2018. "Revisiting the scientometric conceptualization of impact and its measurement," Journal of Informetrics, Elsevier, vol. 12(3), pages 590-597.
    8. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    9. Brito, Ricardo & Rodríguez-Navarro, Alonso, 2018. "Research assessment by percentile-based double rank analysis," Journal of Informetrics, Elsevier, vol. 12(1), pages 315-329.
    10. Franceschet, Massimo & Costantini, Antonio, 2011. "The first Italian research assessment exercise: A bibliometric perspective," Journal of Informetrics, Elsevier, vol. 5(2), pages 275-291.
    11. Giovanni Abramo & Ciriaco Andrea D'Angelo, 2015. "The VQR, Italy's second national research assessment: Methodological failures and ranking distortions," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(11), pages 2202-2214, November.
    12. Takanori Ida & Naomi Fukuzawa, 2013. "Effects of large-scale research funding programs: a Japanese case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(3), pages 1253-1273, March.
    13. Aksnes, Dag W. & Rip, Arie, 2009. "Researchers' perceptions of citations," Research Policy, Elsevier, vol. 38(6), pages 895-905, July.
    14. Alberto Baccini & Giuseppe De Nicolao, 2016. "Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(3), pages 1651-1671, September.
    15. Lutz Bornmann & Julian N. Marewski, 2024. "Opium in science and society: numbers and other quantifications," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(9), pages 5313-5346, September.
    16. Lutz Bornmann & Julian N. Marewski, 2019. "Heuristics as conceptual lens for understanding and studying the usage of bibliometrics in research evaluation," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(2), pages 419-459, August.
    17. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2011. "Assessing the varying level of impact measurement accuracy as a function of the citation window length," Journal of Informetrics, Elsevier, vol. 5(4), pages 659-667.
    18. Jacques Wainer & Paula Vieira, 2013. "Correlations between bibliometrics and peer evaluation for all disciplines: the evaluation of Brazilian scientists," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(2), pages 395-410, August.
    19. Haddawy, Peter & Hassan, Saeed-Ul & Asghar, Awais & Amin, Sarah, 2016. "A comprehensive examination of the relation of three citation-based journal metrics to expert judgment of journal quality," Journal of Informetrics, Elsevier, vol. 10(1), pages 162-173.
    20. K. Jonkers & G. E. Derrick & C. Lopez-Illescas & P. Besselaar, 2014. "Measuring the scientific impact of e-research infrastructures: a citation based approach?," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(2), pages 1179-1194, November.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:csc:cerisp:200615. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Anna Perin or Giancarlo Birello (email available below). General contact details of provider: https://edirc.repec.org/data/cerisit.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.