IDEAS home Printed from https://ideas.repec.org/a/oup/rseval/v24y2015i3p242-255..html
   My bibliography  Save this article

Evaluating scientific research in Italy: The 2004–10 research evaluation exercise

Author

Listed:
  • Alessio Ancaiani
  • Alberto F. Anfossi
  • Anna Barbara
  • Sergio Benedetto
  • Brigida Blasi
  • Valentina Carletti
  • Tindaro Cicero
  • Alberto Ciolfi
  • Filippo Costa
  • Giovanna Colizza
  • Marco Costantini
  • Fabio di Cristina
  • Antonio Ferrara
  • Rosa M. Lacatena
  • Marco Malgarini
  • Irene Mazzotta
  • Carmela A. Nappi
  • Sandra Romagnosi
  • Serena Sileoni

Abstract

The Italian Research Evaluation assessment for the period 2004–10 (VQR 2004–10) has analyzed almost 185,000 articles, books, patents, and other scientific outcomes submitted for evaluation by Italian universities and other public research bodies. This article describes the main features of this exercise, introducing its legal framework and the criteria used for evaluation. The innovative methodology that has been used for evaluation, based on a combination of peer review and bibliometric methods, is discussed and indicators for assessing the quality of participating research bodies are derived accordingly. The article also presents the main results obtained at the University level, trying to understand the existing relationship among research quality and University characteristics such as location, dimension, age, scientific specialization, and funding.

Suggested Citation

  • Alessio Ancaiani & Alberto F. Anfossi & Anna Barbara & Sergio Benedetto & Brigida Blasi & Valentina Carletti & Tindaro Cicero & Alberto Ciolfi & Filippo Costa & Giovanna Colizza & Marco Costantini & F, 2015. "Evaluating scientific research in Italy: The 2004–10 research evaluation exercise," Research Evaluation, Oxford University Press, vol. 24(3), pages 242-255.
    • Handle: RePEc:oup:rseval:v:24:y:2015:i:3:p:242-255.
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1093/reseval/rvv008
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Seidl, Christian & Schmidt, Ulrich & Grösche, Peter, 2005. "The Performance of Peer Review and a Beauty Contest of Referee Processes of Economics Journals/," Estudios de Economia Aplicada, Estudios de Economia Aplicada, vol. 23, pages 505-551, Diciembre.
    2. Zvi Griliches, 1998. "R&D and Productivity: The Econometric Evidence," NBER Books, National Bureau of Economic Research, Inc, number gril98-1.
    3. Griliches, Zvi, 1998. "R&D and Productivity," National Bureau of Economic Research Books, University of Chicago Press, edition 1, number 9780226308869, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Luigi Aldieri & Gennaro Guida & Maxim Kotsemir & Concetto Paolo Vinci, 2019. "An investigation of impact of research collaboration on academic performance in Italy," Quality & Quantity: International Journal of Methodology, Springer, vol. 53(4), pages 2003-2040, July.
    2. Jappelli, Tullio & Nappi, Carmela Anna & Torrini, Roberto, 2017. "Gender effects in research evaluation," Research Policy, Elsevier, vol. 46(5), pages 911-924.
    3. Tindaro Cicero & Marco Malgarini, 2020. "On the use of journal classification in social sciences and humanities: evidence from an Italian database," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1689-1708, November.
    4. Francesco Giovanni Avallone & Alberto Quagli & Paola Ramassa, 2022. "Interdisciplinary research by accounting scholars: An exploratory study," FINANCIAL REPORTING, FrancoAngeli Editore, vol. 2022(2), pages 5-34.
    5. Geuna, Aldo & Piolatto, Matteo, 2016. "Research assessment in the UK and Italy: Costly and difficult, but probably worth it (at least for a while)," Research Policy, Elsevier, vol. 45(1), pages 260-271.
    6. Camil Demetrescu & Andrea Ribichini & Marco Schaerf, 2020. "Are Italian research assessment exercises size-biased?," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 533-549, October.
    7. Graziella Bertocchi & Alfonso Gambardella & Tullio Jappelli & Carmela Anna Nappi & Franco Peracchi, 2016. "Comment to: Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(1), pages 349-353, July.
    8. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    9. Konstantin Fursov & Yana Roschina & Oksana Balmush, 2016. "Determinants of Research Productivity: An Individual-level Lens," Foresight and STI Governance (Foresight-Russia till No. 3/2015), National Research University Higher School of Economics, vol. 10(2), pages 44-56.
    10. Alberto Baccini & Giuseppe De Nicolao, 2016. "Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(3), pages 1651-1671, September.
    11. Franceschini, Fiorenzo & Maisano, Domenico, 2017. "Critical remarks on the Italian research assessment exercise VQR 2011–2014," Journal of Informetrics, Elsevier, vol. 11(2), pages 337-357.
    12. Federico Scarpa & Vincenzo Bianco & Luca A. Tagliafico, 2018. "The impact of the national assessment exercises on self-citation rate and publication venue: an empirical investigation on the engineering academic sector in Italy," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(2), pages 997-1022, November.
    13. Thiago H. P. Silva & Alberto H. F. Laender & Clodoveu A. Davis & Ana Paula Couto Silva & Mirella M. Moro, 2017. "A profile analysis of the top Brazilian Computer Science graduate programs," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(1), pages 237-255, October.
    14. Kulczycki, Emanuel & Korzeń, Marcin & Korytkowski, Przemysław, 2017. "Toward an excellence-based research funding system: Evidence from Poland," Journal of Informetrics, Elsevier, vol. 11(1), pages 282-298.
    15. Buckle, Robert A. & Creedy, John & Ball, Ashley, 2020. "A Schumpeterian Gale: Using Longitudinal Data to Evaluate Responses to Performance-Based Research Funding Systems," Working Paper Series 21104, Victoria University of Wellington, Chair in Public Finance.
    16. Robert A. Buckle & John Creedy, 2022. "Methods to evaluate institutional responses to performance‐based research funding systems," Australian Economic Papers, Wiley Blackwell, vol. 61(3), pages 615-634, September.
    17. Alberto Anfossi & Alberto Ciolfi & Filippo Costa & Giorgio Parisi & Sergio Benedetto, 2016. "Large-scale assessment of research outputs through a weighted combination of bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(2), pages 671-683, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Valeria Costantini & Francesco Crespi & Giovanni Marin & Elena Paglialunga, 2016. "Eco-innovation, sustainable supply chains and environmental performance in European industries," LEM Papers Series 2016/19, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    2. Kumar, Sanjesh & Singh, Baljeet, 2019. "Barriers to the international diffusion of technological innovations," Economic Modelling, Elsevier, vol. 82(C), pages 74-86.
    3. Georgios Gioldasis & Antonio Musolesi & Michel Simioni, 2020. "Model uncertainty, nonlinearities and out-of-sample comparison: evidence from international technology diffusion," Working Papers hal-02790523, HAL.
    4. Ayhan, Fatih & Elal, Onuray, 2023. "The IMPACTS of technological change on employment: Evidence from OECD countries with panel data analysis," Technological Forecasting and Social Change, Elsevier, vol. 190(C).
    5. Cassiman, Bruno & Perez-Castrillo, David & Veugelers, Reinhilde, 2002. "Endogenizing know-how flows through the nature of R&D investments," International Journal of Industrial Organization, Elsevier, vol. 20(6), pages 775-799, June.
    6. Ufuk Akcigit & Douglas Hanley & Stefanie Stantcheva, 2022. "Optimal Taxation and R&D Policies," Econometrica, Econometric Society, vol. 90(2), pages 645-684, March.
    7. Thomas Bolli & Martin Woerter, 2013. "Technological Diversification and Innovation Performance," KOF Working papers 13-336, KOF Swiss Economic Institute, ETH Zurich.
    8. Link, Albert N. & Siegel, Donald S. & Van Fleet, David D., 2011. "Public science and public innovation: Assessing the relationship between patenting at U.S. National Laboratories and the Bayh-Dole Act," Research Policy, Elsevier, vol. 40(8), pages 1094-1099, October.
    9. Pietro Moncada-Paternò-Castello, 2022. "Top R&D investors, structural change and the R&D growth performance of young and old firms," Eurasian Business Review, Springer;Eurasia Business and Economics Society, vol. 12(1), pages 1-33, March.
    10. repec:hal:wpspec:info:hdl:2441/941 is not listed on IDEAS
    11. Beatriz Pereira Almeida & Eduardo Gonçalves & André Suriane Silva & Raquel Coelho Reis, 2021. "Internalization of knowledge spillovers by regions: a measure based on self-citation patents," The Annals of Regional Science, Springer;Western Regional Science Association, vol. 66(2), pages 309-330, April.
    12. Antonelli, Cristiano, 2017. "Digital knowledge generation and the appropriability trade-off," Telecommunications Policy, Elsevier, vol. 41(10), pages 991-1002.
    13. repec:spo:wpmain:info:hdl:2441/941 is not listed on IDEAS
    14. Ramesh Chandra Das & Sujata Mukherjee, 2020. "Do Spending on R&D Influence Income? An Enquiry on the World’s Leading Economies and Groups," Journal of the Knowledge Economy, Springer;Portland International Center for Management of Engineering and Technology (PICMET), vol. 11(4), pages 1295-1315, December.
    15. Soumyananda Dinda, 2018. "Production technology and carbon emission: long-run relation with short-run dynamics," Journal of Applied Economics, Taylor & Francis Journals, vol. 21(1), pages 106-121, January.
    16. Stavins, Robert & Jaffe, Adam & Newell, Richard, 2000. "Technological Change and the Environment," Working Paper Series rwp00-002, Harvard University, John F. Kennedy School of Government.
    17. Diemer, Andreas & Regan, Tanner, 2022. "No inventor is an island: Social connectedness and the geography of knowledge flows in the US," Research Policy, Elsevier, vol. 51(2).
    18. Tan, Yafei & Zhu, Zhaohui, 2022. "The effect of ESG rating events on corporate green innovation in China: The mediating role of financial constraints and managers' environmental awareness," Technology in Society, Elsevier, vol. 68(C).
    19. Hana Kim & Eungdo Kim, 2018. "How an Open Innovation Strategy for Commercialization Affects the Firm Performance of Korean Healthcare IT SMEs," Sustainability, MDPI, vol. 10(7), pages 1-14, July.
    20. Criscuolo, Chiara & Mariagrazia, Squicciarini & Olavi, Lehtoranta, 2010. "R&D, innovation and productivity, and the CIS: sampling, specification and comparability issues," MPRA Paper 39261, University Library of Munich, Germany.
    21. Viroj Jienwatcharamongkhol & Mohammad Hossein Tavassoli, 2012. "Bridging Firm’s Innovation, Productivity and Export: An Analysis using Swedish CIS data," ERSA conference papers ersa12p885, European Regional Science Association.
    22. Giulia Valacchi & Julio Raffo & Alica Daly & David Humphreys, 2019. "Innovation in the Mining Sector and Cycles in Commodity Prices," WIPO Economic Research Working Papers 55, World Intellectual Property Organization - Economics and Statistics Division.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:oup:rseval:v:24:y:2015:i:3:p:242-255.. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Oxford University Press (email available below). General contact details of provider: https://academic.oup.com/rev .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.