IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v13y2019i3p830-840.html
   My bibliography  Save this article

When research assessment exercises leave room for opportunistic behavior by the subjects under evaluation

Author

Listed:
  • Abramo, Giovanni
  • D’Angelo, Ciriaco Andrea
  • Di Costa, Flavia

Abstract

This study inserts in the stream of research on the perverse effects that PBRF systems can induce in the subjects evaluated. The authors’ opinion is that more often than not, it is the doubtful scientific basis of the evaluation criteria that leave room for opportunistic behaviors. The work examines the 2004–2010 Italian national research assessment (VQR) to test the lack of possible opportunistic behavior by universities in order to limit the penalization of their performance (and funding) due to the presence of scientifically unproductive professors in faculty. In particular, institutions may have favored “gift authorship” practices. The analysis thus focuses on the output of professors who were unproductive in the VQR publication window, but became productive (“new productives”) in the following five years. A number of universities show a higher than average share of publications by new productives that are in co-authorship exclusively with colleagues from the same university. Although this might be thought to reflect opportunistic behavior by universities, the empirical evidence does not support this assumption.

Suggested Citation

  • Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Di Costa, Flavia, 2019. "When research assessment exercises leave room for opportunistic behavior by the subjects under evaluation," Journal of Informetrics, Elsevier, vol. 13(3), pages 830-840.
    • Handle: RePEc:eee:infome:v:13:y:2019:i:3:p:830-840
    DOI: 10.1016/j.joi.2019.07.006
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157719300896
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2019.07.006?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Giovanni Abramo & Ciriaco Andrea D’Angelo, 2011. "Evaluating research: from informed peer review to bibliometrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 87(3), pages 499-514, June.
    2. Jesper W. Schneider & Kaare Aagaard & Carter W. Bloch, 2016. "What happens when national research funding is linked to differentiated publication counts? A comparison of the Australian and Norwegian publication-based funding models," Research Evaluation, Oxford University Press, vol. 25(3), pages 244-256.
    3. Ciriaco Andrea D'Angelo & Cristiano Giuffrida & Giovanni Abramo, 2011. "A heuristic approach to author name disambiguation in bibliometrics databases for large-scale research assessments," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 62(2), pages 257-269, February.
    4. Giovanni Abramo & Ciriaco Andrea D’Angelo, 2016. "Refrain from adopting the combination of citation and journal metrics to grade publications, as used in the Italian national research assessment exercise (VQR 2011–2014)," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 2053-2065, December.
    5. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Di Costa, 2014. "Inefficiency in selecting products for submission to national research assessment exercises," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(3), pages 2069-2086, March.
    6. Abramo Giovanni, 2017. "Bibliometric Evaluation of Research Performance: Where Do We Stand?," Voprosy obrazovaniya / Educational Studies Moscow, National Research University Higher School of Economics, issue 1, pages 112-127.
    7. Ciriaco Andrea D'Angelo & Cristiano Giuffrida & Giovanni Abramo, 2011. "A heuristic approach to author name disambiguation in bibliometrics databases for large‐scale research assessments," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 62(2), pages 257-269, February.
    8. Butler, Linda, 2003. "Explaining Australia's increased share of ISI publications--the effects of a funding formula based on publication counts," Research Policy, Elsevier, vol. 32(1), pages 143-155, January.
    9. Peter Woelert & Lachlan McKenzie, 2018. "Follow the money? How Australian universities replicate national performance-based funding mechanisms," Research Evaluation, Oxford University Press, vol. 27(3), pages 184-195.
    10. Linda Butler, 2003. "Modifying publication practices in response to funding formulas," Research Evaluation, Oxford University Press, vol. 12(1), pages 39-46, April.
    11. Kaare Aagaard, 2015. "How incentives trickle down: Local use of a national bibliometric indicator system," Science and Public Policy, Oxford University Press, vol. 42(5), pages 725-737.
    12. Giovanni Abramo & Ciriaco Andrea D’Angelo & Marco Solazzi, 2011. "The relationship between scientists’ research performance and the degree of internationalization of their research," Scientometrics, Springer;Akadémiai Kiadó, vol. 86(3), pages 629-643, March.
    13. Koen Jonkers & Thomas Zacharewicz, 2016. "Research Performance Based Funding Systems: a Comparative Assessment," JRC Research Reports JRC101043, Joint Research Centre.
    14. Giovanni Abramo & Ciriaco Andrea D'Angelo, 2015. "The VQR, Italy's second national research assessment: Methodological failures and ranking distortions," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(11), pages 2202-2214, November.
    15. van den Besselaar, Peter & Heyman, Ulf & Sandström, Ulf, 2017. "Perverse effects of output-based research funding? Butler’s Australian case revisited," Journal of Informetrics, Elsevier, vol. 11(3), pages 905-918.
    16. Rafols, Ismael & Leydesdorff, Loet & O’Hare, Alice & Nightingale, Paul & Stirling, Andy, 2012. "How journal rankings can suppress interdisciplinary research: A comparison between Innovation Studies and Business & Management," Research Policy, Elsevier, vol. 41(7), pages 1262-1282.
    17. Alberto Baccini & Giuseppe De Nicolao, 2016. "Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(3), pages 1651-1671, September.
    18. Giovanni Abramo & Ciriaco Andrea D’Angelo & Francesco Rosati, 2015. "The determinants of academic career advancement: Evidence from Italy," Science and Public Policy, Oxford University Press, vol. 42(6), pages 761-774.
    19. He, Zi-Lin & Geng, Xue-Song & Campbell-Hunt, Colin, 2009. "Research collaboration and research output: A longitudinal study of 65 biomedical scientists in a New Zealand university," Research Policy, Elsevier, vol. 38(2), pages 306-317, March.
    20. Franceschini, Fiorenzo & Maisano, Domenico, 2017. "Critical remarks on the Italian research assessment exercise VQR 2011–2014," Journal of Informetrics, Elsevier, vol. 11(2), pages 337-357.
    21. Hicks, Diana, 2012. "Performance-based university research funding systems," Research Policy, Elsevier, vol. 41(2), pages 251-261.
    22. Giovanni Abramo & Ciriaco Andrea D’Angelo & Francesco Rosati, 2014. "Career advancement and scientific performance in universities," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(2), pages 891-907, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yaşar Tonta & Müge Akbulut, 2020. "Does monetary support increase citation impact of scholarly papers?," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1617-1641, November.
    2. Gregorio González-Alcaide, 2021. "Bibliometric studies outside the information science and library science field: uncontainable or uncontrollable?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(8), pages 6837-6870, August.
    3. Giovanni Abramo & Ciriaco Andrea D’Angelo & Myroslava Hladchenko, 2023. "Assessing the effects of publication requirements for professorship on research performance and publishing behaviour of Ukrainian academics," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4589-4609, August.
    4. Robert A. Buckle & John Creedy, 2022. "Methods to evaluate institutional responses to performance‐based research funding systems," Australian Economic Papers, Wiley Blackwell, vol. 61(3), pages 615-634, September.
    5. Lin Jiang & Biyun Chen, 2023. "A Two-Stage Bilateral Matching Study of Teams-Technology Talents in New R&D Institutions Based on Prospect Theory," Sustainability, MDPI, vol. 15(4), pages 1-27, February.
    6. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Grilli, Leonardo, 2021. "The effects of citation-based research evaluation schemes on self-citation behavior," Journal of Informetrics, Elsevier, vol. 15(4).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Grilli, Leonardo, 2021. "The effects of citation-based research evaluation schemes on self-citation behavior," Journal of Informetrics, Elsevier, vol. 15(4).
    2. Robert A. Buckle & John Creedy, 2022. "Methods to evaluate institutional responses to performance‐based research funding systems," Australian Economic Papers, Wiley Blackwell, vol. 61(3), pages 615-634, September.
    3. Linda Sīle & Raf Vanderstraeten, 2019. "Measuring changes in publication patterns in a context of performance-based research funding systems: the case of educational research in the University of Gothenburg (2005–2014)," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(1), pages 71-91, January.
    4. Buckle, Robert A. & Creedy, John & Ball, Ashley, 2020. "A Schumpeterian Gale: Using Longitudinal Data to Evaluate Responses to Performance-Based Research Funding Systems," Working Paper Series 21104, Victoria University of Wellington, Chair in Public Finance.
    5. Giovanni Abramo & Ciriaco Andrea D’Angelo & Myroslava Hladchenko, 2023. "Assessing the effects of publication requirements for professorship on research performance and publishing behaviour of Ukrainian academics," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4589-4609, August.
    6. Domenico A. Maisano & Luca Mastrogiacomo & Fiorenzo Franceschini, 2020. "Short-term effects of non-competitive funding to single academic researchers," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(3), pages 1261-1280, June.
    7. Przemysław Korytkowski & Emanuel Kulczycki, 2019. "Examining how country-level science policy shapes publication patterns: the case of Poland," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(3), pages 1519-1543, June.
    8. Migheli, Matteo & Zotti, Roberto, 2020. "The strange case of the Matthew effect and beauty contests: Research evaluation and specialisation in Italian universities," Socio-Economic Planning Sciences, Elsevier, vol. 71(C).
    9. van den Besselaar, Peter & Heyman, Ulf & Sandström, Ulf, 2017. "Perverse effects of output-based research funding? Butler’s Australian case revisited," Journal of Informetrics, Elsevier, vol. 11(3), pages 905-918.
    10. Dzieżyc, Maciej & Kazienko, Przemysław, 2022. "Effectiveness of research grants funded by European Research Council and Polish National Science Centre," Journal of Informetrics, Elsevier, vol. 16(1).
    11. Francesco Giovanni Avallone & Alberto Quagli & Paola Ramassa, 2022. "Interdisciplinary research by accounting scholars: An exploratory study," FINANCIAL REPORTING, FrancoAngeli Editore, vol. 2022(2), pages 5-34.
    12. Thomas Zacharewicz & Benedetto Lepori & Emanuela Reale & Koen Jonkers, 2019. "Performance-based research funding in EU Member States—a comparative assessment," Science and Public Policy, Oxford University Press, vol. 46(1), pages 105-115.
    13. Sergey Kolesnikov & Eriko Fukumoto & Barry Bozeman, 2018. "Researchers’ risk-smoothing publication strategies: Is productivity the enemy of impact?," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 1995-2017, September.
    14. Daniella Bayle Deutz & Thea Marie Drachen & Dorte Drongstrup & Niels Opstrup & Charlotte Wien, 2021. "Quantitative quality: a study on how performance-based measures may change the publication patterns of Danish researchers," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(4), pages 3303-3320, April.
    15. Buckle, Robert A. & Creedy, John & Ball, Ashley, 2020. "A Schumpeterian Gale: Using Longitudinal Data to Evaluate Responses to Performance-Based Research Funding Systems," Working Paper Series 9447, Victoria University of Wellington, Chair in Public Finance.
    16. Giovanni Abramo & Ciriaco Andrea D’Angelo, 2016. "Refrain from adopting the combination of citation and journal metrics to grade publications, as used in the Italian national research assessment exercise (VQR 2011–2014)," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 2053-2065, December.
    17. Christian Schneijderberg & Nicolai Götze & Lars Müller, 2022. "A study of 25 years of publication outputs in the German academic profession," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(1), pages 1-28, January.
    18. Giulio Marini & Viviana Meschitti, 2018. "The trench warfare of gender discrimination: evidence from academic promotions to full professor in Italy," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(2), pages 989-1006, May.
    19. Groen-Xu, Moqi & Bös, Gregor & Teixeira, Pedro A. & Voigt, Thomas & Knapp, Bernhard, 2023. "Short-term incentives of research evaluations: Evidence from the UK Research Excellence Framework," Research Policy, Elsevier, vol. 52(6).
    20. Lutz Bornmann & Werner Marx, 2014. "How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 487-509, January.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:13:y:2019:i:3:p:830-840. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.