IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v16y2022i1s1751157722000062.html
   My bibliography  Save this article

Interpreting the outcomes of research assessments: A geometrical approach

Author

Listed:
  • Cappelletti-Montano, Beniamino
  • Columbu, Silvia
  • Montaldo, Stefano
  • Musio, Monica

Abstract

Research evaluations and comparison of the assessments of academic institutions (scientific areas, departments, universities etc.) are among the major issues in recent years in higher education systems. One method, followed by some national evaluation agencies, is to assess the research quality by the evaluation of a limited number of publications in a way that each publication is rated among n classes. This method produces, for each institution, a distribution of the publications in the n classes. In this paper we introduce a natural geometric way to compare these assessments by introducing an ad hoc distance from the performance of an institution to the best possible achievable assessment. Moreover, to avoid the methodological error of comparing non-homogeneous institutions, we introduce a geometric score based on such a distance. The latter represents the probability that an ideal institution, with the same configuration as the one under evaluation, performs worst. We apply our method, based on the geometric score, to rank, in two specific scientific areas, the Italian universities using the results of the research evaluation VQR 2011–2014.

Suggested Citation

  • Cappelletti-Montano, Beniamino & Columbu, Silvia & Montaldo, Stefano & Musio, Monica, 2022. "Interpreting the outcomes of research assessments: A geometrical approach," Journal of Informetrics, Elsevier, vol. 16(1).
  • Handle: RePEc:eee:infome:v:16:y:2022:i:1:s1751157722000062
    DOI: 10.1016/j.joi.2022.101254
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157722000062
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.joi.2022.101254?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Jim Taylor, 1995. "A Statistical Analysis of the 1992 Research Assessment Exercise," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 158(2), pages 241-261, March.
    2. Christian H. Weiß, 2019. "On some measures of ordinal variation," Journal of Applied Statistics, Taylor & Francis Journals, vol. 46(16), pages 2905-2926, December.
    3. Cristiano Varin & Manuela Cattelan & David Firth, 2016. "Statistical modelling of citation exchange between statistics journals," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 179(1), pages 1-63, January.
    4. Christian H. Weiß, 2020. "Distance-Based Analysis of Ordinal Data and Ordinal Time Series," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 115(531), pages 1189-1200, July.
    5. Giovanni Abramo & Ciriaco Andrea D’Angelo, 2016. "Refrain from adopting the combination of citation and journal metrics to grade publications, as used in the Italian national research assessment exercise (VQR 2011–2014)," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 2053-2065, December.
    6. Korytkowski, Przemyslaw & Kulczycki, Emanuel, 2019. "Publication counting methods for a national research evaluation exercise," Journal of Informetrics, Elsevier, vol. 13(3), pages 804-816.
    7. Christina H. Drew & Kristianna G. Pettibone & Fallis Owen Finch & Douglas Giles & Paul Jordan, 2016. "Automated Research Impact Assessment: a new bibliometrics approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(3), pages 987-1005, March.
    8. Demetrescu, Camil & Lupia, Francesco & Mendicelli, Angelo & Ribichini, Andrea & Scarcello, Francesco & Schaerf, Marco, 2019. "On the Shapley value and its application to the Italian VQR research assessment exercise," Journal of Informetrics, Elsevier, vol. 13(1), pages 87-104.
    9. Fionn Murtagh & Michael Orlov & Boris Mirkin, 2018. "Qualitative Judgement of Research Impact: Domain Taxonomy as a Fundamental Framework for Judgement of the Quality of Research," Journal of Classification, Springer;The Classification Society, vol. 35(1), pages 5-28, April.
    10. Alberto Baccini & Giuseppe De Nicolao, 2016. "Do they agree? Bibliometric evaluation versus informed peer review in the Italian research assessment exercise," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(3), pages 1651-1671, September.
    11. Franceschini, Fiorenzo & Maisano, Domenico, 2017. "Critical remarks on the Italian research assessment exercise VQR 2011–2014," Journal of Informetrics, Elsevier, vol. 11(2), pages 337-357.
    12. Alberto Anfossi & Alberto Ciolfi & Filippo Costa & Giorgio Parisi & Sergio Benedetto, 2016. "Large-scale assessment of research outputs through a weighted combination of bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(2), pages 671-683, May.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Camil Demetrescu & Andrea Ribichini & Marco Schaerf, 2020. "Are Italian research assessment exercises size-biased?," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 533-549, October.
    2. Abramo, Giovanni & D’Angelo, Ciriaco Andrea & Di Costa, Flavia, 2019. "When research assessment exercises leave room for opportunistic behavior by the subjects under evaluation," Journal of Informetrics, Elsevier, vol. 13(3), pages 830-840.
    3. Daniele Checchi & Alberto Ciolfi & Gianni De Fraja & Irene Mazzotta & Stefano Verzillo, 2021. "Have you Read This? An Empirical Comparison of the British REF Peer Review and the Italian VQR Bibliometric Algorithm," Economica, London School of Economics and Political Science, vol. 88(352), pages 1107-1129, October.
    4. Alberto Baccini & Lucio Barabesi & Giuseppe De Nicolao, 2020. "On the agreement between bibliometrics and peer review: Evidence from the Italian research assessment exercises," PLOS ONE, Public Library of Science, vol. 15(11), pages 1-28, November.
    5. Giovanni Abramo & Ciriaco Andrea D’Angelo, 2016. "Refrain from adopting the combination of citation and journal metrics to grade publications, as used in the Italian national research assessment exercise (VQR 2011–2014)," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 2053-2065, December.
    6. Demetrescu, Camil & Lupia, Francesco & Mendicelli, Angelo & Ribichini, Andrea & Scarcello, Francesco & Schaerf, Marco, 2019. "On the Shapley value and its application to the Italian VQR research assessment exercise," Journal of Informetrics, Elsevier, vol. 13(1), pages 87-104.
    7. Silvia Bacci & Bruno Bertaccini & Alessandra Petrucci, 2023. "Insights from the co-authorship network of the Italian academic statisticians," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(8), pages 4269-4303, August.
    8. Giovanni Abramo & Ciriaco Andrea D’Angelo & Emanuela Reale, 2019. "Peer review versus bibliometrics: Which method better predicts the scholarly impact of publications?," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 537-554, October.
    9. Abramo, Giovanni, 2018. "Revisiting the scientometric conceptualization of impact and its measurement," Journal of Informetrics, Elsevier, vol. 12(3), pages 590-597.
    10. Francesco Giovanni Avallone & Alberto Quagli & Paola Ramassa, 2022. "Interdisciplinary research by accounting scholars: An exploratory study," FINANCIAL REPORTING, FrancoAngeli Editore, vol. 2022(2), pages 5-34.
    11. Giulio Marini & Viviana Meschitti, 2018. "The trench warfare of gender discrimination: evidence from academic promotions to full professor in Italy," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(2), pages 989-1006, May.
    12. Camil Demetrescu & Irene Finocchi & Andrea Ribichini & Marco Schaerf, 2022. "On computer science research and its temporal evolution," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4913-4938, August.
    13. Domenico A. Maisano & Luca Mastrogiacomo & Fiorenzo Franceschini, 2020. "Short-term effects of non-competitive funding to single academic researchers," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(3), pages 1261-1280, June.
    14. Mike Thelwall & Kayvan Kousha & Emma Stuart & Meiko Makita & Mahshid Abdoli & Paul Wilson & Jonathan Levitt, 2023. "In which fields are citations indicators of research quality?," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 74(8), pages 941-953, August.
    15. Franceschini, Fiorenzo & Maisano, Domenico, 2017. "Critical remarks on the Italian research assessment exercise VQR 2011–2014," Journal of Informetrics, Elsevier, vol. 11(2), pages 337-357.
    16. Mike Thelwall & Kayvan Kousha, 2016. "Are citations from clinical trials evidence of higher impact research? An analysis of ClinicalTrials.gov," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(2), pages 1341-1351, November.
    17. Abramo, Giovanni & Aksnes, Dag W. & D’Angelo, Ciriaco Andrea, 2020. "Comparison of research performance of Italian and Norwegian professors and universities," Journal of Informetrics, Elsevier, vol. 14(2).
    18. Nadeem Shafique Butt & Ahmad Azam Malik & Muhammad Qaiser Shahbaz, 2021. "Bibliometric Analysis of Statistics Journals Indexed in Web of Science Under Emerging Source Citation Index," SAGE Open, , vol. 11(1), pages 21582440209, January.
    19. Buckle, Robert A. & Creedy, John & Ball, Ashley, 2020. "A Schumpeterian Gale: Using Longitudinal Data to Evaluate Responses to Performance-Based Research Funding Systems," Working Paper Series 9447, Victoria University of Wellington, Chair in Public Finance.
    20. Battiston, Pietro & Sacco, Pier Luigi & Stanca, Luca, 2022. "Cover effects on citations uncovered: Evidence from Nature," Journal of Informetrics, Elsevier, vol. 16(2).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:16:y:2022:i:1:s1751157722000062. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.