IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v95y2013i1d10.1007_s11192-012-0874-7.html
   My bibliography  Save this article

Absolute and specific measures of research group excellence

Author

Listed:
  • O. Mryglod

    (Institute for Condensed Matter Physics of the NAS of Ukraine)

  • R. Kenna

    (Coventry University)

  • Yu. Holovatch

    (Institute for Condensed Matter Physics of the NAS of Ukraine)

  • B. Berche

    (Université de Lorraine, Campus de Nancy)

Abstract

A desirable goal of scientific management is to introduce, if it exists, a simple and reliable way to measure the scientific excellence of publicly funded research institutions and universities to serve as a basis for their ranking and financing. While citation-based indicators and metrics are easily accessible, they are far from being universally accepted as way to automate or inform evaluation processes or to replace evaluations based on peer review. Here we consider absolute measurements of research excellence at an amalgamated, institutional level and specific measures of research excellence as performance per head. Using biology research institutions in the UK as a test case, we examine the correlations between peer review-based and citation-based measures of research excellence on these two scales. We find that citation-based indicators are very highly correlated with peer-evaluated measures of group strength, but are poorly correlated with group quality. Thus, and almost paradoxically, our analysis indicates that citation counts could possibly form a basis for deciding on, how to fund research institutions, but they should not be used as a basis for ranking them in terms of quality.

Suggested Citation

  • O. Mryglod & R. Kenna & Yu. Holovatch & B. Berche, 2013. "Absolute and specific measures of research group excellence," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(1), pages 115-127, April.
    • Handle: RePEc:spr:scient:v:95:y:2013:i:1:d:10.1007_s11192-012-0874-7
    DOI: 10.1007/s11192-012-0874-7
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-012-0874-7
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-012-0874-7?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Michael H. MacRoberts & Barbara R. MacRoberts, 1989. "Problems of citation analysis: A critical review," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 40(5), pages 342-349, September.
    2. Péter Vinkler, 2003. "Relations of relative scientometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 58(3), pages 687-694, November.
    3. Péter Vinkler, 2001. "An attempt for defining some basic categories of scientometrics and classifying the indicators of evaluative scientometrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 50(3), pages 539-544, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Stephan B. Bruns & David I. Stern, 2016. "Research assessment using early citation information," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(2), pages 917-935, August.
    2. R. Álvarez & E. Cahué & J. Clemente-Gallardo & A. Ferrer & D. Íñiguez & X. Mellado & A. Rivero & G. Ruiz & F. Sanz & E. Serrano & A. Tarancón & Y. Vergara, 2015. "Analysis of academic productivity based on Complex Networks," Scientometrics, Springer;Akadémiai Kiadó, vol. 104(3), pages 651-672, September.
    3. Paul Benneworth, 2015. "Between certainty and comprehensiveness in evaluating the societal impact of humanities research," CHEPS Working Papers 201502, University of Twente, Center for Higher Education Policy Studies (CHEPS).
    4. Uddin, Shahadat & Khan, Arif, 2016. "The impact of author-selected keywords on citation counts," Journal of Informetrics, Elsevier, vol. 10(4), pages 1166-1177.
    5. O. Mryglod & R. Kenna & Yu. Holovatch & B. Berche, 2015. "Predicting results of the research excellence framework using departmental h-index: revisited," Scientometrics, Springer;Akadémiai Kiadó, vol. 104(3), pages 1013-1017, September.
    6. Onodera, Natsuo, 2016. "Properties of an index of citation durability of an article," Journal of Informetrics, Elsevier, vol. 10(4), pages 981-1004.
    7. Thelwall, Mike & Wilson, Paul, 2014. "Regression for citation data: An evaluation of different methods," Journal of Informetrics, Elsevier, vol. 8(4), pages 963-971.
    8. Zhongyi Wang & Keying Wang & Jiyue Liu & Jing Huang & Haihua Chen, 2022. "Measuring the innovation of method knowledge elements in scientific literature," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(5), pages 2803-2827, May.
    9. O. Mryglod & Yu. Holovatch & R. Kenna, 2022. "Big fish and small ponds: why the departmental h-index should not be used to rank universities," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(6), pages 3279-3292, June.
    10. Elio Atenógenes Villaseñor & Ricardo Arencibia-Jorge & Humberto Carrillo-Calvet, 2017. "Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher education institutions," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(1), pages 77-104, January.
    11. O. Mryglod & R. Kenna & Yu. Holovatch & B. Berche, 2015. "Predicting results of the Research Excellence Framework using departmental h-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(3), pages 2165-2180, March.
    12. Hana Tomaskova & Martin Kopecky, 2020. "Specialization of Business Process Model and Notation Applications in Medicine—A Review," Data, MDPI, vol. 5(4), pages 1-42, October.
    13. Liyin Zhang & Yuchen Qian & Chao Ma & Jiang Li, 2023. "Continued collaboration shortens the transition period of scientists who move to another institution," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(3), pages 1765-1784, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Li, Yunrong & Radicchi, Filippo & Castellano, Claudio & Ruiz-Castillo, Javier, 2013. "Quantitative evaluation of alternative field normalization procedures," Journal of Informetrics, Elsevier, vol. 7(3), pages 746-755.
    2. Radicchi, Filippo & Castellano, Claudio, 2012. "Testing the fairness of citation indicators for comparison across scientific domains: The case of fractional citation counts," Journal of Informetrics, Elsevier, vol. 6(1), pages 121-130.
    3. O. Mryglod & R. Kenna & Yu. Holovatch & B. Berche, 2013. "Comparison of a citation-based indicator and peer review for absolute and specific measures of research-group excellence," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(3), pages 767-777, December.
    4. Peter Vinkler, 2018. "Structure of the scientific research and science policy," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(2), pages 737-756, February.
    5. Dangzhi Zhao & Elisabeth Logan, 2002. "Citation analysis using scientific publications on the Web as data source: A case study in the XML research area," Scientometrics, Springer;Akadémiai Kiadó, vol. 54(3), pages 449-472, July.
    6. Ponomariov, Branco & Toivanen, Hannes, 2014. "Knowledge flows and bases in emerging economy innovation systems: Brazilian research 2005–2009," Research Policy, Elsevier, vol. 43(3), pages 588-596.
    7. Cristina López-Duarte & Jane F. Maley & Marta M. Vidal-Suárez, 2021. "Main challenges to international student mobility in the European arena," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(11), pages 8957-8980, November.
    8. Javier Ruiz-Castillo, 2013. "The role of statistics in establishing the similarity of citation distributions in a static and a dynamic context," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(1), pages 173-181, July.
    9. Malcolm Wright & J. Scott Armstrong, 2008. "The Ombudsman: Verification of Citations: Fawlty Towers of Knowledge?," Interfaces, INFORMS, vol. 38(2), pages 125-139, April.
    10. T. J. Phelan, 1999. "A compendium of issues for citation analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 45(1), pages 117-136, May.
    11. Peter Vinkler, 2008. "Correlation between the structure of scientific research, scientometric indicators and GDP in EU and non-EU countries," Scientometrics, Springer;Akadémiai Kiadó, vol. 74(2), pages 237-254, February.
    12. Jianhua Hou, 2017. "Exploration into the evolution and historical roots of citation analysis by referenced publication year spectroscopy," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(3), pages 1437-1452, March.
    13. Peter Sjögårde & Fereshteh Didegah, 2022. "The association between topic growth and citation impact of research publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(4), pages 1903-1921, April.
    14. Mehmet Ali Köseoglu & John A. Parnell & Melissa Yan Yee Yick, 2021. "Identifying influential studies and maturity level in intellectual structure of fields: evidence from strategic management," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1271-1309, February.
    15. Рубинштейн Александр Яковлевич, "undated". "Ранжирование Российских Экономических Журналов: Научный Метод Или «Игра В Цыфирь»? [Ran Ranking of Russian Economic Journals: The Scientific Method or “Numbers Game”?]," Working papers a:pru175:ye:2016:1, Institute of Economics.
    16. Masaki Eto, 2013. "Evaluations of context-based co-citation searching," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(2), pages 651-673, February.
    17. Christian Schloegl & Juan Gorraiz & Christoph Bart & Monika Bargmann, 2003. "Evaluating two Austrian university departments: Lessons learned," Scientometrics, Springer;Akadémiai Kiadó, vol. 56(3), pages 289-299, March.
    18. Martin Meyer, 2000. "What is Special about Patent Citations? Differences between Scientific and Patent Citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 49(1), pages 93-123, August.
    19. Petridis, Konstantinos & Malesios, Chrisovalantis & Arabatzis, Garyfallos & Thanassoulis, Emmanuel, 2013. "Efficiency analysis of forestry journals: Suggestions for improving journals’ quality," Journal of Informetrics, Elsevier, vol. 7(2), pages 505-521.
    20. repec:hig:journl:v:6:y:2012:i:1:p:38-58 is not listed on IDEAS
    21. Nelson, Andrew J., 2012. "Putting university research in context: Assessing alternative measures of production and diffusion at Stanford," Research Policy, Elsevier, vol. 41(4), pages 678-691.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:95:y:2013:i:1:d:10.1007_s11192-012-0874-7. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.