IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0179722.html
   My bibliography  Save this article

Metric-based vs peer-reviewed evaluation of a research output: Lesson learnt from UK’s national research assessment exercise

Author

Listed:
  • Kushwanth Koya
  • Gobinda Chowdhury

Abstract

Purpose: There is a general inquisition regarding the monetary value of a research output, as a substantial amount of funding in modern academia is essentially awarded to good research presented in the form of journal articles, conferences papers, performances, compositions, exhibitions, books and book chapters etc., which, eventually leads to another question if the value varies across different disciplines. Answers to these questions will not only assist academics and researchers, but will also help higher education institutions (HEIs) make informed decisions in their administrative and research policies. Design and methodology: To examine both the questions, we applied the United Kingdom’s recently concluded national research assessment exercise known as the Research Excellence Framework (REF) 2014 as a case study. All the data for this study is sourced from the openly available publications which arose from the digital repositories of REF’s results and HEFCE’s funding allocations. Findings: A world leading output earns between £7504 and £14,639 per year within the REF cycle, whereas an internationally excellent output earns between £1876 and £3659, varying according to their area of research. Secondly, an investigation into the impact rating of 25315 journal articles submitted in five areas of research by UK HEIs and their awarded funding revealed a linear relationship between the percentage of quartile-one journal publications and percentage of 4* outputs in Clinical Medicine, Physics and Psychology/Psychiatry/Neuroscience UoAs, and no relationship was found in the Classics and Anthropology/Development Studies UoAs, due to the fact that most publications in the latter two disciplines are not journal articles. Practical implications: The findings provide an indication of the monetary value of a research output, from the perspectives of government funding for research, and also what makes a good output, i.e. whether a relationship exists between good quality output and the source of its publication. The findings may also influence future REF submission strategies in HEIs and ascertain that the impact rating of the journals is not necessarily a reflection of the quality of research in every discipline, and this may have a significant influence on the future of scholarly communications in general. Originality: According to the author’s knowledge, this is the first time an investigation has estimated the monetary value of a good research output.

Suggested Citation

  • Kushwanth Koya & Gobinda Chowdhury, 2017. "Metric-based vs peer-reviewed evaluation of a research output: Lesson learnt from UK’s national research assessment exercise," PLOS ONE, Public Library of Science, vol. 12(7), pages 1-18, July.
    • Handle: RePEc:plo:pone00:0179722
    DOI: 10.1371/journal.pone.0179722
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0179722
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0179722&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0179722?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Giovanni Abramo & Ciriaco Andrea D’Angelo, 2011. "Evaluating research: from informed peer review to bibliometrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 87(3), pages 499-514, June.
    2. Lutz Bornmann, 2013. "What is societal impact of research and how can it be assessed? a literature survey," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(2), pages 217-233, February.
    3. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    4. Henk F. Moed & Gali Halevi, 2015. "Multidimensional assessment of scholarly research impact," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(10), pages 1988-2002, October.
    5. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Caprasecca, Alessandro, 2009. "Allocative efficiency in public research funding: Can bibliometrics help?," Research Policy, Elsevier, vol. 38(1), pages 206-215, February.
    6. Jerome K. Vanclay, 2012. "Impact factor: outdated artefact or stepping-stone to journal certification?," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(2), pages 211-238, August.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    2. Nathalie Taverdet-Popiolek, 2022. "Economic Footprint of a Large French Research and Technology Organisation in Europe: Deciphering a Simplified Model and Appraising the Results," Journal of the Knowledge Economy, Springer;Portland International Center for Management of Engineering and Technology (PICMET), vol. 13(1), pages 44-69, March.
    3. Eliseo Reategui & Alause Pires & Michel Carniato & Sergio Roberto Kieling Franco, 2020. "Evaluation of Brazilian research output in education: confronting international and national contexts," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 427-444, October.
    4. Eugenio Petrovich, 2022. "Bibliometrics in Press. Representations and uses of bibliometric indicators in the Italian daily newspapers," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(5), pages 2195-2233, May.
    5. Mingkun Wei, 2020. "Research on impact evaluation of open access journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(2), pages 1027-1049, February.
    6. Parreiras, R.O. & Kokshenev, I. & Carvalho, M.O.M. & Willer, A.C.M. & Dellezzopolles, C.F. & Nacif, D.B. & Santana, J.A., 2019. "A flexible multicriteria decision-making methodology to support the strategic management of Science, Technology and Innovation research funding programs," European Journal of Operational Research, Elsevier, vol. 272(2), pages 725-739.
    7. Han Zheng & L. G. Pee & Dan Zhang, 2021. "Societal impact of research: a text mining study of impact types," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 7397-7417, September.
    8. Bornmann, Lutz & Haunschild, Robin & Adams, Jonathan, 2019. "Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF)," Journal of Informetrics, Elsevier, vol. 13(1), pages 325-340.
    9. Migheli, Matteo & Zotti, Roberto, 2020. "The strange case of the Matthew effect and beauty contests: Research evaluation and specialisation in Italian universities," Socio-Economic Planning Sciences, Elsevier, vol. 71(C).
    10. Jianhua Hou & Hao Li & Yang Zhang, 2023. "Altmetrics-based sleeping beauties: necessity or just a supplement?," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(10), pages 5477-5506, October.
    11. Siluo Yang & Xin Xing & Dietmar Wolfram, 2018. "Difference in the impact of open-access papers published by China and the USA," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(2), pages 1017-1037, May.
    12. Bornmann, Lutz & Haunschild, Robin, 2016. "Citation score normalized by cited references (CSNCR): The introduction of a new citation impact indicator," Journal of Informetrics, Elsevier, vol. 10(3), pages 875-887.
    13. Kayvan Kousha & Mike Thelwall, 2024. "Factors associating with or predicting more cited or higher quality journal articles: An Annual Review of Information Science and Technology (ARIST) paper," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 75(3), pages 215-244, March.
    14. Giovanni Abramo & Ciriaco Andrea D’Angelo & Flavia Di Costa, 2011. "National research assessment exercises: a comparison of peer review and bibliometrics rankings," Scientometrics, Springer;Akadémiai Kiadó, vol. 89(3), pages 929-941, December.
    15. Hsing Kenneth Cheng & D. Daniel Sokol & Xinyu Zang, 2024. "The rise of empirical online platform research in the new millennium," Journal of Economics & Management Strategy, Wiley Blackwell, vol. 33(2), pages 416-451, March.
    16. Alessandro Margherita & Gianluca Elia & Claudio Petti, 2022. "What Is Quality in Research? Building a Framework of Design, Process and Impact Attributes and Evaluation Perspectives," Sustainability, MDPI, vol. 14(5), pages 1-18, March.
    17. Adam Dinsmore & Liz Allen & Kevin Dolby, 2014. "Alternative Perspectives on Impact: The Potential of ALMs and Altmetrics to Inform Funders about Research Impact," PLOS Biology, Public Library of Science, vol. 12(11), pages 1-4, November.
    18. Matteo Pedrini & Valentina Langella & Mario Alberto Battaglia & Paola Zaratin, 2018. "Assessing the health research’s social impact: a systematic review," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 1227-1250, March.
    19. Zhaoping Yan & Kaiyu Fan, 2024. "An integrated indicator for evaluating scientific papers: considering academic impact and novelty," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(11), pages 6909-6929, November.
    20. Hou, Jianhua & Yang, Xiucai, 2020. "Social media-based sleeping beauties: Defining, identifying and features," Journal of Informetrics, Elsevier, vol. 14(2).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0179722. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.