IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0168533.html
   My bibliography  Save this article

Beyond Academia – Interrogating Research Impact in the Research Excellence Framework

Author

Listed:
  • Emma Terämä
  • Melanie Smallman
  • Simon J Lock
  • Charlotte Johnson
  • Martin Zaltz Austwick

Abstract

Big changes to the way in which research funding is allocated to UK universities were brought about in the Research Excellence Framework (REF), overseen by the Higher Education Funding Council, England. Replacing the earlier Research Assessment Exercise, the purpose of the REF was to assess the quality and reach of research in UK universities–and allocate funding accordingly. For the first time, this included an assessment of research ‘impact’, accounting for 20% of the funding allocation. In this article we use a text mining technique to investigate the interpretations of impact put forward via impact case studies in the REF process. We find that institutions have developed a diverse interpretation of impact, ranging from commercial applications to public and cultural engagement activities. These interpretations of impact vary from discipline to discipline and between institutions, with more broad-based institutions depicting a greater variety of impacts. Comparing the interpretations with the score given by REF, we found no evidence of one particular interpretation being more highly rewarded than another. Importantly, we also found a positive correlation between impact score and [overall research] quality score, suggesting that impact is not being achieved at the expense of research excellence.

Suggested Citation

  • Emma Terämä & Melanie Smallman & Simon J Lock & Charlotte Johnson & Martin Zaltz Austwick, 2016. "Beyond Academia – Interrogating Research Impact in the Research Excellence Framework," PLOS ONE, Public Library of Science, vol. 11(12), pages 1-18, December.
  • Handle: RePEc:plo:pone00:0168533
    DOI: 10.1371/journal.pone.0168533
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0168533
    Download Restriction: no

    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0168533&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pone.0168533?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Steven Wooding & Thed N Van Leeuwen & Sarah Parks & Shitij Kapur & Jonathan Grant, 2015. "UK Doubles Its “World-Leading” Research in Life Sciences and Medicine in Six Years: Testing the Claim?," PLOS ONE, Public Library of Science, vol. 10(7), pages 1-10, July.
    2. J Lomas, 1990. "Finding Audiences, Changing Beliefs: The Structure of Research Use in Canadian Health Policy," Centre for Health Economics and Policy Analysis Working Paper Series 1990-10, Centre for Health Economics and Policy Analysis (CHEPA), McMaster University, Hamilton, Canada.
    3. David Bholat & Stephen Hans & Pedro Santos & Cheryl Schonhardt-Bailey, 2015. "Text mining for central banks," Handbooks, Centre for Central Banking Studies, Bank of England, number 33, April.
    4. Lisa Klautzer & Stephen Hanney & Edward Nason & Jennifer Rubin & Jonathan Grant & Steven Wooding, 2011. "Assessing policy and practice impacts of social science research: the application of the Payback Framework to assess the Future of Work programme," Research Evaluation, Oxford University Press, vol. 20(3), pages 201-209, September.
    5. Creso M. Sá & Andrew Kretz & Kristjan Sigurdson, 2013. "Accountability, performance assessment, and evaluation: Policy pressures and responses from research councils," Research Evaluation, Oxford University Press, vol. 22(2), pages 105-117, January.
    6. Sarah Bell & Ben Shaw & Annette Boaz, 2011. "Real-world approaches to assessing the impact of environmental research on policy," Research Evaluation, Oxford University Press, vol. 20(3), pages 227-237, September.
    7. Teresa Penfield & Matthew J. Baker & Rosa Scoble & Michael C. Wykes, 2013. "Assessment, evaluations, and definitions of research impact: A review," Research Evaluation, Oxford University Press, vol. 23(1), pages 21-32, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Levine, Kendra K., 2019. "New Metrics for Measuring Academic Research Outside the Ivory Tower," Institute of Transportation Studies, Research Reports, Working Papers, Proceedings qt05s6857s, Institute of Transportation Studies, UC Berkeley.
    2. Han Zheng & L. G. Pee & Dan Zhang, 2021. "Societal impact of research: a text mining study of impact types," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 7397-7417, September.
    3. Bornmann, Lutz & Haunschild, Robin & Adams, Jonathan, 2019. "Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF)," Journal of Informetrics, Elsevier, vol. 13(1), pages 325-340.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lutz Bornmann, 2013. "What is societal impact of research and how can it be assessed? a literature survey," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(2), pages 217-233, February.
    2. Bornmann, Lutz & Haunschild, Robin & Adams, Jonathan, 2019. "Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF)," Journal of Informetrics, Elsevier, vol. 13(1), pages 325-340.
    3. Aaryan Gupta & Vinya Dengre & Hamza Abubakar Kheruwala & Manan Shah, 2020. "Comprehensive review of text-mining applications in finance," Financial Innovation, Springer;Southwestern University of Finance and Economics, vol. 6(1), pages 1-25, December.
    4. Fraccaroli, Nicolò & Giovannini, Alessandro & Jamet, Jean-François & Persson, Eric, 2022. "Ideology and monetary policy. The role of political parties’ stances in the European Central Bank’s parliamentary hearings," European Journal of Political Economy, Elsevier, vol. 74(C).
    5. Leif Anders Thorsrud, 2016. "Nowcasting using news topics Big Data versus big bank," Working Papers No 6/2016, Centre for Applied Macro- and Petroleum economics (CAMP), BI Norwegian Business School.
    6. Antoine Bouveret, 2018. "Cyber Risk for the Financial Sector: A Framework for Quantitative Assessment," IMF Working Papers 2018/143, International Monetary Fund.
    7. Martin Baumgaertner & Johannes Zahner, 2021. "Whatever it takes to understand a central banker - Embedding their words using neural networks," MAGKS Papers on Economics 202130, Philipps-Universität Marburg, Faculty of Business Administration and Economics, Department of Economics (Volkswirtschaftliche Abteilung).
    8. Daniela V. Guío-Martínez & Juan J. Ospina-Tejeiro & Germán A. Muñoz-Bravo & Julián A. Parra-Polanía, 2020. "Descripción de las Minutas e Informes de Política Monetaria a partir de herramientas de Lingüística Computacional," Borradores de Economia 1108, Banco de la Republica de Colombia.
    9. Diane Coyle & Adam Muhtar, 2022. "You’re not speaking my language - policy discontinuity and coordination gaps between the UK’s national economic strategies and its place-based policies," Working Papers 019, The Productivity Institute.
    10. Paola Cerchiello & Giancarlo Nicola, 2018. "Assessing News Contagion in Finance," Econometrics, MDPI, vol. 6(1), pages 1-19, February.
    11. Juha-Pekka Lauronen, 2022. "Tension in Interpretations of the Social Impact of the Social Sciences: Walking a Tightrope Between Divergent Conceptualizations of Research Utilization," SAGE Open, , vol. 12(2), pages 21582440221, April.
    12. Flood, M. D. & Jagadish, H. V. & Raschid, L., 2016. "Big data challenges and opportunities in financial stability monitoring," Financial Stability Review, Banque de France, issue 20, pages 129-142, April.
    13. Youngjoon Lee & Soohyon Kim & Ki Young Park, 2018. "Deciphering Monetary Policy Committee Minutes with Text Mining Approach: A Case of South Korea," Working papers 2018rwp-132, Yonsei University, Yonsei Economics Research Institute.
    14. Johannes Zahner, 2020. "Above, but close to two percent. Evidence on the ECB’s inflation target using text mining," MAGKS Papers on Economics 202046, Philipps-Universität Marburg, Faculty of Business Administration and Economics, Department of Economics (Volkswirtschaftliche Abteilung).
    15. Antón Sarabia Arturo & Bazdresch Santiago & Lelo-de-Larrea Alejandra, 2023. "The Influence of Central Bank's Projections and Economic Narrative on Professional Forecasters' Expectations: Evidence from Mexico," Working Papers 2023-21, Banco de México.
    16. Kumar, Rahul & Deb, Soumya Guha & Mukherjee, Shubhadeep, 2020. "Do words reveal the latent truth? Identifying communication patterns of corporate losers," Journal of Behavioral and Experimental Finance, Elsevier, vol. 26(C).
    17. Stefan Angrick & Naoyuki Yoshino, 2020. "From Window Guidance to Interbank Rates: Tracing the Transition of Monetary Policy in Japan and China," International Journal of Central Banking, International Journal of Central Banking, vol. 16(3), pages 279-316, June.
    18. Ashok Banerjee & Ayush Kanodia & Partha Ray, 2021. "Deciphering Indian inflationary expectations through text mining: an exploratory approach," Indian Economic Review, Springer, vol. 56(1), pages 49-66, June.
    19. Leonardo N. Ferreira, 2021. "Forecasting with VAR-teXt and DFM-teXt Models:exploring the predictive power of central bank communication," Working Papers Series 559, Central Bank of Brazil, Research Department.
    20. Abramo, Giovanni & D'Angelo, Ciriaco Andrea & Grilli, Leonardo, 2021. "The effects of citation-based research evaluation schemes on self-citation behavior," Journal of Informetrics, Elsevier, vol. 15(4).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0168533. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.