IDEAS home Printed from https://ideas.repec.org/a/oup/rseval/v32y2023i2p273-285..html
   My bibliography  Save this article

How unpredictable is research impact? Evidence from the UK’s Research Excellence Framework

Author

Listed:
  • Ohid Yaqub
  • Dmitry Malkov
  • Josh Siepel

Abstract

Although ex post evaluation of impact is increasingly common, the extent to which research impacts emerge largely as anticipated by researchers, or as the result of serendipitous and unpredictable processes, is not well understood. In this article, we explore whether predictions of impact made at the funding stage align with realized impact, using data from the UK’s Research Excellence Framework (REF). We exploit REF impact cases traced back to research funding applications, as a dataset of 2,194 case–grant pairs, to compare impact topics with funder remits. For 209 of those pairs, we directly compare their descriptions of ex ante and ex post impact. We find that impact claims in these case–grant pairs are often congruent with each other, with 76% showing alignment between anticipated impact at funding stage and the eventual claimed impact in the REF. Co-production of research, often perceived as a model for impactful research, was a feature of just over half of our cases. Our results show that, contrary to other preliminary studies of the REF, impact appears to be broadly predictable, although unpredictability remains important. We suggest that co-production is a reasonably good mechanism for addressing the balance of predictable and unpredictable impact outcomes.

Suggested Citation

  • Ohid Yaqub & Dmitry Malkov & Josh Siepel, 2023. "How unpredictable is research impact? Evidence from the UK’s Research Excellence Framework," Research Evaluation, Oxford University Press, vol. 32(2), pages 273-285.
  • Handle: RePEc:oup:rseval:v:32:y:2023:i:2:p:273-285.
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1093/reseval/rvad019
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Power, Michael, 2018. "Creativity, Risk and the Research Impact Agenda in the United Kingdom," European Review, Cambridge University Press, vol. 26(S1), pages 25-34, February.
    2. Maryam Razmgir & Sirous Panahi & Leila Ghalichi & Seyed Ali Javad Mousavi & Shahram Sedghi, 2021. "Exploring research impact models: A systematic scoping review [AsseCssment of the Impact of a Clinical and Health Services Research Call in Catalonia]," Research Evaluation, Oxford University Press, vol. 30(4), pages 443-457.
    3. Reed, M.S. & Ferré, M. & Martin-Ortega, J. & Blanche, R. & Lawford-Rolfe, R. & Dallimer, M. & Holden, J., 2021. "Evaluating impact from research: A methodological framework," Research Policy, Elsevier, vol. 50(4).
    4. Joly, P.B. & Gaunand, A. & Colinet, L. & Larédo, P. & Lemarié, S. & Matt, M., 2015. "ASIRPA: a comprehensive theory-based approach to assessing the societal impacts of a research organization," Working Papers 2015-04, Grenoble Applied Economics Laboratory (GAEL).
    5. Valeria Arza & Mariela Carattoli, 2017. "Personal ties in university-industry linkages: a case-study from Argentina," The Journal of Technology Transfer, Springer, vol. 42(4), pages 814-840, August.
    6. Sandra Nutley, 2010. "Debate: Are we all co-producers of research now?," Public Money & Management, Taylor & Francis Journals, vol. 30(5), pages 263-265, September.
    7. Fiona Armstrong & Adrian Alsop, 2010. "Debate: Co-production can contribute to research impact in the social sciences," Public Money & Management, Taylor & Francis Journals, vol. 30(4), pages 208-210, July.
    8. Gunnar Sivertsen & Ingeborg Meijer, 2020. "Normal versus extraordinary societal impact: how to understand, evaluate, and improve research activities in their relations to society?," Research Evaluation, Oxford University Press, vol. 29(1), pages 66-70.
    9. Power, Michael, 2018. "Creativity, risk and the research impact agenda in the United Kingdom," LSE Research Online Documents on Economics 87731, London School of Economics and Political Science, LSE Library.
    10. Yaqub, Ohid, 2018. "Serendipity: Towards a taxonomy and a theory," Research Policy, Elsevier, vol. 47(1), pages 169-179.
    11. Smith, Simon & Ward, Vicky & House, Allan, 2011. "‘Impact’ in the proposals for the UK's Research Excellence Framework: Shifting the boundaries of academic autonomy," Research Policy, Elsevier, vol. 40(10), pages 1369-1379.
    12. Hannah Durrant & Eleanor MacKillop, 2022. "University policy engagement bodies in the UK and the variable meanings of and approaches to impact," Research Evaluation, Oxford University Press, vol. 31(3), pages 372-384.
    13. Nicola Grassano & Daniele Rotolo & Joshua Hutton & Frédérique Lang & Michael M. Hopkins, 2017. "Funding Data from Publication Acknowledgments: Coverage, Uses, and Limitations," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(4), pages 999-1017, April.
    14. Gary Hickey & Tessa Richards & Jeff Sheehy, 2018. "Co-production from proposal to paper," Nature, Nature, vol. 562(7725), pages 29-31, October.
    15. Hicks, Diana, 2012. "Performance-based university research funding systems," Research Policy, Elsevier, vol. 41(2), pages 251-261.
    16. Steven Hill, 2016. "Assessing (for) impact: future assessment of the societal impact of research," Palgrave Communications, Palgrave Macmillan, vol. 2(1), pages 1-7, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ohid Yaqub & Dmitry Malkov & Josh Siepel, 2024. "How unpredictable is research impact? Evidence from the UK’s Research Excellence Framework," Research Evaluation, Oxford University Press, vol. 32(2), pages 273-285.
    2. Ferreira, Paula & Rocha, Ana & Araujo, Madalena & Afonso, Joao L. & Antunes, Carlos Henggeler & Lopes, Marta A.R. & Osório, Gerardo J. & Catalão, João P.S. & Lopes, João Peças, 2023. "Assessing the societal impact of smart grids: Outcomes of a collaborative research project," Technology in Society, Elsevier, vol. 72(C).
    3. Daniele Rotolo & Michael Hopkins & Nicola Grassano, 2023. "Do funding sources complement or substitute? Examining the impact of cancer research publications," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 74(1), pages 50-66, January.
    4. Dotti, Nicola Francesco & Walczyk, Julia, 2022. "What is the societal impact of university research? A policy-oriented review to map approaches, identify monitoring methods and success factors," Evaluation and Program Planning, Elsevier, vol. 95(C).
    5. Ciarli, Tommaso & Ràfols, Ismael, 2019. "The relation between research priorities and societal demands: The case of rice," Research Policy, Elsevier, vol. 48(4), pages 949-967.
    6. Melinda Craike & Bojana Klepac & Amy Mowle & Therese Riley, 2023. "Theory of systems change: An initial, middle-range theory of public health research impact," Research Evaluation, Oxford University Press, vol. 32(3), pages 603-621.
    7. Rebora, Gianfranco & Turri, Matteo, 2013. "The UK and Italian research assessment exercises face to face," Research Policy, Elsevier, vol. 42(9), pages 1657-1666.
    8. Giliberto Capano & Benedetto Lepori, 2024. "Designing policies that could work: understanding the interaction between policy design spaces and organizational responses in public sector," Policy Sciences, Springer;Society of Policy Sciences, vol. 57(1), pages 53-82, March.
    9. Degl’Innocenti, Marta & Matousek, Roman & Tzeremes, Nickolaos G., 2019. "The interconnections of academic research and universities’ “third mission”: Evidence from the UK," Research Policy, Elsevier, vol. 48(9), pages 1-1.
    10. Helka Kalliomäki & Sampo Ruoppila & Jenni Airaksinen, 2021. "It takes two to tango: Examining productive interactions in urban research collaboration [Generating Research Questions through Problematization]," Research Evaluation, Oxford University Press, vol. 30(4), pages 529-539.
    11. Molly Morgan Jones & Catriona Manville & Joanna Chataway, 2022. "Learning from the UK’s research impact assessment exercise: a case study of a retrospective impact assessment exercise and questions for the future," The Journal of Technology Transfer, Springer, vol. 47(3), pages 722-746, June.
    12. Richard McManus & Karen Mumford & Cristina Sechel, 2022. "Measuring research excellence amongst economics lecturers in the UK," Bulletin of Economic Research, Wiley Blackwell, vol. 74(2), pages 386-404, April.
    13. Salter, Ammon & Salandra, Rossella & Walker, James, 2017. "Exploring preferences for impact versus publications among UK business and management academics," Research Policy, Elsevier, vol. 46(10), pages 1769-1782.
    14. Alis Oancea, 2019. "Research governance and the future(s) of research assessment," Palgrave Communications, Palgrave Macmillan, vol. 5(1), pages 1-12, December.
    15. Richard McManus & Karen Mumford & Cristina Sechel, 2017. "The Selection of Economics Lecturers into the 2014 UK Research Excellence Framework Exercise: Outputs and Gender," Discussion Papers 17/16, Department of Economics, University of York.
    16. Owen, Richard & Pansera, Mario & Macnaghten, Phil & Randles, Sally, 2021. "Organisational institutionalisation of responsible innovation," Research Policy, Elsevier, vol. 50(1).
    17. Jorrit P Smit & Laurens K Hessels, 2021. "The production of scientific and societal value in research evaluation: a review of societal impact assessment methods [Systems Thinking, Knowledge and Action: Towards Better Models and Methods]," Research Evaluation, Oxford University Press, vol. 30(3), pages 323-335.
    18. Berlemann, Michael & Haucap, Justus, 2015. "Which factors drive the decision to opt out of individual research rankings? An empirical study of academic resistance to change," Research Policy, Elsevier, vol. 44(5), pages 1108-1115.
    19. Melinda Craike & Bojana Klepac & Amy Mowle & Therese Riley, 2024. "Theory of systems change: An initial, middle-range theory of public health research impact," Research Evaluation, Oxford University Press, vol. 32(3), pages 603-621.
    20. Paget, Nicolas & Le Gal, Pierre-Yves & Goulet, Frédéric, 2024. "Motivations and challenges of intrapreneurship in research organizations. The case of decision support systems in agricultural research for development," Technovation, Elsevier, vol. 130(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:oup:rseval:v:32:y:2023:i:2:p:273-285.. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Oxford University Press (email available below). General contact details of provider: https://academic.oup.com/rev .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.