IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0156978.html
   My bibliography  Save this article

Measuring the Impact of Research: Lessons from the UK’s Research Excellence Framework 2014

Author

Listed:
  • Gobinda Chowdhury
  • Kushwanth Koya
  • Pete Philipson

Abstract

Impactful academic research plays a stellar role in society, pressing to ask the question of how one measures the impact created by different areas of academic research. Measuring the societal, cultural, economic and scientific impact of research is currently the priority of the National Science Foundation, European Commission and several research funding agencies. The recently concluded United Kingdom’s national research quality exercise, the Research Excellence Framework (REF) 2014, which piloted impact assessment as part of the overall evaluation offers a lens to view how impact of research in different disciplines can be measured. Overall research quality was assessed through quality of outputs, ‘impact’ and research environment. We performed two studies using the REF 2014 as a case study. The first study on 363 Impact Case Studies (ICSs) submitted in 5 research areas (UoAs) reveals that, in general, the impact scores were constructed upon a combination of factors i.e. quantity of quartile-one (Q1) publications, quantity and value of grants/income, number of researchers stated in the ICSs, spin-offs created, discoveries/patents and presentation of esteem data, informing researchers/ academics of the factors to consider in order to achieve a better impact score in research impact assessments. However, there were differences among disciplines in terms of the role played by the factors in achieving their overall scores for the ICSs. The outcome of this study is thus a set of impact indicators, and their relationship with the overall score of impact of research in different disciplines as determined in REF2014, which would in the first instance provide some answers to impact measures that would be useful for researchers in different disciplines. The second study extracts the general themes of impact reported by universities by performing a word frequency analysis in all the ICSs submitted in the five chosen research areas, which were substantially varied owing to their fields.

Suggested Citation

  • Gobinda Chowdhury & Kushwanth Koya & Pete Philipson, 2016. "Measuring the Impact of Research: Lessons from the UK’s Research Excellence Framework 2014," PLOS ONE, Public Library of Science, vol. 11(6), pages 1-15, June.
    • Handle: RePEc:plo:pone00:0156978
    DOI: 10.1371/journal.pone.0156978
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0156978
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0156978&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0156978?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Silvia Ferrari & Francisco Cribari-Neto, 2004. "Beta Regression for Modelling Rates and Proportions," Journal of Applied Statistics, Taylor & Francis Journals, vol. 31(7), pages 799-815.
    2. Smith, Simon & Ward, Vicky & House, Allan, 2011. "‘Impact’ in the proposals for the UK's Research Excellence Framework: Shifting the boundaries of academic autonomy," Research Policy, Elsevier, vol. 40(10), pages 1369-1379.
    3. Lutz Bornmann, 2013. "What is societal impact of research and how can it be assessed? a literature survey," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(2), pages 217-233, February.
    4. Cribari-Neto, Francisco & Zeileis, Achim, 2010. "Beta Regression in R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 34(i02).
    5. Anthony F. J. van Raan, 2005. "Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods," Scientometrics, Springer;Akadémiai Kiadó, vol. 62(1), pages 133-143, January.
    6. Butler, Linda, 2003. "Explaining Australia's increased share of ISI publications--the effects of a funding formula based on publication counts," Research Policy, Elsevier, vol. 32(1), pages 143-155, January.
    7. Simas, Alexandre B. & Barreto-Souza, Wagner & Rocha, Andréa V., 2010. "Improved estimators for a general class of beta regression models," Computational Statistics & Data Analysis, Elsevier, vol. 54(2), pages 348-366, February.
    8. Alberto Pepe & Michael J Kurtz, 2012. "A Measure of Total Research Impact Independent of Time and Discipline," PLOS ONE, Public Library of Science, vol. 7(11), pages 1-7, November.
    9. Diana Hicks & Paul Wouters & Ludo Waltman & Sarah de Rijcke & Ismael Rafols, 2015. "Bibliometrics: The Leiden Manifesto for research metrics," Nature, Nature, vol. 520(7548), pages 429-431, April.
    10. Rafols, Ismael & Leydesdorff, Loet & O’Hare, Alice & Nightingale, Paul & Stirling, Andy, 2012. "How journal rankings can suppress interdisciplinary research: A comparison between Innovation Studies and Business & Management," Research Policy, Elsevier, vol. 41(7), pages 1262-1282.
    11. Saisana, Michaela & d'Hombres, Béatrice & Saltelli, Andrea, 2011. "Rickety numbers: Volatility of university rankings and policy implications," Research Policy, Elsevier, vol. 40(1), pages 165-177, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Groen-Xu, Moqi & Bös, Gregor & Teixeira, Pedro A. & Voigt, Thomas & Knapp, Bernhard, 2023. "Short-term incentives of research evaluations: Evidence from the UK Research Excellence Framework," Research Policy, Elsevier, vol. 52(6).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Berlemann, Michael & Haucap, Justus, 2015. "Which factors drive the decision to opt out of individual research rankings? An empirical study of academic resistance to change," Research Policy, Elsevier, vol. 44(5), pages 1108-1115.
    2. Grün, Bettina & Kosmidis, Ioannis & Zeileis, Achim, 2012. "Extended Beta Regression in R: Shaken, Stirred, Mixed, and Partitioned," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 48(i11).
    3. Mehdi Rhaiem & Nabil Amara, 2020. "Determinants of research efficiency in Canadian business schools: evidence from scholar-level data," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 53-99, October.
    4. Li-Chu Chien, 2013. "Multiple deletion diagnostics in beta regression models," Computational Statistics, Springer, vol. 28(4), pages 1639-1661, August.
    5. Frank A. La Sorte & Alison Johnston & Toby R. Ault, 2021. "Global trends in the frequency and duration of temperature extremes," Climatic Change, Springer, vol. 166(1), pages 1-14, May.
    6. Pablo Mitnik & Sunyoung Baek, 2013. "The Kumaraswamy distribution: median-dispersion re-parameterizations for regression modeling and simulation-based estimation," Statistical Papers, Springer, vol. 54(1), pages 177-192, February.
    7. Chen, Kee Kuo & Chiu, Rong-Her & Chang, Ching-Ter, 2017. "Using beta regression to explore the relationship between service attributes and likelihood of customer retention for the container shipping industry," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 104(C), pages 1-16.
    8. Rafols, Ismael & Stirling, Andy, 2020. "Designing indicators for opening up evaluation. Insights from research assessment," SocArXiv h2fxp, Center for Open Science.
    9. Yiyun Shou & Michael Smithson, 2015. "Evaluating Predictors of Dispersion: A Comparison of Dominance Analysis and Bayesian Model Averaging," Psychometrika, Springer;The Psychometric Society, vol. 80(1), pages 236-256, March.
    10. Oscar Melo & Carlos Melo & Jorge Mateu, 2015. "Distance-based beta regression for prediction of mutual funds," AStA Advances in Statistical Analysis, Springer;German Statistical Society, vol. 99(1), pages 83-106, January.
    11. Souza, Tatiene C. & Cribari–Neto, Francisco, 2018. "Intelligence and religious disbelief in the United States," Intelligence, Elsevier, vol. 68(C), pages 48-57.
    12. Cepeda-Cuervo Edilberto & Garrido Liliana, 2015. "Bayesian beta regression models with joint mean and dispersion modeling," Monte Carlo Methods and Applications, De Gruyter, vol. 21(1), pages 49-58, March.
    13. Artur J. Lemonte & Germán Moreno-Arenas, 2020. "On a heavy-tailed parametric quantile regression model for limited range response variables," Computational Statistics, Springer, vol. 35(1), pages 379-398, March.
    14. Edouard Civel & Nathaly Cruz-Garcia, 2018. "Green, yellow or red lemons? Framed field experiment on houses energy labels perception," Working Papers hal-04141696, HAL.
    15. Edouard Civel & Nathaly Cruz, 2018. "Green, yellow or red lemons? Artefactual field experiment on houses energy labels perception," Working Papers 1809, Chaire Economie du climat.
    16. Collier, Benjamin, 2013. "Exclusive finance: How unmanaged systemic risk continues to limit financial services for the poor in a booming sector," 2013 Annual Meeting, August 4-6, 2013, Washington, D.C. 150433, Agricultural and Applied Economics Association.
    17. Logan C Brooks & David C Farrow & Sangwon Hyun & Ryan J Tibshirani & Roni Rosenfeld, 2015. "Flexible Modeling of Epidemics with an Empirical Bayes Framework," PLOS Computational Biology, Public Library of Science, vol. 11(8), pages 1-18, August.
    18. Domingo Docampo & Lawrence Cram, 2019. "Highly cited researchers: a moving target," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(3), pages 1011-1025, March.
    19. Paulus, Anne & Hagemann, Nina & Baaken, Marieke C. & Roilo, Stephanie & Alarcón-Segura, Viviana & Cord, Anna F. & Beckmann, Michael, 2022. "Landscape context and farm characteristics are key to farmers' adoption of agri-environmental schemes," Land Use Policy, Elsevier, vol. 121(C).
    20. Alexander Kalgin & Olga Kalgina & Anna Lebedeva, 2019. "Publication Metrics as a Tool for Measuring Research Productivity and Their Relation to Motivation," Voprosy obrazovaniya / Educational Studies Moscow, National Research University Higher School of Economics, issue 1, pages 44-86.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0156978. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.