IDEAS home Printed from https://ideas.repec.org/a/oup/scippl/v48y2021i6p763-775..html
   My bibliography  Save this article

Does the inclusion of non-academic reviewers make any difference for grant impact panels?
[Understanding the Long Term Impact of the Framework Programme, European Policy Evaluation Consortium (EPEC) Report]

Author

Listed:
  • Junwen Luo
  • Lai Ma
  • Kalpana Shankar

Abstract

Broader impact of scientific research beyond academia has become increasingly important in research evaluation. To evaluate broader impact of research proposals, some funding agencies compose mixed panels that include peer experts and non-academic stakeholders. Whether and how non-academic reviewers bring any difference to panel discussions has been understudied. We analysed 164 review reports (2014–6) from the Investigators Programme (funding Science, Technology, Engineering, and Mathematics fields) at Science Foundation Ireland, where two types of panels, with and without non-academics, were composed for impact assessments. We find that the mixed panel reviews were longer and touched upon broader and more concrete impact topics. Also, mixed panels commented on causality and attribution of impact towards characteristics of applicants and research process more than scientific excellence. A survey of the same reviewer pool supplements our understanding of the pros and cons of the inclusion of non-academic reviewers. We discuss some policy recommendations for funding agencies to organise review panels.

Suggested Citation

  • Junwen Luo & Lai Ma & Kalpana Shankar, 2021. "Does the inclusion of non-academic reviewers make any difference for grant impact panels? [Understanding the Long Term Impact of the Framework Programme, European Policy Evaluation Consortium (EPEC," Science and Public Policy, Oxford University Press, vol. 48(6), pages 763-775.
    • Handle: RePEc:oup:scippl:v:48:y:2021:i:6:p:763-775.
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1093/scipol/scab046
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Paul Nightingale & Alister Scott, 2007. "Peer review and the relevance gap: Ten suggestions for policy-makers," Science and Public Policy, Oxford University Press, vol. 34(8), pages 543-553, October.
    2. Stefan P. L. de Jong & Jorrit Smit & Leonie van Drooge, 2016. "Scientists’ response to societal impact policies: A policy paradox," Science and Public Policy, Oxford University Press, vol. 43(1), pages 102-114.
    3. Gunnar Sivertsen & Ingeborg Meijer, 2020. "Normal versus extraordinary societal impact: how to understand, evaluate, and improve research activities in their relations to society?," Research Evaluation, Oxford University Press, vol. 29(1), pages 66-70.
    4. Rip, Arie, 2000. "Higher forms of nonsense," European Review, Cambridge University Press, vol. 8(4), pages 467-485, October.
    5. Lutz Bornmann, 2013. "What is societal impact of research and how can it be assessed? a literature survey," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(2), pages 217-233, February.
    6. Stefan P L de Jong & Reetta Muhonen, 2020. "Who benefits from ex ante societal impact evaluation in the European funding arena? A cross-country comparison of societal impact capacity in the social sciences and humanities," Research Evaluation, Oxford University Press, vol. 29(1), pages 22-33.
    7. Martin Reinhart, 2010. "Peer review practices: a content analysis of external reviews in science funding," Research Evaluation, Oxford University Press, vol. 19(5), pages 317-331, December.
    8. Pablo D’Este & Irene Ramos-Vielba & Richard Woolley & Nabil Amara, 2018. "How do researchers generate scientific and societal impacts? Toward an analytical and operational framework," Science and Public Policy, Oxford University Press, vol. 45(6), pages 752-763.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Rebecca Abma-Schouten & Joey Gijbels & Wendy Reijmerink & Ingeborg Meijer, 2023. "Evaluation of research proposals by peer review panels: broader panels for broader assessments?," Science and Public Policy, Oxford University Press, vol. 50(4), pages 619-632.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Helka Kalliomäki & Sampo Ruoppila & Jenni Airaksinen, 2021. "It takes two to tango: Examining productive interactions in urban research collaboration [Generating Research Questions through Problematization]," Research Evaluation, Oxford University Press, vol. 30(4), pages 529-539.
    2. Lai Ma & Rachael Agnew, 2022. "Deconstructing impact: A framework for impact evaluation in grant applications [Evidencing Impact from Art Research: Analysis of Impact Case Studies from the REF 2014]," Science and Public Policy, Oxford University Press, vol. 49(2), pages 289-301.
    3. Zhang, Lin & Sivertsen, Gunnar & Du, Huiying & HUANG, Ying & Glänzel, Wolfgang, 2021. "Gender differences in the aims and impacts of research," SocArXiv 9n347, Center for Open Science.
    4. Jorrit P Smit & Laurens K Hessels, 2021. "The production of scientific and societal value in research evaluation: a review of societal impact assessment methods [Systems Thinking, Knowledge and Action: Towards Better Models and Methods]," Research Evaluation, Oxford University Press, vol. 30(3), pages 323-335.
    5. Lin Zhang & Gunnar Sivertsen & Huiying Du & Ying Huang & Wolfgang Glänzel, 2021. "Gender differences in the aims and impacts of research," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(11), pages 8861-8886, November.
    6. Stefan P L de Jong & Corina Balaban, 2022. "How universities influence societal impact practices: Academics’ sense-making of organizational impact strategies [Between Relevance and Excellence? Research Impact Agenda and the Production of Pol," Science and Public Policy, Oxford University Press, vol. 49(4), pages 609-620.
    7. Stefan P L de Jong & Corina Balaban & Maria Nedeva, 2022. "From ‘productive interactions’ to ‘enabling conditions’: The role of organizations in generating societal impact of academic research [One Size Does Not Fit All! New Perspectives on the University ," Science and Public Policy, Oxford University Press, vol. 49(4), pages 643-645.
    8. Rosa Kuipers-Dirven & Matthijs Janssen & Jarno Hoekman, 2023. "Assessing university policies for enhancing societal impact of academic research: A multicriteria mapping approach," Research Evaluation, Oxford University Press, vol. 32(2), pages 371-383.
    9. Conor O’Kane & Jing A. Zhang & Jarrod Haar & James A. Cunningham, 2023. "How scientists interpret and address funding criteria: value creation and undesirable side effects," Small Business Economics, Springer, vol. 61(2), pages 799-826, August.
    10. Ole Henning Sørensen & Jakob Bjørner & Andreas Holtermann & Johnny Dyreborg & Jorid Birkelund Sørli & Jesper Kristiansen & Steffen Bohni Nielsen, 2022. "Measuring societal impact of research—Developing and validating an impact instrument for occupational health and safety," Research Evaluation, Oxford University Press, vol. 31(1), pages 118-131.
    11. Lutz Bornmann & Robin Haunschild & Werner Marx, 2016. "Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents?," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1477-1495, December.
    12. Oscar LLOPIS & Joaquin AZAGRA-CARO, 2015. "Who do you care about? Scientists’ personality traits and perceived beneficiary impact," Cahiers du GREThA (2007-2019) 2015-29, Groupe de Recherche en Economie Théorique et Appliquée (GREThA).
    13. Hans Jonker & Florian Vanlee & Walter Ysebaert, 2022. "Societal impact of university research in the written press: media attention in the context of SIUR and the open science agenda among social scientists in Flanders, Belgium," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7289-7306, December.
    14. repec:oup:rseval:v:32:y:2024:i:2:p:348-355. is not listed on IDEAS
    15. Pandey, Poonam & Pansera, Mario, 2020. "Bringing Laxmi and Saraswati together: Nano-scientists and academic entrepreneurship in India," Technology in Society, Elsevier, vol. 63(C).
    16. Gunnar Sivertsen & Ingeborg Meijer, 2020. "Normal versus extraordinary societal impact: how to understand, evaluate, and improve research activities in their relations to society?," Research Evaluation, Oxford University Press, vol. 29(1), pages 66-70.
    17. Hashem Atapour & Robabeh Maddahi & Rasoul Zavaraqi, 2024. "Policy citations of scientometric articles: an altmetric study," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(7), pages 4423-4436, July.
    18. Kroll, Henning & Hansmeier, Hendrik & Hufnagl, Miriam, 2022. "Productive interactions in basic research an enquiry into impact pathways at the DESY synchrotron," Technological Forecasting and Social Change, Elsevier, vol. 175(C).
    19. David Barberá-Tomás & Joaquín M. Azagra-Caro & Pablo D’Este, 2022. "Dynamic perspectives on technology transfer: introduction to the special section," The Journal of Technology Transfer, Springer, vol. 47(5), pages 1299-1307, October.
    20. Juha-Pekka Lauronen, 2022. "The epistemic, production, and accountability prospects of social impact: An analysis of strategic research proposals," Research Evaluation, Oxford University Press, vol. 31(2), pages 214-225.
    21. Matteo Pedrini & Valentina Langella & Mario Alberto Battaglia & Paola Zaratin, 2018. "Assessing the health research’s social impact: a systematic review," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 1227-1250, March.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:oup:scippl:v:48:y:2021:i:6:p:763-775.. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Oxford University Press (email available below). General contact details of provider: https://academic.oup.com/spp .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.