IDEAS home Printed from https://ideas.repec.org/a/oup/rseval/v32y2023i2p371-383..html
   My bibliography  Save this article

Assessing university policies for enhancing societal impact of academic research: A multicriteria mapping approach

Author

Listed:
  • Rosa Kuipers-Dirven
  • Matthijs Janssen
  • Jarno Hoekman

Abstract

While there is a prolific debate on evaluating the societal impact of academic research, so far little attention has been paid to assessing the design and implementation of suitable organizational strategies. This article argues that evaluation methods are needed which are both formative and sensitive to diverging views on what defines and constitutes societal impact. We use a participatory deliberation method, the Multicriteria Mapping (MCM) approach, to examine how stakeholders appraise the use of university policy options for enhancing societal impact, and understand on what basis they judge the performance of these options. Focusing on a large Dutch research-based university, we conduct 22 interviews with academics, management, and support staff as well as strategic policy officers to examine how they rank and discuss the expected performance of university policy options identified in previous literature. Our results show that interviewees base their scores on criteria related to policy options’ expected organizational output and external outcomes, as well as their practical and, to a lesser degree, cultural, and financial feasibility. The resulting rankings also point at contrasts in the perceived potential of policy options, with interviewees assigning priority to providing researchers with recognition and rewards for impact-based activities. We conclude by discussing how MCM can be used as a formative evaluation method to assess and select policies and inform decision-making that fit a university’s particular situation. Besides drawing lessons for the context of our illustrative case, we also reflect on the relevance of the evaluation method and our findings for other universities.

Suggested Citation

  • Rosa Kuipers-Dirven & Matthijs Janssen & Jarno Hoekman, 2023. "Assessing university policies for enhancing societal impact of academic research: A multicriteria mapping approach," Research Evaluation, Oxford University Press, vol. 32(2), pages 371-383.
  • Handle: RePEc:oup:rseval:v:32:y:2023:i:2:p:371-383.
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1093/reseval/rvac045
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Jordi Molas-Gallart & Alejandra Boni & Sandro Giachi & Johan Schot, 2021. "A formative approach to the evaluation of Transformative Innovation Policies [The Need for Reflexive Evaluation Approaches in Development Cooperation]," Research Evaluation, Oxford University Press, vol. 30(4), pages 431-442.
    2. Rogge, Karoline S. & Reichardt, Kristin, 2016. "Policy mixes for sustainability transitions: An extended concept and framework for analysis," Research Policy, Elsevier, vol. 45(8), pages 1620-1635.
    3. Brian Belcher & Janet Halliwell, 2021. "Conceptualizing the elements of research impact: towards semantic standards," Palgrave Communications, Palgrave Macmillan, vol. 8(1), pages 1-6, December.
    4. Perkmann, Markus & Salandra, Rossella & Tartari, Valentina & McKelvey, Maureen & Hughes, Alan, 2021. "Academic engagement: A review of the literature 2011-2019," Research Policy, Elsevier, vol. 50(1).
    5. Stefan P. L. de Jong & Jorrit Smit & Leonie van Drooge, 2016. "Scientists’ response to societal impact policies: A policy paradox," Science and Public Policy, Oxford University Press, vol. 43(1), pages 102-114.
    6. Chris Woolston, 2021. "Impact factor abandoned by Dutch university in hiring and promotion decisions," Nature, Nature, vol. 595(7867), pages 462-462, July.
    7. Kateryna Wowk & Larry McKinney & Frank Muller-Karger & Russell Moll & Susan Avery & Elva Escobar-Briones & David Yoskowitz & Richard McLaughlin, 2017. "Evolving academic culture to meet societal needs," Palgrave Communications, Palgrave Macmillan, vol. 3(1), pages 1-7, December.
    8. Ole Henning Sørensen & Jakob Bjørner & Andreas Holtermann & Johnny Dyreborg & Jorid Birkelund Sørli & Jesper Kristiansen & Steffen Bohni Nielsen, 2022. "Measuring societal impact of research—Developing and validating an impact instrument for occupational health and safety," Research Evaluation, Oxford University Press, vol. 31(1), pages 118-131.
    9. Laurens K Hessels & Harro van Lente & Ruud Smits, 2009. "In search of relevance: The changing contract between science and society," Science and Public Policy, Oxford University Press, vol. 36(5), pages 387-401, June.
    10. Jordi Molas-Gallart & Puay Tang, 2011. "Tracing ‘productive interactions’ to identify social impacts: an example from the social sciences," Research Evaluation, Oxford University Press, vol. 20(3), pages 219-226, September.
    11. Jorrit P Smit & Laurens K Hessels, 2021. "The production of scientific and societal value in research evaluation: a review of societal impact assessment methods [Systems Thinking, Knowledge and Action: Towards Better Models and Methods]," Research Evaluation, Oxford University Press, vol. 30(3), pages 323-335.
    12. Pablo D’Este & Irene Ramos-Vielba & Richard Woolley & Nabil Amara, 2018. "How do researchers generate scientific and societal impacts? Toward an analytical and operational framework," Science and Public Policy, Oxford University Press, vol. 45(6), pages 752-763.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. repec:oup:rseval:v:32:y:2024:i:2:p:371-383. is not listed on IDEAS
    2. Thomas, Duncan Andrew & Ramos-Vielba, Irene, 2022. "Reframing study of research(er) funding towards configurations and trails," SocArXiv uty2v, Center for Open Science.
    3. Stefan P L de Jong & Corina Balaban & Maria Nedeva, 2022. "From ‘productive interactions’ to ‘enabling conditions’: The role of organizations in generating societal impact of academic research [One Size Does Not Fit All! New Perspectives on the University ," Science and Public Policy, Oxford University Press, vol. 49(4), pages 643-645.
    4. Adrian Rauchfleisch & Mike S Schäfer & Dario Siegen, 2021. "Beyond the ivory tower: Measuring and explaining academic engagement with journalists, politicians and industry representatives among Swiss professorss," PLOS ONE, Public Library of Science, vol. 16(5), pages 1-20, May.
    5. Helka Kalliomäki & Sampo Ruoppila & Jenni Airaksinen, 2021. "It takes two to tango: Examining productive interactions in urban research collaboration [Generating Research Questions through Problematization]," Research Evaluation, Oxford University Press, vol. 30(4), pages 529-539.
    6. Junwen Luo & Lai Ma & Kalpana Shankar, 2021. "Does the inclusion of non-academic reviewers make any difference for grant impact panels? [Understanding the Long Term Impact of the Framework Programme, European Policy Evaluation Consortium (EPEC," Science and Public Policy, Oxford University Press, vol. 48(6), pages 763-775.
    7. Haddad, Carolina R. & Bergek, Anna, 2023. "Towards an integrated framework for evaluating transformative innovation policy," Research Policy, Elsevier, vol. 52(2).
    8. Suominen, Arho & Kauppinen, Henni & Hyytinen, Kirsi, 2021. "‘Gold’, ‘Ribbon’ or ‘Puzzle’: What motivates researchers to work in Research and Technology Organizations," Technological Forecasting and Social Change, Elsevier, vol. 170(C).
    9. Gunnar Sivertsen & Ingeborg Meijer, 2020. "Normal versus extraordinary societal impact: how to understand, evaluate, and improve research activities in their relations to society?," Research Evaluation, Oxford University Press, vol. 29(1), pages 66-70.
    10. Kwadwo Atta-Owusu & Rune Dahl Fitjar, 2022. "What motivates academics for external engagement? Exploring the effects of motivational drivers and organizational fairness [The Nature of Academic Entrepreneurship in the UK: Widening the Focus on," Science and Public Policy, Oxford University Press, vol. 49(2), pages 201-218.
    11. Paula Kivimaa & Jani Lukkarinen & David Lazarevic, 2023. "Analysis of COVID-19 recovery and resilience policy in Finland: a transformative policy mix approach," Science and Public Policy, Oxford University Press, vol. 50(4), pages 681-694.
    12. Kroll, Henning & Hansmeier, Hendrik & Hufnagl, Miriam, 2022. "Productive interactions in basic research an enquiry into impact pathways at the DESY synchrotron," Technological Forecasting and Social Change, Elsevier, vol. 175(C).
    13. Marques, Marcelo, 2021. "How do policy instruments generate new ones? Analysing policy instruments feedback and interaction in educational research in England, 1986-2014," Research Policy, Elsevier, vol. 50(10).
    14. David Barberá-Tomás & Joaquín M. Azagra-Caro & Pablo D’Este, 2022. "Dynamic perspectives on technology transfer: introduction to the special section," The Journal of Technology Transfer, Springer, vol. 47(5), pages 1299-1307, October.
    15. Frank J. Rijnsoever & Laurens K. Hessels, 2021. "How academic researchers select collaborative research projects: a choice experiment," The Journal of Technology Transfer, Springer, vol. 46(6), pages 1917-1948, December.
    16. Stefan P L de Jong & Corina Balaban, 2022. "How universities influence societal impact practices: Academics’ sense-making of organizational impact strategies [Between Relevance and Excellence? Research Impact Agenda and the Production of Pol," Science and Public Policy, Oxford University Press, vol. 49(4), pages 609-620.
    17. Leonie Drooge & Jack Spaapen, 2022. "Evaluation and monitoring of transdisciplinary collaborations," The Journal of Technology Transfer, Springer, vol. 47(3), pages 747-761, June.
    18. Chams, Nour & Guesmi, Bouali & Gil, Jose M. & Molins, Mireia & Cubel, Rosa, 2021. "Between “Research Producers” and “Research Adopters”: The Role of Knowledge and Innovation Transfer on Sustainability Impact," 2021 Conference, August 17-31, 2021, Virtual 315264, International Association of Agricultural Economists.
    19. Boncinelli, Fabio & Bartolini, Fabio & Casini, Leonardo, 2018. "Structural factors of labour allocation for farm diversification activities," Land Use Policy, Elsevier, vol. 71(C), pages 204-212.
    20. Hoekman, Jarno & Rake, Bastian, 2024. "Geography of authorship: How geography shapes authorship attribution in big team science," Research Policy, Elsevier, vol. 53(2).
    21. Caloffi, Annalisa & Freo, Marzia & Ghinoi, Stefano & Mariani, Marco & Rossi, Federica, 2022. "Assessing the effects of a deliberate policy mix: The case of technology and innovation advisory services and innovation vouchers," Research Policy, Elsevier, vol. 51(6).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:oup:rseval:v:32:y:2023:i:2:p:371-383.. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Oxford University Press (email available below). General contact details of provider: https://academic.oup.com/rev .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.