IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v98y2014i1d10.1007_s11192-013-1020-x.html
   My bibliography  Save this article

How should the societal impact of research be generated and measured? A proposal for a simple and practicable approach to allow interdisciplinary comparisons

Author

Listed:
  • Lutz Bornmann

    (Administrative Headquarters)

  • Werner Marx

    (Max Planck Institute for Solid State Research)

Abstract

Since the 1990s, the scope of research evaluation has widened to encompass the societal products (outputs), societal use (societal references) and societal benefits (changes in society) of research. Research evaluation has been extended to include measures of the (1) social, (2) cultural, (3) environmental and (4) economic returns from publicly funded research. Even though no robust or reliable methods for measuring societal impact have yet been developed. In this study, we would like to introduce an approach which, unlike the currently common case study approach (and others), is relatively simple, can be used in almost every subject area and delivers results regarding societal impact which can be compared between disciplines. Our approach to societal impact starts with the actual function of science in society: to generate reliable knowledge. That is why a study (which we would like to refer to as an assessment report) summarising the status of the research on a certain subject represents knowledge which is available for society to access. Societal impact is given when the content of a report is addressed outside of science (in a government document, for example).

Suggested Citation

  • Lutz Bornmann & Werner Marx, 2014. "How should the societal impact of research be generated and measured? A proposal for a simple and practicable approach to allow interdisciplinary comparisons," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 211-219, January.
    • Handle: RePEc:spr:scient:v:98:y:2014:i:1:d:10.1007_s11192-013-1020-x
    DOI: 10.1007/s11192-013-1020-x
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-013-1020-x
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-013-1020-x?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Bornmann, Lutz & Mutz, Rüdiger & Hug, Sven E. & Daniel, Hans-Dieter, 2011. "A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants," Journal of Informetrics, Elsevier, vol. 5(3), pages 346-359.
    2. J Britt Holbrook & Robert Frodeman, 2011. "Peer review and the ex ante assessment of societal impacts," Research Evaluation, Oxford University Press, vol. 20(3), pages 239-246, September.
    3. Salter, Ammon J. & Martin, Ben R., 2001. "The economic benefits of publicly funded basic research: a critical review," Research Policy, Elsevier, vol. 30(3), pages 509-532, March.
    4. Bornmann, Lutz & Mutz, Rüdiger & Daniel, Hans-Dieter, 2007. "Gender differences in grant peer review: A meta-analysis," Journal of Informetrics, Elsevier, vol. 1(3), pages 226-238.
    5. Erik Ernø-Kjølhede & Finn Hansson, 2011. "Measuring research performance during a changing relationship between science and society," Research Evaluation, Oxford University Press, vol. 20(2), pages 131-143, June.
    6. Heather Piwowar, 2013. "Value all research products," Nature, Nature, vol. 493(7431), pages 159-159, January.
    7. Barend van der Meulen & Arie Rip, 2000. "Evaluation of societal quality of public sector research in the Netherlands," Research Evaluation, Oxford University Press, vol. 9(1), pages 11-25, April.
    8. Petit, Jean-Claude, 2004. "Why do we need fundamental research?," European Review, Cambridge University Press, vol. 12(2), pages 191-207, May.
    9. Ben R Martin, 2011. "The Research Excellence Framework and the ‘impact agenda’: are we creating a Frankenstein monster?," Research Evaluation, Oxford University Press, vol. 20(3), pages 247-254, September.
    10. Sarah Bell & Ben Shaw & Annette Boaz, 2011. "Real-world approaches to assessing the impact of environmental research on policy," Research Evaluation, Oxford University Press, vol. 20(3), pages 227-237, September.
    11. Lutz Bornmann, 2013. "What is societal impact of research and how can it be assessed? a literature survey," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(2), pages 217-233, February.
    12. Claire Donovan, 2011. "State of the art in assessing research impact: introduction to a special issue," Research Evaluation, Oxford University Press, vol. 20(3), pages 175-179, September.
    13. Birgitte Gregersen & Lisbeth Tved Linde & Jorgen Gulddahl Rasmussen, 2009. "Linking between Danish universities and society," Science and Public Policy, Oxford University Press, vol. 36(2), pages 151-156, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Keungoui Kim & Dieter F. Kogler & Sira Maliphol, 2024. "Identifying interdisciplinary emergence in the science of science: combination of network analysis and BERTopic," Palgrave Communications, Palgrave Macmillan, vol. 11(1), pages 1-15, December.
    2. Bornmann, Lutz, 2014. "Validity of altmetrics data for measuring societal impact: A study using data from Altmetric and F1000Prime," Journal of Informetrics, Elsevier, vol. 8(4), pages 935-950.
    3. Ardito, Lorenzo & Petruzzelli, Antonio Messeni & Ghisetti, Claudia, 2019. "The impact of public research on the technological development of industry in the green energy field," Technological Forecasting and Social Change, Elsevier, vol. 144(C), pages 25-35.
    4. Andrea Bonaccorsi & Filippo Chiarello & Gualtiero Fantoni, 2021. "Impact for whom? Mapping the users of public research with lexicon-based text mining," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1745-1774, February.
    5. Ferreira, Paula & Rocha, Ana & Araujo, Madalena & Afonso, Joao L. & Antunes, Carlos Henggeler & Lopes, Marta A.R. & Osório, Gerardo J. & Catalão, João P.S. & Lopes, João Peças, 2023. "Assessing the societal impact of smart grids: Outcomes of a collaborative research project," Technology in Society, Elsevier, vol. 72(C).
    6. João Carlos Nabout & Fabrício Barreto Teresa & Karine Borges Machado & Vitor Hugo Mendonça Prado & Luis Mauricio Bini & José Alexandre Felizola Diniz-Filho, 2018. "Do traditional scientometric indicators predict social media activity on scientific knowledge? An analysis of the ecological literature," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(2), pages 1007-1015, May.
    7. Matteo Pedrini & Valentina Langella & Mario Alberto Battaglia & Paola Zaratin, 2018. "Assessing the health research’s social impact: a systematic review," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 1227-1250, March.
    8. Marco Gatti, 2018. "The Impact of Management Accounting Research: An Analysis of the Past and a Look at the Future," International Journal of Business and Management, Canadian Center of Science and Education, vol. 13(5), pages 1-47, March.
    9. Han Zheng & L. G. Pee & Dan Zhang, 2021. "Societal impact of research: a text mining study of impact types," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 7397-7417, September.
    10. Rousseau, Sandra & Catalano, Giuseppe & Daraio, Cinzia, 2021. "Can we estimate a monetary value of scientific publications?," Research Policy, Elsevier, vol. 50(1).
    11. Morretta, Valentina & Vurchio, Davide & Carrazza, Stefano, 2022. "The socio-economic value of scientific publications: The case of Earth Observation satellites," Technological Forecasting and Social Change, Elsevier, vol. 180(C).
    12. Ana Sofia Brandão & José M. R. C. A. Santos, 2024. "Sustainability from Policy to Practice: Assessing the Impact of European Research and Innovation Frameworks on Circular Bioeconomy," Sustainability, MDPI, vol. 16(6), pages 1-25, March.
    13. Dotti, Nicola Francesco & Walczyk, Julia, 2022. "What is the societal impact of university research? A policy-oriented review to map approaches, identify monitoring methods and success factors," Evaluation and Program Planning, Elsevier, vol. 95(C).
    14. Lutz Bornmann & Robin Haunschild & Werner Marx, 2016. "Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents?," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1477-1495, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lutz Bornmann, 2013. "What is societal impact of research and how can it be assessed? a literature survey," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(2), pages 217-233, February.
    2. Matteo Pedrini & Valentina Langella & Mario Alberto Battaglia & Paola Zaratin, 2018. "Assessing the health research’s social impact: a systematic review," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 1227-1250, March.
    3. Joly, P.B. & Gaunand, A. & Colinet, L. & Larédo, P. & Lemarié, S. & Matt, M., 2015. "ASIRPA: a comprehensive theory-based approach to assessing the societal impacts of a research organization," Working Papers 2015-04, Grenoble Applied Economics Laboratory (GAEL).
    4. Andrea Bonaccorsi & Filippo Chiarello & Gualtiero Fantoni, 2021. "Impact for whom? Mapping the users of public research with lexicon-based text mining," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1745-1774, February.
    5. Irwin Feller, 2022. "Assessing the societal impact of publicly funded research," The Journal of Technology Transfer, Springer, vol. 47(3), pages 632-650, June.
    6. Kroll, Henning & Hansmeier, Hendrik & Hufnagl, Miriam, 2022. "Productive interactions in basic research an enquiry into impact pathways at the DESY synchrotron," Technological Forecasting and Social Change, Elsevier, vol. 175(C).
    7. Caitlin Drummond Otten & Baruch Fischhoff, 2022. "Assessing broader impacts of funded research: the US National Science Foundation v. Lamar Smith [What is Societal Impact of Research and How Can it Be Assessed? A Literature Survey]," Science and Public Policy, Oxford University Press, vol. 49(2), pages 313-323.
    8. Gaunand, A. & Hocdé, A. & Lemarié, S. & Matt, M. & Turckheim, E.de, 2015. "How does public agricultural research impact society? A characterization of various patterns," Research Policy, Elsevier, vol. 44(4), pages 849-861.
    9. Bornmann, Lutz & Haunschild, Robin & Adams, Jonathan, 2019. "Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF)," Journal of Informetrics, Elsevier, vol. 13(1), pages 325-340.
    10. Lutz Bornmann & Robin Haunschild & Werner Marx, 2016. "Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents?," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1477-1495, December.
    11. Alessandro Margherita & Gianluca Elia & Claudio Petti, 2022. "What Is Quality in Research? Building a Framework of Design, Process and Impact Attributes and Evaluation Perspectives," Sustainability, MDPI, vol. 14(5), pages 1-18, March.
    12. Eva María de la Torre & Fernando Casani & Carmen Pérez Esparrells, 2021. "Measuring universities’ engagement: a revision of the European research projects and the actual use of the so-called ‘third mission’ indicators," Revista de Estudios Regionales, Universidades Públicas de Andalucía, vol. 1, pages 97-128.
    13. J. Britt Holbrook, 2017. "The future of the impact agenda depends on the revaluation of academic freedom," Palgrave Communications, Palgrave Macmillan, vol. 3(1), pages 1-9, December.
    14. Matt, M. & Gaunand, A. & Joly, P-B. & Colinet, L., 2017. "Opening the black box of impact – Ideal-type impact pathways in a public agricultural research organization," Research Policy, Elsevier, vol. 46(1), pages 207-218.
    15. Pierre Benoit Joly & Laurence Colinet & Ariane Gaunand & Stephane Lemarié & Mireille Matt, 2016. "Agricultural research impact assessment: issues, methods and challenges," Working Papers hal-01431457, HAL.
    16. Irwin Feller, 2013. "Peer review and expert panels as techniques for evaluating the quality of academic research," Chapters, in: Albert N. Link & Nicholas S. Vonortas (ed.), Handbook on the Theory and Practice of Program Evaluation, chapter 5, pages 115-142, Edward Elgar Publishing.
    17. Dotti, Nicola Francesco & Walczyk, Julia, 2022. "What is the societal impact of university research? A policy-oriented review to map approaches, identify monitoring methods and success factors," Evaluation and Program Planning, Elsevier, vol. 95(C).
    18. Jonathan P. Doh & Lorraine Eden & Anne S. Tsui & Srilata Zaheer, 2023. "Developing international business scholarship for global societal impact," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 54(5), pages 757-767, July.
    19. Pablo D’Este & Irene Ramos-Vielba & Richard Woolley & Nabil Amara, 2018. "How do researchers generate scientific and societal impacts? Toward an analytical and operational framework," Science and Public Policy, Oxford University Press, vol. 45(6), pages 752-763.
    20. Juha-Pekka Lauronen, 2022. "Tension in Interpretations of the Social Impact of the Social Sciences: Walking a Tightrope Between Divergent Conceptualizations of Research Utilization," SAGE Open, , vol. 12(2), pages 21582440221, April.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:98:y:2014:i:1:d:10.1007_s11192-013-1020-x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.