IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v109y2016i3d10.1007_s11192-016-2115-y.html
   My bibliography  Save this article

Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents?

Author

Listed:
  • Lutz Bornmann

    (Administrative Headquarters of the Max Planck Society)

  • Robin Haunschild

    (Max Planck Institute for Solid State Research)

  • Werner Marx

    (Max Planck Institute for Solid State Research)

Abstract

In the current UK Research Excellence Framework (REF) and the Excellence in Research for Australia (ERA), societal impact measurements are inherent parts of the national evaluation systems. In this study, we deal with a relatively new form of societal impact measurements. Recently, Altmetric—a start-up providing publication level metrics—started to make data for publications available which have been mentioned in policy documents. We regard this data source as an interesting possibility to specifically measure the (societal) impact of research. Using a comprehensive dataset with publications on climate change as an example, we study the usefulness of the new data source for impact measurement. Only 1.2 % (n = 2341) out of 191,276 publications on climate change in the dataset have at least one policy mention. We further reveal that papers published in Nature and Science as well as from the areas “Earth and related environmental sciences” and “Social and economic geography” are especially relevant in the policy context. Given the low coverage of the climate change literature in policy documents, this study can be only a first attempt to study this new source of altmetrics data. Further empirical studies are necessary, because mentions in policy documents are of special interest in the use of altmetrics data for measuring target-oriented the broader impact of research.

Suggested Citation

  • Lutz Bornmann & Robin Haunschild & Werner Marx, 2016. "Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents?," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1477-1495, December.
    • Handle: RePEc:spr:scient:v:109:y:2016:i:3:d:10.1007_s11192-016-2115-y
    DOI: 10.1007/s11192-016-2115-y
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-016-2115-y
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-016-2115-y?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Paul Nightingale & Alister Scott, 2007. "Peer review and the relevance gap: Ten suggestions for policy-makers," Science and Public Policy, Oxford University Press, vol. 34(8), pages 543-553, October.
    2. Erik Ernø-Kjølhede & Finn Hansson, 2011. "Measuring research performance during a changing relationship between science and society," Research Evaluation, Oxford University Press, vol. 20(2), pages 131-143, June.
    3. Lutz Bornmann, 2013. "What is societal impact of research and how can it be assessed? a literature survey," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(2), pages 217-233, February.
    4. Jonathan Grant, 1999. "Evaluating the outcomes of biomedical research on healthcare," Research Evaluation, Oxford University Press, vol. 8(1), pages 33-38, April.
    5. Peter M. Atkinson, 2014. "Assess the real cost of research assessment," Nature, Nature, vol. 516(7530), pages 145-145, December.
    6. Spruijt, Pita & Knol, Anne B. & Petersen, Arthur C. & Lebret, Erik, 2016. "Differences in views of experts about their role in particulate matter policy advice: Empirical evidence from an international expert consultation," Environmental Science & Policy, Elsevier, vol. 59(C), pages 44-52.
    7. Gabrielle N. Samuel & Gemma E. Derrick, 2015. "Societal impact evaluation: Exploring evaluator perceptions of the characterization of impact under the REF2014," Research Evaluation, Oxford University Press, vol. 24(3), pages 229-241.
    8. Bornmann, Lutz, 2014. "Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics," Journal of Informetrics, Elsevier, vol. 8(4), pages 895-903.
    9. Hanan Khazragui & John Hudson, 2015. "Measuring the benefits of university research: impact and the REF in the UK," Research Evaluation, Oxford University Press, vol. 24(1), pages 51-62.
    10. Lutz Bornmann & Werner Marx, 2014. "How should the societal impact of research be generated and measured? A proposal for a simple and practicable approach to allow interdisciplinary comparisons," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 211-219, January.
    11. C. Weiss, 2006. "Precaution: The Willingness to Accept Costs to Avert Uncertain Danger," Lecture Notes in Economics and Mathematical Systems, in: Coping with Uncertainty, pages 315-330, Springer.
    12. Charles Weiss, 2003. "Scientific Uncertainty and Science-Based Precaution," International Environmental Agreements: Politics, Law and Economics, Springer, vol. 3(2), pages 137-166, June.
    13. Petit, Jean-Claude, 2004. "Why do we need fundamental research?," European Review, Cambridge University Press, vol. 12(2), pages 191-207, May.
    14. Amber Dance, 2013. "Impact: Pack a punch," Nature, Nature, vol. 502(7471), pages 397-398, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Andrea Bonaccorsi & Filippo Chiarello & Gualtiero Fantoni, 2021. "Impact for whom? Mapping the users of public research with lexicon-based text mining," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1745-1774, February.
    2. Matteo Pedrini & Valentina Langella & Mario Alberto Battaglia & Paola Zaratin, 2018. "Assessing the health research’s social impact: a systematic review," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 1227-1250, March.
    3. Dotti, Nicola Francesco & Walczyk, Julia, 2022. "What is the societal impact of university research? A policy-oriented review to map approaches, identify monitoring methods and success factors," Evaluation and Program Planning, Elsevier, vol. 95(C).
    4. Shahzad, Murtuza & Alhoori, Hamed & Freedman, Reva & Rahman, Shaikh Abdul, 2022. "Quantifying the online long-term interest in research," Journal of Informetrics, Elsevier, vol. 16(2).
    5. Bornmann, Lutz, 2014. "Validity of altmetrics data for measuring societal impact: A study using data from Altmetric and F1000Prime," Journal of Informetrics, Elsevier, vol. 8(4), pages 935-950.
    6. Lutz Bornmann, 2013. "What is societal impact of research and how can it be assessed? a literature survey," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(2), pages 217-233, February.
    7. Marco Gatti, 2018. "The Impact of Management Accounting Research: An Analysis of the Past and a Look at the Future," International Journal of Business and Management, Canadian Center of Science and Education, vol. 13(5), pages 1-47, March.
    8. Bornmann, Lutz & Haunschild, Robin & Adams, Jonathan, 2019. "Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF)," Journal of Informetrics, Elsevier, vol. 13(1), pages 325-340.
    9. Alessandro Margherita & Gianluca Elia & Claudio Petti, 2022. "What Is Quality in Research? Building a Framework of Design, Process and Impact Attributes and Evaluation Perspectives," Sustainability, MDPI, vol. 14(5), pages 1-18, March.
    10. Lutz Bornmann & Werner Marx, 2014. "How should the societal impact of research be generated and measured? A proposal for a simple and practicable approach to allow interdisciplinary comparisons," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 211-219, January.
    11. Sander C. S. Clahsen & Irene van Kamp & Betty C. Hakkert & Theo G. Vermeire & Aldert H. Piersma & Erik Lebret, 2019. "Why Do Countries Regulate Environmental Health Risks Differently? A Theoretical Perspective," Risk Analysis, John Wiley & Sons, vol. 39(2), pages 439-461, February.
    12. Keungoui Kim & Dieter F. Kogler & Sira Maliphol, 2024. "Identifying interdisciplinary emergence in the science of science: combination of network analysis and BERTopic," Palgrave Communications, Palgrave Macmillan, vol. 11(1), pages 1-15, December.
    13. Oscar LLOPIS & Joaquin AZAGRA-CARO, 2015. "Who do you care about? Scientists’ personality traits and perceived beneficiary impact," Cahiers du GREThA (2007-2019) 2015-29, Groupe de Recherche en Economie Théorique et Appliquée (GREThA).
    14. Junwen Luo & Lai Ma & Kalpana Shankar, 2021. "Does the inclusion of non-academic reviewers make any difference for grant impact panels? [Understanding the Long Term Impact of the Framework Programme, European Policy Evaluation Consortium (EPEC," Science and Public Policy, Oxford University Press, vol. 48(6), pages 763-775.
    15. Pita Spruijt & Anne B. Knol & Arthur C. Petersen & Erik Lebret, 2019. "Expert Views on Their Role as Policy Advisor: Pilot Study for the Cases of Electromagnetic Fields, Particulate Matter, and Antimicrobial Resistance," Risk Analysis, John Wiley & Sons, vol. 39(5), pages 968-974, May.
    16. Perkmann, Markus & Salandra, Rossella & Tartari, Valentina & McKelvey, Maureen & Hughes, Alan, 2021. "Academic engagement: A review of the literature 2011-2019," Research Policy, Elsevier, vol. 50(1).
    17. Zhang, Lin & Sivertsen, Gunnar & Du, Huiying & HUANG, Ying & Glänzel, Wolfgang, 2021. "Gender differences in the aims and impacts of research," SocArXiv 9n347, Center for Open Science.
    18. Jorrit P Smit & Laurens K Hessels, 2021. "The production of scientific and societal value in research evaluation: a review of societal impact assessment methods [Systems Thinking, Knowledge and Action: Towards Better Models and Methods]," Research Evaluation, Oxford University Press, vol. 30(3), pages 323-335.
    19. Peter T. Jacobs & Alexis Habiyaremye & Bhekiwe Fakudze & Kgabo Ramoroka & Siyanda Jonas, 2019. "Producing Knowledge to Raise Rural Living Standards: How Universities Connect with Resource-Poor Municipalities in South Africa," The European Journal of Development Research, Palgrave Macmillan;European Association of Development Research and Training Institutes (EADI), vol. 31(4), pages 881-901, September.
    20. Hashem Atapour & Robabeh Maddahi & Rasoul Zavaraqi, 2024. "Policy citations of scientometric articles: an altmetric study," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(7), pages 4423-4436, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:109:y:2016:i:3:d:10.1007_s11192-016-2115-y. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.