IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v109y2016i3d10.1007_s11192-016-2115-y.html
   My bibliography  Save this article

Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents?

Author

Listed:
  • Lutz Bornmann

    (Administrative Headquarters of the Max Planck Society)

  • Robin Haunschild

    (Max Planck Institute for Solid State Research)

  • Werner Marx

    (Max Planck Institute for Solid State Research)

Abstract

In the current UK Research Excellence Framework (REF) and the Excellence in Research for Australia (ERA), societal impact measurements are inherent parts of the national evaluation systems. In this study, we deal with a relatively new form of societal impact measurements. Recently, Altmetric—a start-up providing publication level metrics—started to make data for publications available which have been mentioned in policy documents. We regard this data source as an interesting possibility to specifically measure the (societal) impact of research. Using a comprehensive dataset with publications on climate change as an example, we study the usefulness of the new data source for impact measurement. Only 1.2 % (n = 2341) out of 191,276 publications on climate change in the dataset have at least one policy mention. We further reveal that papers published in Nature and Science as well as from the areas “Earth and related environmental sciences” and “Social and economic geography” are especially relevant in the policy context. Given the low coverage of the climate change literature in policy documents, this study can be only a first attempt to study this new source of altmetrics data. Further empirical studies are necessary, because mentions in policy documents are of special interest in the use of altmetrics data for measuring target-oriented the broader impact of research.

Suggested Citation

  • Lutz Bornmann & Robin Haunschild & Werner Marx, 2016. "Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents?," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1477-1495, December.
    • Handle: RePEc:spr:scient:v:109:y:2016:i:3:d:10.1007_s11192-016-2115-y
    DOI: 10.1007/s11192-016-2115-y
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-016-2115-y
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-016-2115-y?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Paul Nightingale & Alister Scott, 2007. "Peer review and the relevance gap: Ten suggestions for policy-makers," Science and Public Policy, Oxford University Press, vol. 34(8), pages 543-553, October.
    2. Erik Ernø-Kjølhede & Finn Hansson, 2011. "Measuring research performance during a changing relationship between science and society," Research Evaluation, Oxford University Press, vol. 20(2), pages 131-143, June.
    3. Hanan Khazragui & John Hudson, 2015. "Measuring the benefits of university research: impact and the REF in the UK," Research Evaluation, Oxford University Press, vol. 24(1), pages 51-62.
    4. Lutz Bornmann, 2013. "What is societal impact of research and how can it be assessed? a literature survey," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(2), pages 217-233, February.
    5. Lutz Bornmann & Werner Marx, 2014. "How should the societal impact of research be generated and measured? A proposal for a simple and practicable approach to allow interdisciplinary comparisons," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 211-219, January.
    6. C. Weiss, 2006. "Precaution: The Willingness to Accept Costs to Avert Uncertain Danger," Lecture Notes in Economics and Mathematical Systems, in: Coping with Uncertainty, pages 315-330, Springer.
    7. Jonathan Grant, 1999. "Evaluating the outcomes of biomedical research on healthcare," Research Evaluation, Oxford University Press, vol. 8(1), pages 33-38, April.
    8. Peter M. Atkinson, 2014. "Assess the real cost of research assessment," Nature, Nature, vol. 516(7530), pages 145-145, December.
    9. Spruijt, Pita & Knol, Anne B. & Petersen, Arthur C. & Lebret, Erik, 2016. "Differences in views of experts about their role in particulate matter policy advice: Empirical evidence from an international expert consultation," Environmental Science & Policy, Elsevier, vol. 59(C), pages 44-52.
    10. Charles Weiss, 2003. "Scientific Uncertainty and Science-Based Precaution," International Environmental Agreements: Politics, Law and Economics, Springer, vol. 3(2), pages 137-166, June.
    11. Petit, Jean-Claude, 2004. "Why do we need fundamental research?," European Review, Cambridge University Press, vol. 12(2), pages 191-207, May.
    12. Amber Dance, 2013. "Impact: Pack a punch," Nature, Nature, vol. 502(7471), pages 397-398, October.
    13. Gabrielle N. Samuel & Gemma E. Derrick, 2015. "Societal impact evaluation: Exploring evaluator perceptions of the characterization of impact under the REF2014," Research Evaluation, Oxford University Press, vol. 24(3), pages 229-241.
    14. Bornmann, Lutz, 2014. "Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics," Journal of Informetrics, Elsevier, vol. 8(4), pages 895-903.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. D’Este, Pablo & Robinson-García, Nicolás, 2023. "Interdisciplinary research and the societal visibility of science: The advantages of spanning multiple and distant scientific fields," Research Policy, Elsevier, vol. 52(2).
    2. Daniela Filippo & Pablo Sastrón-Toledo, 2023. "Influence of research on open science in the public policy sphere," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(3), pages 1995-2017, March.
    3. Yang, Chao & Huang, Cui, 2022. "Quantitative mapping of the evolution of AI policy distribution, targets and focuses over three decades in China," Technological Forecasting and Social Change, Elsevier, vol. 174(C).
    4. Bornmann, Lutz & Haunschild, Robin & Adams, Jonathan, 2019. "Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF)," Journal of Informetrics, Elsevier, vol. 13(1), pages 325-340.
    5. Bornmann, Lutz & Haunschild, Robin, 2018. "Normalization of zero-inflated data: An empirical analysis of a new indicator family and its use with altmetrics data," Journal of Informetrics, Elsevier, vol. 12(3), pages 998-1011.
    6. Liang Hu & Win-bin Huang & Yi Bu, 2024. "Interdisciplinary research attracts greater attention from policy documents: evidence from COVID-19," Palgrave Communications, Palgrave Macmillan, vol. 11(1), pages 1-10, December.
    7. Caterina Lucarelli & Camilla Mazzoli & Michela Rancan & Sabrina Severini, 2020. "Classification of Sustainable Activities: EU Taxonomy and Scientific Literature," Sustainability, MDPI, vol. 12(16), pages 1-25, August.
    8. Xiaozan Lyu & Rodrigo Costas, 2020. "How do academic topics shift across altmetric sources? A case study of the research area of Big Data," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(2), pages 909-943, May.
    9. Diana Hicks & Julia Melkers & Kimberley R. Isett, 2019. "A characterization of professional media and its links to research," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(2), pages 827-843, May.
    10. Lutz Bornmann & Robin Haunschild, 2017. "Does evaluative scientometrics lose its main focus on scientific quality by the new orientation towards societal impact?," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(2), pages 937-943, February.
    11. Dotti, Nicola Francesco & Walczyk, Julia, 2022. "What is the societal impact of university research? A policy-oriented review to map approaches, identify monitoring methods and success factors," Evaluation and Program Planning, Elsevier, vol. 95(C).
    12. Zhihong Huang & Qianjin Zong & Xuerui Ji, 2022. "The associations between scientific collaborations of LIS research and its policy impact," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(11), pages 6453-6470, November.
    13. Rachel Einecker & Andrew Kirby, 2020. "Climate Change: A Bibliometric Study of Adaptation, Mitigation and Resilience," Sustainability, MDPI, vol. 12(17), pages 1-16, August.
    14. Huang, Cui & Yang, Chao & Su, Jun, 2021. "Identifying core policy instruments based on structural holes: A case study of China’s nuclear energy policy," Journal of Informetrics, Elsevier, vol. 15(2).
    15. Cui Huang & Chao Yang & Jun Su, 2018. "Policy change analysis based on “policy target–policy instrument” patterns: a case study of China’s nuclear energy policy," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(2), pages 1081-1114, November.
    16. Knut Blind & Alex Fenton, 2022. "Standard-relevant publications: evidence, processes and influencing factors," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(1), pages 577-602, January.
    17. Marco Gatti, 2018. "The Impact of Management Accounting Research: An Analysis of the Past and a Look at the Future," International Journal of Business and Management, Canadian Center of Science and Education, vol. 13(5), pages 1-47, March.
    18. Dorte Drongstrup & Shafaq Malik & Naif Radi Aljohani & Salem Alelyani & Iqra Safder & Saeed-Ul Hassan, 2020. "Can social media usage of scientific literature predict journal indices of AJG, SNIP and JCR? An altmetric study of economics," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1541-1558, November.
    19. Robin Haunschild & Lutz Bornmann, 2017. "How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(3), pages 1209-1216, March.
    20. Andrea Bonaccorsi & Filippo Chiarello & Gualtiero Fantoni, 2021. "Impact for whom? Mapping the users of public research with lexicon-based text mining," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1745-1774, February.
    21. Yang, Chao & Huang, Cui & Su, Jun, 2020. "A bibliometrics-based research framework for exploring policy evolution: A case study of China's information technology policies," Technological Forecasting and Social Change, Elsevier, vol. 157(C).
    22. Clemens Blümel & Alexander Schniedermann, 2020. "Studying review articles in scientometrics and beyond: a research agenda," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(1), pages 711-728, July.
    23. Houqiang Yu & Xueting Cao & Tingting Xiao & Zhenyi Yang, 2020. "How accurate are policy document mentions? A first look at the role of altmetrics database," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1517-1540, November.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Andrea Bonaccorsi & Filippo Chiarello & Gualtiero Fantoni, 2021. "Impact for whom? Mapping the users of public research with lexicon-based text mining," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1745-1774, February.
    2. Matteo Pedrini & Valentina Langella & Mario Alberto Battaglia & Paola Zaratin, 2018. "Assessing the health research’s social impact: a systematic review," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 1227-1250, March.
    3. Dotti, Nicola Francesco & Walczyk, Julia, 2022. "What is the societal impact of university research? A policy-oriented review to map approaches, identify monitoring methods and success factors," Evaluation and Program Planning, Elsevier, vol. 95(C).
    4. Lutz Bornmann, 2013. "What is societal impact of research and how can it be assessed? a literature survey," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(2), pages 217-233, February.
    5. Shahzad, Murtuza & Alhoori, Hamed & Freedman, Reva & Rahman, Shaikh Abdul, 2022. "Quantifying the online long-term interest in research," Journal of Informetrics, Elsevier, vol. 16(2).
    6. Marco Gatti, 2018. "The Impact of Management Accounting Research: An Analysis of the Past and a Look at the Future," International Journal of Business and Management, Canadian Center of Science and Education, vol. 13(5), pages 1-47, March.
    7. Bornmann, Lutz & Haunschild, Robin & Adams, Jonathan, 2019. "Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF)," Journal of Informetrics, Elsevier, vol. 13(1), pages 325-340.
    8. Alessandro Margherita & Gianluca Elia & Claudio Petti, 2022. "What Is Quality in Research? Building a Framework of Design, Process and Impact Attributes and Evaluation Perspectives," Sustainability, MDPI, vol. 14(5), pages 1-18, March.
    9. Bornmann, Lutz, 2014. "Validity of altmetrics data for measuring societal impact: A study using data from Altmetric and F1000Prime," Journal of Informetrics, Elsevier, vol. 8(4), pages 935-950.
    10. Lutz Bornmann & Werner Marx, 2014. "How should the societal impact of research be generated and measured? A proposal for a simple and practicable approach to allow interdisciplinary comparisons," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 211-219, January.
    11. Juha-Pekka Lauronen, 2022. "Tension in Interpretations of the Social Impact of the Social Sciences: Walking a Tightrope Between Divergent Conceptualizations of Research Utilization," SAGE Open, , vol. 12(2), pages 21582440221, April.
    12. Han Zheng & L. G. Pee & Dan Zhang, 2021. "Societal impact of research: a text mining study of impact types," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 7397-7417, September.
    13. Sander C. S. Clahsen & Irene van Kamp & Betty C. Hakkert & Theo G. Vermeire & Aldert H. Piersma & Erik Lebret, 2019. "Why Do Countries Regulate Environmental Health Risks Differently? A Theoretical Perspective," Risk Analysis, John Wiley & Sons, vol. 39(2), pages 439-461, February.
    14. Keungoui Kim & Dieter F. Kogler & Sira Maliphol, 2024. "Identifying interdisciplinary emergence in the science of science: combination of network analysis and BERTopic," Palgrave Communications, Palgrave Macmillan, vol. 11(1), pages 1-15, December.
    15. Lin Zhang & Gunnar Sivertsen & Huiying Du & Ying Huang & Wolfgang Glänzel, 2021. "Gender differences in the aims and impacts of research," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(11), pages 8861-8886, November.
    16. Salter, Ammon & Salandra, Rossella & Walker, James, 2017. "Exploring preferences for impact versus publications among UK business and management academics," Research Policy, Elsevier, vol. 46(10), pages 1769-1782.
    17. Oscar LLOPIS & Joaquin AZAGRA-CARO, 2015. "Who do you care about? Scientists’ personality traits and perceived beneficiary impact," Cahiers du GREThA (2007-2019) 2015-29, Groupe de Recherche en Economie Théorique et Appliquée (GREThA).
    18. Junwen Luo & Lai Ma & Kalpana Shankar, 2021. "Does the inclusion of non-academic reviewers make any difference for grant impact panels? [Understanding the Long Term Impact of the Framework Programme, European Policy Evaluation Consortium (EPEC," Science and Public Policy, Oxford University Press, vol. 48(6), pages 763-775.
    19. Pita Spruijt & Anne B. Knol & Arthur C. Petersen & Erik Lebret, 2019. "Expert Views on Their Role as Policy Advisor: Pilot Study for the Cases of Electromagnetic Fields, Particulate Matter, and Antimicrobial Resistance," Risk Analysis, John Wiley & Sons, vol. 39(5), pages 968-974, May.
    20. J. Britt Holbrook, 2017. "The future of the impact agenda depends on the revaluation of academic freedom," Palgrave Communications, Palgrave Macmillan, vol. 3(1), pages 1-9, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:109:y:2016:i:3:d:10.1007_s11192-016-2115-y. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.