IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v18y2024i3s1751157724000439.html
   My bibliography  Save this article

Societal and scientific impact of policy research: A large-scale empirical study of some explanatory factors using Altmetric and Overton

Author

Listed:
  • Dorta-González, Pablo
  • Rodríguez-Caro, Alejandro
  • Dorta-González, María Isabel

Abstract

This study investigates how scientific research influences policymaking by analyzing citations of research articles in policy documents (policy impact) for nearly 125,000 articles across 434 public policy journals. We reveal distinct citation patterns between policymakers and other stakeholders like researchers, journalists, and the public. News and blog mentions, social media engagement, and open access publications (excluding fully open access) significantly increase the likelihood of a research article being cited in policy documents. Conversely, articles locked behind paywalls and those published under the full open access model (based on Altmetric data) have a lower chance of being policy-cited. Publication year and policy type show no significant influence. Our findings emphasize the crucial role of science communication channels like news media and social media in bridging the gap between research and policy. Interestingly, academic citations hold a weaker influence on policy citations compared to news mentions, suggesting a potential disconnect between how researchers reference research and how policymakers utilize it. This highlights the need for improved communication strategies to ensure research informs policy decisions more effectively. This study provides valuable insights for researchers, policymakers, and science communicators. Researchers can tailor their dissemination efforts to reach policymakers through media channels. Policymakers can leverage these findings to identify research with higher policy relevance. Science communicators can play a critical role in translating research for policymakers and fostering dialogue between the scientific and policymaking communities.

Suggested Citation

  • Dorta-González, Pablo & Rodríguez-Caro, Alejandro & Dorta-González, María Isabel, 2024. "Societal and scientific impact of policy research: A large-scale empirical study of some explanatory factors using Altmetric and Overton," Journal of Informetrics, Elsevier, vol. 18(3).
    • Handle: RePEc:eee:infome:v:18:y:2024:i:3:s1751157724000439
    DOI: 10.1016/j.joi.2024.101530
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157724000439
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2024.101530?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Joly, P.B. & Gaunand, A. & Colinet, L. & Larédo, P. & Lemarié, S. & Matt, M., 2015. "ASIRPA: a comprehensive theory-based approach to assessing the societal impacts of a research organization," Working Papers 2015-04, Grenoble Applied Economics Laboratory (GAEL).
    2. Mostafa Mesgari & Chitu Okoli & Mohamad Mehdi & Finn Årup Nielsen & Arto Lanamäki, 2015. "“The sum of all human knowledge”: A systematic review of scholarly research on the content of Wikipedia," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(2), pages 219-245, February.
    3. Hanan Khazragui & John Hudson, 2015. "Measuring the benefits of university research: impact and the REF in the UK," Research Evaluation, Oxford University Press, vol. 24(1), pages 51-62.
    4. Hadas Shema & Judit Bar-Ilan & Mike Thelwall, 2015. "How is research blogged? A content analysis approach," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(6), pages 1136-1149, June.
    5. Yu, Houqiang & Xu, Shenmeng & Xiao, Tingting & Hemminger, Brad M. & Yang, Siluo, 2017. "Global science discussed in local altmetrics: Weibo and its comparison with Twitter," Journal of Informetrics, Elsevier, vol. 11(2), pages 466-482.
    6. Lutz Bornmann, 2013. "What is societal impact of research and how can it be assessed? a literature survey," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(2), pages 217-233, February.
    7. Sarah de Rijcke & Paul F. Wouters & Alex D. Rushforth & Thomas P. Franssen & Björn Hammarfelt, 2016. "Evaluation practices and effects of indicator use—a literature review," Research Evaluation, Oxford University Press, vol. 25(2), pages 161-169.
    8. Ozili, Peterson K, 2023. "The acceptable R-square in empirical modelling for social science research," MPRA Paper 115769, University Library of Munich, Germany.
    9. Robin Haunschild & Lutz Bornmann, 2017. "How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(3), pages 1209-1216, March.
    10. Lutz Bornmann & Robin Haunschild & Werner Marx, 2016. "Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents?," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1477-1495, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bornmann, Lutz & Haunschild, Robin & Adams, Jonathan, 2019. "Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF)," Journal of Informetrics, Elsevier, vol. 13(1), pages 325-340.
    2. Hashem Atapour & Robabeh Maddahi & Rasoul Zavaraqi, 2024. "Policy citations of scientometric articles: an altmetric study," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(7), pages 4423-4436, July.
    3. Andrea Bonaccorsi & Filippo Chiarello & Gualtiero Fantoni, 2021. "Impact for whom? Mapping the users of public research with lexicon-based text mining," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1745-1774, February.
    4. Houqiang Yu & Xueting Cao & Tingting Xiao & Zhenyi Yang, 2020. "How accurate are policy document mentions? A first look at the role of altmetrics database," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1517-1540, November.
    5. Jorrit P Smit & Laurens K Hessels, 2021. "The production of scientific and societal value in research evaluation: a review of societal impact assessment methods [Systems Thinking, Knowledge and Action: Towards Better Models and Methods]," Research Evaluation, Oxford University Press, vol. 30(3), pages 323-335.
    6. Dotti, Nicola Francesco & Walczyk, Julia, 2022. "What is the societal impact of university research? A policy-oriented review to map approaches, identify monitoring methods and success factors," Evaluation and Program Planning, Elsevier, vol. 95(C).
    7. Nathalie Taverdet-Popiolek, 2022. "Economic Footprint of a Large French Research and Technology Organisation in Europe: Deciphering a Simplified Model and Appraising the Results," Journal of the Knowledge Economy, Springer;Portland International Center for Management of Engineering and Technology (PICMET), vol. 13(1), pages 44-69, March.
    8. Marco Gatti, 2018. "The Impact of Management Accounting Research: An Analysis of the Past and a Look at the Future," International Journal of Business and Management, Canadian Center of Science and Education, vol. 13(5), pages 1-47, March.
    9. Lutz Bornmann & Robin Haunschild & Werner Marx, 2016. "Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents?," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1477-1495, December.
    10. Robin Haunschild & Lutz Bornmann, 2017. "How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(3), pages 1209-1216, March.
    11. Knut Blind & Alex Fenton, 2022. "Standard-relevant publications: evidence, processes and influencing factors," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(1), pages 577-602, January.
    12. D’Este, Pablo & Robinson-García, Nicolás, 2023. "Interdisciplinary research and the societal visibility of science: The advantages of spanning multiple and distant scientific fields," Research Policy, Elsevier, vol. 52(2).
    13. Arnott, James C., 2021. "Pens and purse strings: Exploring the opportunities and limits to funding actionable sustainability science," Research Policy, Elsevier, vol. 50(10).
    14. Alessandro Margherita & Gianluca Elia & Claudio Petti, 2022. "What Is Quality in Research? Building a Framework of Design, Process and Impact Attributes and Evaluation Perspectives," Sustainability, MDPI, vol. 14(5), pages 1-18, March.
    15. Cui Huang & Chao Yang & Jun Su, 2018. "Policy change analysis based on “policy target–policy instrument” patterns: a case study of China’s nuclear energy policy," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(2), pages 1081-1114, November.
    16. Liang Hu & Win-bin Huang & Yi Bu, 2024. "Interdisciplinary research attracts greater attention from policy documents: evidence from COVID-19," Palgrave Communications, Palgrave Macmillan, vol. 11(1), pages 1-10, December.
    17. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    18. Dorte Drongstrup & Shafaq Malik & Naif Radi Aljohani & Salem Alelyani & Iqra Safder & Saeed-Ul Hassan, 2020. "Can social media usage of scientific literature predict journal indices of AJG, SNIP and JCR? An altmetric study of economics," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1541-1558, November.
    19. Daniela Filippo & Pablo Sastrón-Toledo, 2023. "Influence of research on open science in the public policy sphere," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(3), pages 1995-2017, March.
    20. Perkmann, Markus & Salandra, Rossella & Tartari, Valentina & McKelvey, Maureen & Hughes, Alan, 2021. "Academic engagement: A review of the literature 2011-2019," Research Policy, Elsevier, vol. 50(1).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:18:y:2024:i:3:s1751157724000439. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.