IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v127y2022i11d10.1007_s11192-022-04510-7.html
   My bibliography  Save this article

Identifying science in the news: An assessment of the precision and recall of Altmetric.com news mention data

Author

Listed:
  • Alice Fleerackers

    (Simon Fraser University)

  • Lise Nehring

    (University of Victoria)

  • Lauren A. Maggio

    (Uniformed Services University)

  • Asura Enkhbayar

    (Simon Fraser University)

  • Laura Moorhead

    (San Francisco State University)

  • Juan Pablo Alperin

    (Simon Fraser University)

Abstract

The company Altmetric is often used to collect mentions of research in online news stories, yet there have been concerns about the quality of this data. This study investigates these concerns. Using a manual content analysis of 400 news stories as a comparison method, we analyzed the precision and recall with which Altmetric identified mentions of research in 8 news outlets. We also used logistic regression to identify the characteristics of research mentions that influence their likelihood of being successfully identified. We find that, for a predefined set of outlets, Altmetric’s news mention data were relatively accurate (F-score = 0.80), with very high precision (0.95) and acceptable recall (0.70), although recall is below 0.50 for some news outlets. Altmetric is more likely to successfully identify mentions of research that include a hyperlink to the research item, an author name, and/or the title of a publication venue. This data source appears to be less reliable for mentions of research that provide little or no bibliometric information, as well as for identifying mentions of scholarly monographs, conference presentations, dissertations, and non-English research articles. Our findings suggest that, with caveats, scholars can use Altmetric news mention data as a relatively reliable source to identify research mentions across a range of outlets with high precision and acceptable recall, offering scholars the potential to conserve resources during data collection. Our study does not, however, offer an assessment of completeness or accuracy of Altmetric news data overall.

Suggested Citation

  • Alice Fleerackers & Lise Nehring & Lauren A. Maggio & Asura Enkhbayar & Laura Moorhead & Juan Pablo Alperin, 2022. "Identifying science in the news: An assessment of the precision and recall of Altmetric.com news mention data," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(11), pages 6109-6123, November.
    • Handle: RePEc:spr:scient:v:127:y:2022:i:11:d:10.1007_s11192-022-04510-7
    DOI: 10.1007/s11192-022-04510-7
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-022-04510-7
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-022-04510-7?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Mousumi Karmakar & Sumit Kumar Banshal & Vivek Kumar Singh, 2021. "A large-scale comparison of coverage and mentions captured by the two altmetric aggregators: Altmetric.com and PlumX," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(5), pages 4465-4489, May.
    2. Diana Hicks, 2005. "The Four Literatures Of Social Sciences," IBT Journal of Business Studies (JBS), Ilma University, Faculty of Management Science, vol. 1(1), pages 1-1.
    3. Christine Meschede & Tobias Siebenlist, 2018. "Cross-metric compatability and inconsistencies of altmetrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(1), pages 283-297, April.
    4. Markus Lehmkuhl & Nikolai Promies, 2020. "Frequency distribution of journalistic attention for scientific studies and scientific sources: An input–output analysis," PLOS ONE, Public Library of Science, vol. 15(11), pages 1-20, November.
    5. José Luis Ortega, 2020. "Blogs and news sources coverage in altmetrics data providers: a comparative analysis by country, language, and subject," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(1), pages 555-572, January.
    6. Diana Hicks, 2005. "The Four Literatures Of Social Sciences," IBT Journal of Business Studies (JBS), Ilma University, Faculty of Management Science, vol. 1(1), pages 1-20.
    7. Ortega, José Luis, 2021. "How do media mention research papers? Structural analysis of blogs and news networks using citation coupling," Journal of Informetrics, Elsevier, vol. 15(3).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Daniel Torres-Salinas & Domingo Docampo & Wenceslao Arroyo-Machado & Nicolas Robinson-Garcia, 2024. "The many publics of science: using altmetrics to identify common communication channels by scientific field," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(7), pages 3705-3723, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mousumi Karmakar & Sumit Kumar Banshal & Vivek Kumar Singh, 2021. "A large-scale comparison of coverage and mentions captured by the two altmetric aggregators: Altmetric.com and PlumX," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(5), pages 4465-4489, May.
    2. Qingzhou Luo & Jianhong Cecilia Xia & Gaby Haddow & Michele Willson & Jun Yang, 2018. "Does distance hinder the collaboration between Australian universities in the humanities, arts and social sciences?," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(2), pages 695-715, May.
    3. Galt, Ryan E. & Pinzón, Natalia & Robinson, Nicholas Ian & Baukloh Coronil, Marcela Beatriz, 2024. "Agroecology and the social sciences: A half-century systematic review," Agricultural Systems, Elsevier, vol. 216(C).
    4. Yu, Houqiang & Li, Longfei & Cao, Xueting & Chen, Tao, 2022. "Exploring country's preference over news mentions to academic papers," Journal of Informetrics, Elsevier, vol. 16(4).
    5. Steffen Lemke & Athanasios Mazarakis & Isabella Peters, 2021. "Conjoint analysis of researchers' hidden preferences for bibliometrics, altmetrics, and usage metrics," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 72(6), pages 777-792, June.
    6. Claudia N. González Brambila & José Luis Olivares-Vázquez, 2021. "Patterns and evolution of publication and co-authorship in Social Sciences in Mexico," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(3), pages 2595-2626, March.
    7. Nestor Gandelman & Osiris J. Parcero & Matilde Pereira & Flavia Roldán, 2021. "Ventajas comparativas reveladas en disciplinas científicas y tecnológicas en Uruguay," Documentos de Investigación 125, Universidad ORT Uruguay. Facultad de Administración y Ciencias Sociales.
    8. Sergio Copiello, 2020. "Other than detecting impact in advance, alternative metrics could act as early warning signs of retractions: tentative findings of a study into the papers retracted by PLoS ONE," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2449-2469, December.
    9. Pantea Kamrani & Isabelle Dorsch & Wolfgang G. Stock, 2021. "Do researchers know what the h-index is? And how do they estimate its importance?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 5489-5508, July.
    10. Ortega, José Luis, 2021. "How do media mention research papers? Structural analysis of blogs and news networks using citation coupling," Journal of Informetrics, Elsevier, vol. 15(3).
    11. Ortega, José Luis, 2020. "Proposal of composed altmetric indicators based on prevalence and impact dimensions," Journal of Informetrics, Elsevier, vol. 14(4).
    12. Zhichao Fang & Jonathan Dudek & Rodrigo Costas, 2020. "The stability of Twitter metrics: A study on unavailable Twitter mentions of scientific publications," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 71(12), pages 1455-1469, December.
    13. Enrique Orduña-Malea & Adolfo Alonso-Arroyo & José-Antonio Ontalba-Ruipérez & Ferrán Catalá-López, 2023. "Evaluating the online impact of reporting guidelines for randomised trial reports and protocols: a cross-sectional web-based data analysis of CONSORT and SPIRIT initiatives," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(1), pages 407-440, January.
    14. Daniel Torres-Salinas & Domingo Docampo & Wenceslao Arroyo-Machado & Nicolas Robinson-Garcia, 2024. "The many publics of science: using altmetrics to identify common communication channels by scientific field," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(7), pages 3705-3723, July.
    15. Yang, Siluo & Zheng, Mengxue & Yu, Yonghao & Wolfram, Dietmar, 2021. "Are Altmetric.com scores effective for research impact evaluation in the social sciences and humanities?," Journal of Informetrics, Elsevier, vol. 15(1).
    16. Yashan Li & Jinge Mao & Lin Zhang & Dongbo Wang & Si Shen & Ying Huang, 2022. "How scientific research incorporates policy: an examination using the case of China’s science and technology evaluation system," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(9), pages 5283-5306, September.
    17. Houqiang Yu & Xinyun Yu & Xueting Cao, 2022. "How accurate are news mentions of scholarly output? A content analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(7), pages 4075-4096, July.
    18. Houqiang Yu & Xueting Cao & Tingting Xiao & Zhenyi Yang, 2020. "How accurate are policy document mentions? A first look at the role of altmetrics database," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1517-1540, November.
    19. Latefa Ali Dardas & Malik Sallam & Amanda Woodward & Nadia Sweis & Narjes Sweis & Faleh A. Sawair, 2023. "Evaluating Research Impact Based on Semantic Scholar Highly Influential Citations, Total Citations, and Altmetric Attention Scores: The Quest for Refined Measures Remains Illusive," Publications, MDPI, vol. 11(1), pages 1-16, January.
    20. Ramani, Ravi S. & Aguinis, Herman & Coyle-Shapiro, Jacqueline A.M., 2022. "Defining, measuring, and rewarding scholarly impact: mind the level of analysis," LSE Research Online Documents on Economics 117286, London School of Economics and Political Science, LSE Library.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:127:y:2022:i:11:d:10.1007_s11192-022-04510-7. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.