IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v18y2024i3s1751157724000750.html
   My bibliography  Save this article

SMIAltmetric: A comprehensive metric for evaluating social media impact of scientific papers on Twitter (X)

Author

Listed:
  • Wang, Zuzheng
  • Lu, Yongxu
  • Zhou, Yuanyuan
  • Ji, Jiaojiao

Abstract

The rise of social media has significantly influenced scholarly communication, knowledge dissemination, and research evaluation, leading to the enrichment of alternative metrics (altmetrics) for evaluating academic papers’ social impact, which assesses the social impact of academic papers through online activities, including reading, bookmarking, downloading, and commenting. However, these altmetrics often focus on the number of mentions on social media rather than thoroughly evaluating the source, content, and dissemination of these mentions. To address this gap, this study introduces the social media impact altmetric (SMIAltmetric), which is based on 44,087 publications and 860,680 tweets (now “posts”), a comprehensive scoring system for evaluating scientific papers on Twitter (now “X”), using diverse features, including literature-related, social media engagement-related, user-related, and content-related features. Employing Altmetric Attention Acores (AAS) as labels, we tested eight machine learning algorithms, with XGBoost demonstrating the highest accuracy at 0.8672. Crucial factors influencing SMIAltmetric, as identified by the SHAP value, were followers, retweets, mentions, and citation. Furthermore, consistency analysis and convergent validation between the proposed SMIAltmetric and AAS confirm the reliability and finer differentiation of SMIAltmetric. The proposed SMIAltmetric provides a more comprehensive understanding of a paper’s social media impact, enhancing the evaluation of scientific discourse and its engagement with society.

Suggested Citation

  • Wang, Zuzheng & Lu, Yongxu & Zhou, Yuanyuan & Ji, Jiaojiao, 2024. "SMIAltmetric: A comprehensive metric for evaluating social media impact of scientific papers on Twitter (X)," Journal of Informetrics, Elsevier, vol. 18(3).
  • Handle: RePEc:eee:infome:v:18:y:2024:i:3:s1751157724000750
    DOI: 10.1016/j.joi.2024.101562
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157724000750
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.joi.2024.101562?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Enrique Orduña-Malea & Rodrigo Costas, 2021. "Link-based approach to study scientific software usage: the case of VOSviewer," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 8153-8186, September.
    2. Isabella Peters & Peter Kraker & Elisabeth Lex & Christian Gumpenberger & Juan Gorraiz, 2016. "Research data explored: an extended analysis of citations and altmetrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(2), pages 723-744, May.
    3. Cassidy R. Sugimoto & Sam Work & Vincent Larivière & Stefanie Haustein, 2017. "Scholarly use of social media and altmetrics: A review of the literature," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(9), pages 2037-2062, September.
    4. Zhichao Fang & Rodrigo Costas & Paul Wouters, 2022. "User engagement with scholarly tweets of scientific papers: a large-scale and cross-disciplinary analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4523-4546, August.
    5. Samara Klar & Yanna Krupnikov & John Barry Ryan & Kathleen Searles & Yotam Shmargad, 2020. "Using social media to promote academic research: Identifying the benefits of twitter for sharing academic work," PLOS ONE, Public Library of Science, vol. 15(4), pages 1-15, April.
    6. Stefanie Haustein & Timothy D. Bowman & Kim Holmberg & Andrew Tsou & Cassidy R. Sugimoto & Vincent Larivière, 2016. "Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 67(1), pages 232-238, January.
    7. Saeed-Ul Hassan & Mubashir Imran & Uzair Gillani & Naif Radi Aljohani & Timothy D. Bowman & Fereshteh Didegah, 2017. "Measuring social media activity of scientific literature: an exhaustive comparison of scopus and novel altmetrics big data," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(2), pages 1037-1057, November.
    8. Yian Yin & Yuxiao Dong & Kuansan Wang & Dashun Wang & Benjamin F. Jones, 2022. "Public use and public funding of science," Nature Human Behaviour, Nature, vol. 6(10), pages 1344-1350, October.
    9. Zohreh Zahedi & Rodrigo Costas & Paul Wouters, 2017. "Mendeley readership as a filtering tool to identify highly cited publications," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 68(10), pages 2511-2521, October.
    10. Bornmann, Lutz, 2014. "Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics," Journal of Informetrics, Elsevier, vol. 8(4), pages 895-903.
    11. Adam Dinsmore & Liz Allen & Kevin Dolby, 2014. "Alternative Perspectives on Impact: The Potential of ALMs and Altmetrics to Inform Funders about Research Impact," PLOS Biology, Public Library of Science, vol. 12(11), pages 1-4, November.
    12. Emanuela Reale & Dragana Avramov & Kubra Canhial & Claire Donovan & Ramon Flecha & Poul Holm & Charles Larkin & Benedetto Lepori & Judith Mosoni-Fried & Esther Oliver & Emilia Primeri & Lidia Puigvert, 2018. "A review of literature on evaluating the scientific, social and political impact of social sciences and humanities research," Research Evaluation, Oxford University Press, vol. 27(4), pages 298-308.
    13. Ying Guo & Xiantao Xiao, 2022. "Author-level altmetrics for the evaluation of Chinese scholars," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(2), pages 973-990, February.
    14. Ortega, José Luis, 2020. "Proposal of composed altmetric indicators based on prevalence and impact dimensions," Journal of Informetrics, Elsevier, vol. 14(4).
    15. Zhichao Fang & Rodrigo Costas & Wencan Tian & Xianwen Wang & Paul Wouters, 2020. "An extensive analysis of the presence of altmetric data for Web of Science publications across subject fields and research topics," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(3), pages 2519-2549, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Enrique Orduña-Malea & Cristina I. Font-Julián, 2022. "Are patents linked on Twitter? A case study of Google patents," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(11), pages 6339-6362, November.
    2. González-Betancor, Sara M. & Dorta-González, Pablo, 2023. "Does society show differential attention to researchers based on gender and field?," Journal of Informetrics, Elsevier, vol. 17(4).
    3. Daniel Torres-Salinas & Domingo Docampo & Wenceslao Arroyo-Machado & Nicolas Robinson-Garcia, 2024. "The many publics of science: using altmetrics to identify common communication channels by scientific field," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(7), pages 3705-3723, July.
    4. Sergio Copiello, 2020. "Other than detecting impact in advance, alternative metrics could act as early warning signs of retractions: tentative findings of a study into the papers retracted by PLoS ONE," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2449-2469, December.
    5. Ying Guo & Xiantao Xiao, 2022. "Author-level altmetrics for the evaluation of Chinese scholars," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(2), pages 973-990, February.
    6. Wang, Zhiqi & Chen, Yue & Glänzel, Wolfgang, 2020. "Preprints as accelerator of scholarly communication: An empirical analysis in Mathematics," Journal of Informetrics, Elsevier, vol. 14(4).
    7. Daniel Torres-Salinas & Nicolás Robinson-Garcia & Juan Gorraiz, 2017. "Filling the citation gap: measuring the multidimensional impact of the academic book at institutional level with PlumX," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(3), pages 1371-1384, December.
    8. Zhichao Fang & Jonathan Dudek & Rodrigo Costas, 2020. "The stability of Twitter metrics: A study on unavailable Twitter mentions of scientific publications," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 71(12), pages 1455-1469, December.
    9. Yingxin Estella Ye & Jin-Cheon Na & Poong Oh, 2022. "Are automated accounts driving scholarly communication on Twitter? a case study of dissemination of COVID-19 publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(5), pages 2151-2172, May.
    10. Solanki Gupta & Vivek Kumar Singh & Sumit Kumar Banshal, 2024. "Altmetric data quality analysis using Benford’s law," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(7), pages 4597-4621, July.
    11. Zhichao Fang & Rodrigo Costas & Wencan Tian & Xianwen Wang & Paul Wouters, 2020. "An extensive analysis of the presence of altmetric data for Web of Science publications across subject fields and research topics," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(3), pages 2519-2549, September.
    12. Saeed-Ul Hassan & Timothy D. Bowman & Mudassir Shabbir & Aqsa Akhtar & Mubashir Imran & Naif Radi Aljohani, 2019. "Influential tweeters in relation to highly cited articles in altmetric big data," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(1), pages 481-493, April.
    13. Anwar Said & Timothy D. Bowman & Rabeeh Ayaz Abbasi & Naif Radi Aljohani & Saeed-Ul Hassan & Raheel Nawaz, 2019. "Mining network-level properties of Twitter altmetrics data," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(1), pages 217-235, July.
    14. Lu Liu & Benjamin F. Jones & Brian Uzzi & Dashun Wang, 2023. "Data, measurement and empirical methods in the science of science," Nature Human Behaviour, Nature, vol. 7(7), pages 1046-1058, July.
    15. Xi Zhang & Xianhai Wang & Hongke Zhao & Patricia Ordóñez de Pablos & Yongqiang Sun & Hui Xiong, 2019. "An effectiveness analysis of altmetrics indices for different levels of artificial intelligence publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(3), pages 1311-1344, June.
    16. Yajie Wang & Alesia Zuccala, 2021. "Scholarly book publishers as publicity agents for SSH titles on Twitter," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(6), pages 4817-4840, June.
    17. Chan, Ho Fai & Önder, Ali Sina & Schweitzer, Sascha & Torgler, Benno, 2023. "Twitter and citations," Economics Letters, Elsevier, vol. 231(C).
    18. Yang, Siluo & Zheng, Mengxue & Yu, Yonghao & Wolfram, Dietmar, 2021. "Are Altmetric.com scores effective for research impact evaluation in the social sciences and humanities?," Journal of Informetrics, Elsevier, vol. 15(1).
    19. Yaxue Ma & Zhichao Ba & Yuxiang Zhao & Jin Mao & Gang Li, 2021. "Understanding and predicting the dissemination of scientific papers on social media: a two-step simultaneous equation modeling–artificial neural network approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(8), pages 7051-7085, August.
    20. Mike Thelwall, 2018. "Early Mendeley readers correlate with later citation counts," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(3), pages 1231-1240, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:18:y:2024:i:3:s1751157724000750. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.