IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0197133.html
   My bibliography  Save this article

Do altmetrics correlate with the quality of papers? A large-scale empirical study based on F1000Prime data

Author

Listed:
  • Lutz Bornmann
  • Robin Haunschild

Abstract

In this study, we address the question whether (and to what extent, respectively) altmetrics are related to the scientific quality of papers (as measured by peer assessments). Only a few studies have previously investigated the relationship between altmetrics and assessments by peers. In the first step, we analyse the underlying dimensions of measurement for traditional metrics (citation counts) and altmetrics–by using principal component analysis (PCA) and factor analysis (FA). In the second step, we test the relationship between the dimensions and quality of papers (as measured by the post-publication peer-review system of F1000Prime assessments)–using regression analysis. The results of the PCA and FA show that altmetrics operate along different dimensions, whereas Mendeley counts are related to citation counts, and tweets form a separate dimension. The results of the regression analysis indicate that citation-based metrics and readership counts are significantly more related to quality, than tweets. This result on the one hand questions the use of Twitter counts for research evaluation purposes and on the other hand indicates potential use of Mendeley reader counts.

Suggested Citation

  • Lutz Bornmann & Robin Haunschild, 2018. "Do altmetrics correlate with the quality of papers? A large-scale empirical study based on F1000Prime data," PLOS ONE, Public Library of Science, vol. 13(5), pages 1-12, May.
    • Handle: RePEc:plo:pone00:0197133
    DOI: 10.1371/journal.pone.0197133
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0197133
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0197133&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0197133?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Mojisola Erdt & Aarthy Nagarajan & Sei-Ching Joanna Sin & Yin-Leng Theng, 2016. "Altmetrics: an analysis of the state-of-the-art in measuring research impact on social media," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(2), pages 1117-1166, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Cristina López-Duarte & Jane F. Maley & Marta M. Vidal-Suárez, 2021. "Main challenges to international student mobility in the European arena," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(11), pages 8957-8980, November.
    2. Ying Guo & Xiantao Xiao, 2022. "Author-level altmetrics for the evaluation of Chinese scholars," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(2), pages 973-990, February.
    3. Jianhua Hou & Xiucai Yang & Yang Zhang, 2023. "The effect of social media knowledge cascade: an analysis of scientific papers diffusion," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(9), pages 5169-5195, September.
    4. Sergio Copiello, 2019. "Research Interest: another undisclosed (and redundant) algorithm by ResearchGate," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(1), pages 351-360, July.
    5. Bai, Xiaomei & Zhang, Fuli & Lee, Ivan, 2019. "Predicting the citations of scholarly paper," Journal of Informetrics, Elsevier, vol. 13(1), pages 407-418.
    6. Parreiras, R.O. & Kokshenev, I. & Carvalho, M.O.M. & Willer, A.C.M. & Dellezzopolles, C.F. & Nacif, D.B. & Santana, J.A., 2019. "A flexible multicriteria decision-making methodology to support the strategic management of Science, Technology and Innovation research funding programs," European Journal of Operational Research, Elsevier, vol. 272(2), pages 725-739.
    7. Łukasz Wiechetek & Zbigniew Pastuszak, 2022. "Academic social networks metrics: an effective indicator for university performance?," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(3), pages 1381-1401, March.
    8. Sergio Copiello & Pietro Bonifaci, 2019. "ResearchGate Score, full-text research items, and full-text reads: a follow-up study," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(2), pages 1255-1262, May.
    9. Barbara McGillivray & Paola Marongiu & Nilo Pedrazzini & Marton Ribary & Mandy Wigdorowitz & Eleonora Zordan, 2022. "Deep Impact: A Study on the Impact of Data Papers and Datasets in the Humanities and Social Sciences," Publications, MDPI, vol. 10(4), pages 1-40, October.
    10. Siluo Yang & Xin Xing & Dietmar Wolfram, 2018. "Difference in the impact of open-access papers published by China and the USA," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(2), pages 1017-1037, May.
    11. Feiheng Luo & Aixin Sun & Mojisola Erdt & Aravind Sesagiri Raamkumar & Yin-Leng Theng, 2018. "Exploring prestigious citations sourced from top universities in bibliometrics and altmetrics: a case study in the computer science discipline," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(1), pages 1-17, January.
    12. Sergio Copiello & Pietro Bonifaci, 2018. "A few remarks on ResearchGate score and academic reputation," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(1), pages 301-306, January.
    13. Hans Jonker & Florian Vanlee & Walter Ysebaert, 2022. "Societal impact of university research in the written press: media attention in the context of SIUR and the open science agenda among social scientists in Flanders, Belgium," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(12), pages 7289-7306, December.
    14. Rongying Zhao & Mingkun Wei, 2017. "Academic impact evaluation of Wechat in view of social media perspective," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(3), pages 1777-1791, September.
    15. Sergio Copiello, 2020. "The alleged citation advantage of video abstracts may be a matter of self-citations and self-selection bias. Comment on “The impact of video abstract on citation counts” by Zong et al," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(1), pages 751-757, January.
    16. Jianhua Hou & Da Ma, 2020. "How the high-impact papers formed? A study using data from social media and citation," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2597-2615, December.
    17. Pavol Prokop & Rudolf Masarovič & Sandra Hajdúchová & Zuzana Ježová & Martina Zvaríková & Peter Fedor, 2022. "Prioritisation of Charismatic Animals in Major Conservation Journals Measured by the Altmetric Attention Score," Sustainability, MDPI, vol. 14(24), pages 1-10, December.
    18. Houqiang Yu & Xinyun Yu & Xueting Cao, 2022. "How accurate are news mentions of scholarly output? A content analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(7), pages 4075-4096, July.
    19. Dimity Stephen & Stephan Stahlschmidt, 2024. "Contrasting cross-correlation: Meta-analyses of the associations between citations and 13 altmetrics, incorporating moderating variables," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(10), pages 6049-6063, October.
    20. Daniela De Filippo & Fernanda Morillo & Borja González-Albo, 2023. "Measuring the Impact and Influence of Scientific Activity in the Humanities and Social Sciences," Publications, MDPI, vol. 11(2), pages 1-17, June.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0197133. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.