IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v114y2018i1d10.1007_s11192-017-2559-8.html
   My bibliography  Save this article

And now for something completely different: the congruence of the Altmetric Attention Score’s structure between different article groups

Author

Listed:
  • Bhaskar Mukherjee

    (Banaras Hindu University)

  • Siniša Subotić

    (University of Banja Luka
    CEON/CEES)

  • Ajay Kumar Chaubey

    (Banaras Hindu University)

Abstract

Altmetric Attention Score (AAS) is an increasingly popular composite altmetric measure, which is being criticized for an inappropriate and arbitrary aggregation of different altmetric sources into a single measure. We examined this issue empirically, by testing unidimensionality and the component structure congruence of the five ‘key’ AAS components: News, Blogs, Twitter, Facebook, and Google+. As a reference point, these tests were also done on different citation data: WoS, Scopus, and Google Scholar. All tests were done for groups of articles with: (1) high citations, but lower AAS (HCGs), and (2) high AAS, but lower citations (HAGs). Changes in component structures over time (from 2016 to 2017) were also considered. Citation data consistently formed congruent unidimensional structures for all groups and over time. Altmetric data formed congruent unidimensional structures only for the HCGs, with much inconsistency for the HAGs (including change over time). The relationship between Twitter and News counts was shown to be curvilinear. It was not possible to obtain a satisfactory congruent and reliable linear unidimensional altmetric structure between the groups for any variable combination, even after Mendeley and CiteULike altmetric counts were included. Correlations of altmetric aggregates and citations were fairly inconsistent between the groups. We advise against the usage of composite altmetric measures (including the AAS) for any group comparison purposes, until the measurement invariance issues are dealt with. The underlying pattern of associations between individual altmetrics is likely too complex and inconsistent across conditions to justify them being simply aggregated into a single score.

Suggested Citation

  • Bhaskar Mukherjee & Siniša Subotić & Ajay Kumar Chaubey, 2018. "And now for something completely different: the congruence of the Altmetric Attention Score’s structure between different article groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(1), pages 253-275, January.
    • Handle: RePEc:spr:scient:v:114:y:2018:i:1:d:10.1007_s11192-017-2559-8
    DOI: 10.1007/s11192-017-2559-8
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-017-2559-8
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-017-2559-8?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Christian Gumpenberger & Wolfgang Glänzel & Juan Gorraiz, 2016. "The ecstasy and the agony of the altmetric score," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(2), pages 977-982, August.
    2. Philippe Mongeon & Adèle Paul-Hus, 2016. "The journal coverage of Web of Science and Scopus: a comparative analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 106(1), pages 213-228, January.
    3. Liwen Vaughan & Debora Shaw, 2005. "Web citation data for impact assessment: A comparison of four science disciplines," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 56(10), pages 1075-1087, August.
    4. Pardeep Sud & Mike Thelwall, 2014. "Evaluating altmetrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(2), pages 1131-1143, February.
    5. John Horn, 1965. "A rationale and test for the number of factors in factor analysis," Psychometrika, Springer;The Psychometric Society, vol. 30(2), pages 179-185, June.
    6. Stefanie Haustein & Isabella Peters & Judit Bar-Ilan & Jason Priem & Hadas Shema & Jens Terliesner, 2014. "Coverage and adoption of altmetrics sources in the bibliometric community," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(2), pages 1145-1163, November.
    7. Mojisola Erdt & Aarthy Nagarajan & Sei-Ching Joanna Sin & Yin-Leng Theng, 2016. "Altmetrics: an analysis of the state-of-the-art in measuring research impact on social media," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(2), pages 1117-1166, November.
    8. Wolfgang Glänzel & Juan Gorraiz, 2015. "Usage metrics versus altmetrics: confusing terminology?," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(3), pages 2161-2164, March.
    9. Ortega, José Luis, 2015. "Relationship between altmetric and bibliometric indicators across academic social sites: The case of CSIC's members," Journal of Informetrics, Elsevier, vol. 9(1), pages 39-49.
    10. William Meredith, 1993. "Measurement invariance, factor analysis and factorial invariance," Psychometrika, Springer;The Psychometric Society, vol. 58(4), pages 525-543, December.
    11. Lutz Bornmann, 2015. "Alternative metrics in scientometrics: a meta-analysis of research into three altmetrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(3), pages 1123-1144, June.
    12. Juan Gorraiz & Christian Gumpenberger & Christian Schlögl, 2014. "Usage versus citation behaviours in four subject areas," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(2), pages 1077-1095, November.
    13. Wayne Velicer, 1976. "Determining the number of components from the matrix of partial correlations," Psychometrika, Springer;The Psychometric Society, vol. 41(3), pages 321-327, September.
    14. Bornmann, Lutz, 2014. "Validity of altmetrics data for measuring societal impact: A study using data from Altmetric and F1000Prime," Journal of Informetrics, Elsevier, vol. 8(4), pages 935-950.
    15. Nuredini, Kaltrina & Peters, Isabella, 2016. "Enriching the knowledge of altmetrics studies by exploring social media metrics for Economic and Business Studies journals," EconStor Conference Papers 146879, ZBW - Leibniz Information Centre for Economics.
    16. Richard Zinbarg & William Revelle & Iftah Yovel & Wen Li, 2005. "Cronbach’s α, Revelle’s β, and Mcdonald’s ω H : their relations with each other and two alternative conceptualizations of reliability," Psychometrika, Springer;The Psychometric Society, vol. 70(1), pages 123-133, March.
    17. Xianwen Wang & Zhichao Fang & Xiaoling Sun, 2016. "Usage patterns of scholarly articles on Web of Science: a study on Web of Science usage count," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(2), pages 917-926, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ortega, José Luis, 2020. "Proposal of composed altmetric indicators based on prevalence and impact dimensions," Journal of Informetrics, Elsevier, vol. 14(4).
    2. Christine Meschede & Tobias Siebenlist, 2018. "Cross-metric compatability and inconsistencies of altmetrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(1), pages 283-297, April.
    3. Maryam Moshtagh & Tahereh Jowkar & Maryam Yaghtin & Hajar Sotudeh, 2023. "The moderating effect of altmetrics on the correlations between single and multi-faceted university ranking systems: the case of THE and QS vs. Nature Index and Leiden," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(1), pages 761-781, January.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sergio Copiello, 2020. "Other than detecting impact in advance, alternative metrics could act as early warning signs of retractions: tentative findings of a study into the papers retracted by PLoS ONE," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2449-2469, December.
    2. Maryam Moshtagh & Tahereh Jowkar & Maryam Yaghtin & Hajar Sotudeh, 2023. "The moderating effect of altmetrics on the correlations between single and multi-faceted university ranking systems: the case of THE and QS vs. Nature Index and Leiden," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(1), pages 761-781, January.
    3. Ying Guo & Xiantao Xiao, 2022. "Author-level altmetrics for the evaluation of Chinese scholars," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(2), pages 973-990, February.
    4. Bornmann, Lutz & Haunschild, Robin & Adams, Jonathan, 2019. "Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF)," Journal of Informetrics, Elsevier, vol. 13(1), pages 325-340.
    5. Sergio Copiello & Pietro Bonifaci, 2019. "ResearchGate Score, full-text research items, and full-text reads: a follow-up study," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(2), pages 1255-1262, May.
    6. Sergio Copiello & Pietro Bonifaci, 2018. "A few remarks on ResearchGate score and academic reputation," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(1), pages 301-306, January.
    7. Sergio Copiello, 2019. "Research Interest: another undisclosed (and redundant) algorithm by ResearchGate," Scientometrics, Springer;Akadémiai Kiadó, vol. 120(1), pages 351-360, July.
    8. Martín-Martín, Alberto & Orduna-Malea, Enrique & Delgado López-Cózar, Emilio, 2018. "Author-level metrics in the new academic profile platforms: The online behaviour of the Bibliometrics community," Journal of Informetrics, Elsevier, vol. 12(2), pages 494-509.
    9. Hou, Jianhua & Yang, Xiucai, 2020. "Social media-based sleeping beauties: Defining, identifying and features," Journal of Informetrics, Elsevier, vol. 14(2).
    10. Christine Meschede & Tobias Siebenlist, 2018. "Cross-metric compatability and inconsistencies of altmetrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(1), pages 283-297, April.
    11. Daniela De Filippo & Fernanda Morillo & Borja González-Albo, 2023. "Measuring the Impact and Influence of Scientific Activity in the Humanities and Social Sciences," Publications, MDPI, vol. 11(2), pages 1-17, June.
    12. Isidro F. Aguillo, 2020. "Altmetrics of the Open Access Institutional Repositories: a webometrics approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(3), pages 1181-1192, June.
    13. Francisco J. Conejo & Lawrence F. Cunningham & Clifford E. Young, 2020. "Revisiting the Brand Luxury Index: new empirical evidence and future directions," Journal of Brand Management, Palgrave Macmillan, vol. 27(1), pages 108-122, January.
    14. Carlos Miguel Lemos & Ross Joseph Gore & Ivan Puga-Gonzalez & F LeRon Shults, 2019. "Dimensionality and factorial invariance of religiosity among Christians and the religiously unaffiliated: A cross-cultural analysis based on the International Social Survey Programme," PLOS ONE, Public Library of Science, vol. 14(5), pages 1-36, May.
    15. Ana-Beatriz Hernández-Lara & Maria-Victoria Sánchez-Rebull & Angels Niñerola, 2021. "Six Sigma in Health Literature, What Matters?," IJERPH, MDPI, vol. 18(16), pages 1-13, August.
    16. Mojisola Erdt & Htet Htet Aung & Ashley Sara Aw & Charlie Rapple & Yin-Leng Theng, 2017. "Analysing researchers’ outreach efforts and the association with publication metrics: A case study of Kudos," PLOS ONE, Public Library of Science, vol. 12(8), pages 1-25, August.
    17. Liwei Zhang & Jue Wang, 2021. "What affects publications’ popularity on Twitter?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(11), pages 9185-9198, November.
    18. Salim Moussa, 2016. "A Comment on the Estimation of the Reliability of Multidimensional Marketing Constructs: A Store Personality Scale Application," Global Business Review, International Management Institute, vol. 17(5), pages 1125-1144, October.
    19. Łukasz Wiechetek & Zbigniew Pastuszak, 2022. "Academic social networks metrics: an effective indicator for university performance?," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(3), pages 1381-1401, March.
    20. Jeanne A. Teresi & Katja Ocepek-Welikson & John A. Toner & Marjorie Kleinman & Mildred Ramirez & Joseph P. Eimicke & Barry J. Gurland & Albert Siu, 2017. "Methodological Issues in Measuring Subjective Well-Being and Quality-of-Life: Applications to Assessment of Affect in Older, Chronically and Cognitively Impaired, Ethnically Diverse Groups Using the F," Applied Research in Quality of Life, Springer;International Society for Quality-of-Life Studies, vol. 12(2), pages 251-288, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:114:y:2018:i:1:d:10.1007_s11192-017-2559-8. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.