IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v129y2024i7d10.1007_s11192-024-05061-9.html
   My bibliography  Save this article

Altmetric data quality analysis using Benford’s law

Author

Listed:
  • Solanki Gupta

    (Banaras Hindu University)

  • Vivek Kumar Singh

    (Banaras Hindu University
    University of Delhi)

  • Sumit Kumar Banshal

    (Alliance University)

Abstract

Altmetrics, or alternative metrics, refer to the newer kind of events around scholarly articles, such as the number of times the article is read, tweeted, mentioned in blog posts etc. These metrics have gained a lot of popularity during last few years and are now being collected and used in several ways, ranging from early measure of article impact to a potential indicator of societal relevance of research. However, there are several studies which have cautioned about use of altmetrics on account of quality and reliability of altmetric data, as they may be more prone to manipulations and artificial inflations. This study proposes a framework based on application of Benford’s Law to evaluate the quality of altmetric data. A large sized altmetric data sample is considered and the fits with Benford’s Law are computed. The analysis is performed by doing plots of the empirical data distributions and the theoretical Benford's, and by employing relevant statistical measures and tests. Results for fit on first and second leading digit of altmetric data show conformity to Benford's distribution. To further explore the usefulness of the framework, the altmetric data is subjected to artificial manipulations through a systematic process and the fits to Benford’s law are reassessed to see if there are distortions. The results and analysis suggest that Benford’s Law based framework can be used to test the quality of altmetric data. Relevant implications of the research are discussed.

Suggested Citation

  • Solanki Gupta & Vivek Kumar Singh & Sumit Kumar Banshal, 2024. "Altmetric data quality analysis using Benford’s law," Scientometrics, Springer;Akadémiai Kiadó, vol. 129(7), pages 4597-4621, July.
    • Handle: RePEc:spr:scient:v:129:y:2024:i:7:d:10.1007_s11192-024-05061-9
    DOI: 10.1007/s11192-024-05061-9
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-024-05061-9
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-024-05061-9?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Mr. Jesus R Gonzalez-Garcia & Mr. Gonzalo C Pastor Campos, 2009. "Benford’s Law and Macroeconomic Data Quality," IMF Working Papers 2009/010, International Monetary Fund.
    2. Ausloos, Marcel & Castellano, Rosella & Cerqueti, Roy, 2016. "Regularities and discrepancies of credit default swaps: a data science approach through Benford's law," Chaos, Solitons & Fractals, Elsevier, vol. 90(C), pages 8-17.
    3. Banshal, Sumit Kumar & Gupta, Solanki & Lathabai, Hiran H & Singh, Vivek Kumar, 2022. "Power Laws in altmetrics: An empirical analysis," Journal of Informetrics, Elsevier, vol. 16(3).
    4. Huang, Yasheng & Niu, Zhiyong & Yang, Clair, 2020. "Testing firm-level data quality in China against Benford’s Law," Economics Letters, Elsevier, vol. 192(C).
    5. José Luis Ortega, 2016. "To be or not to be on Twitter, and its relationship with the tweeting and citation of research papers," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(2), pages 1353-1364, November.
    6. Roy Cerqueti & M. Maggi & J. Riccioni, 2022. "Statistical methods for decision support systems in finance: How Benford’s law predicts financial risk," Post-Print hal-03788859, HAL.
    7. Isabella Peters & Peter Kraker & Elisabeth Lex & Christian Gumpenberger & Juan Gorraiz, 2016. "Research data explored: an extended analysis of citations and altmetrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(2), pages 723-744, May.
    8. Michal Brzezinski, 2015. "Power laws in citation distributions: evidence from Scopus," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(1), pages 213-228, April.
    9. Alexandre Donizeti Alves & Horacio Hideki Yanasse & Nei Yoshihiro Soma, 2014. "Benford’s Law and articles of scientific journals: comparison of JCR® and Scopus data," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 173-184, January.
    10. Mike Thelwall, 2021. "Measuring Societal Impacts Of Research With Altmetrics? Common Problems And Mistakes," Journal of Economic Surveys, Wiley Blackwell, vol. 35(5), pages 1302-1314, December.
    11. Stefanie Haustein & Isabella Peters & Judit Bar-Ilan & Jason Priem & Hadas Shema & Jens Terliesner, 2014. "Coverage and adoption of altmetrics sources in the bibliometric community," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(2), pages 1145-1163, November.
    12. Zohreh Zahedi & Rodrigo Costas & Paul Wouters, 2014. "How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(2), pages 1491-1513, November.
    13. Mike Thelwall, 2018. "Early Mendeley readers correlate with later citation counts," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(3), pages 1231-1240, June.
    14. Stefanie Haustein & Timothy D. Bowman & Kim Holmberg & Andrew Tsou & Cassidy R. Sugimoto & Vincent Larivière, 2016. "Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 67(1), pages 232-238, January.
    15. Stefanie Haustein & Isabella Peters & Cassidy R. Sugimoto & Mike Thelwall & Vincent Larivière, 2014. "Tweeting biomedicine: An analysis of tweets and citations in the biomedical literature," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 65(4), pages 656-669, April.
    16. Riccioni, Jessica & Cerqueti, Roy, 2018. "Regular paths in financial markets: Investigating the Benford's law," Chaos, Solitons & Fractals, Elsevier, vol. 107(C), pages 186-194.
    17. Bornmann, Lutz, 2014. "Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics," Journal of Informetrics, Elsevier, vol. 8(4), pages 895-903.
    18. Ronald Snijder, 2016. "Revisiting an open access monograph experiment: measuring citations and tweets 5 years later," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1855-1875, December.
    19. Mousumi Karmakar & Sumit Kumar Banshal & Vivek Kumar Singh, 2020. "Does presence of social media plugins in a journal website result in higher social media attention of its research publications?," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(3), pages 2103-2143, September.
    20. Juan Miguel Campanario & María Angeles Coslado, 2011. "Benford’s law and citations, articles and impact factors of scientific journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 88(2), pages 421-432, August.
    21. Micha Kaiser, 2019. "Benford'S Law As An Indicator Of Survey Reliability—Can We Trust Our Data?," Journal of Economic Surveys, Wiley Blackwell, vol. 33(5), pages 1602-1618, December.
    22. Man Kit Cheung, 2013. "Altmetrics: Too soon for use in assessment," Nature, Nature, vol. 494(7436), pages 176-176, February.
    23. Hajar Sotudeh & Zahra Mazarei & Mahdieh Mirzabeigi, 2015. "CiteULike bookmarks are correlated to citations at journal and author levels in library and information science," Scientometrics, Springer;Akadémiai Kiadó, vol. 105(3), pages 2237-2248, December.
    24. Drahomira Herrmannova & Robert M. Patton & Petr Knoth & Christopher G. Stahl, 2018. "Do citations and readership identify seminal publications?," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(1), pages 239-262, April.
    25. Héctor Rubén Morales & Marcela Porporato & Nicolas Epelbaum, 2022. "Benford's law for integrity tests of high-volume databases: a case study of internal audit in a state-owned enterprise," Journal of Economics, Finance and Administrative Science, Emerald Group Publishing Limited, vol. 27(53), pages 154-174, February.
    26. Adam Marcus & Ivan Oransky, 2011. "The paper is not sacred," Nature, Nature, vol. 480(7378), pages 449-450, December.
    27. Bornmann, Lutz, 2014. "Validity of altmetrics data for measuring societal impact: A study using data from Altmetric and F1000Prime," Journal of Informetrics, Elsevier, vol. 8(4), pages 935-950.
    28. Horton, Joanne & Krishna Kumar, Dhanya & Wood, Anthony, 2020. "Detecting academic fraud using Benford law: The case of Professor James Hunton," Research Policy, Elsevier, vol. 49(8).
    29. Thelwall, Mike & Nevill, Tamara, 2018. "Could scientists use Altmetric.com scores to predict longer term citation counts?," Journal of Informetrics, Elsevier, vol. 12(1), pages 237-248.
    30. Ehsan Mohammadi & Mike Thelwall, 2014. "Mendeley readership altmetrics for the social sciences and humanities: Research evaluation and knowledge flows," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 65(8), pages 1627-1638, August.
    31. Hadas Shema & Judit Bar-Ilan & Mike Thelwall, 2014. "Do blog citations correlate with a higher number of future citations? Research blogs as a potential source for alternative metrics," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 65(5), pages 1018-1027, May.
    32. Rodrigo Costas & Zohreh Zahedi & Paul Wouters, 2015. "Do “altmetrics” correlate with citations? Extensive comparison of altmetric indicators with citations from a multidisciplinary perspective," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(10), pages 2003-2019, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Maryam Moshtagh & Tahereh Jowkar & Maryam Yaghtin & Hajar Sotudeh, 2023. "The moderating effect of altmetrics on the correlations between single and multi-faceted university ranking systems: the case of THE and QS vs. Nature Index and Leiden," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(1), pages 761-781, January.
    2. Sergio Copiello, 2020. "Other than detecting impact in advance, alternative metrics could act as early warning signs of retractions: tentative findings of a study into the papers retracted by PLoS ONE," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2449-2469, December.
    3. Banshal, Sumit Kumar & Gupta, Solanki & Lathabai, Hiran H & Singh, Vivek Kumar, 2022. "Power Laws in altmetrics: An empirical analysis," Journal of Informetrics, Elsevier, vol. 16(3).
    4. Ortega, José Luis, 2018. "The life cycle of altmetric impact: A longitudinal study of six metrics from PlumX," Journal of Informetrics, Elsevier, vol. 12(3), pages 579-589.
    5. Liwei Zhang & Jue Wang, 2021. "What affects publications’ popularity on Twitter?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(11), pages 9185-9198, November.
    6. Mojisola Erdt & Aarthy Nagarajan & Sei-Ching Joanna Sin & Yin-Leng Theng, 2016. "Altmetrics: an analysis of the state-of-the-art in measuring research impact on social media," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(2), pages 1117-1166, November.
    7. Yu Liu & Dan Lin & Xiujuan Xu & Shimin Shan & Quan Z. Sheng, 2018. "Multi-views on Nature Index of Chinese academic institutions," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 823-837, March.
    8. Liwei Zhang & Jue Wang, 2018. "Why highly cited articles are not highly tweeted? A biology case," Scientometrics, Springer;Akadémiai Kiadó, vol. 117(1), pages 495-509, October.
    9. Lutz Bornmann, 2015. "Alternative metrics in scientometrics: a meta-analysis of research into three altmetrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(3), pages 1123-1144, June.
    10. Isidro F. Aguillo, 2020. "Altmetrics of the Open Access Institutional Repositories: a webometrics approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(3), pages 1181-1192, June.
    11. Bornmann, Lutz & Haunschild, Robin & Adams, Jonathan, 2019. "Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF)," Journal of Informetrics, Elsevier, vol. 13(1), pages 325-340.
    12. Yaxue Ma & Zhichao Ba & Yuxiang Zhao & Jin Mao & Gang Li, 2021. "Understanding and predicting the dissemination of scientific papers on social media: a two-step simultaneous equation modeling–artificial neural network approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(8), pages 7051-7085, August.
    13. Mohammadamin Erfanmanesh & A. Noorhidawati & A. Abrizah, 2019. "What can Bookmetrix tell us about the impact of Springer Nature’s books," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 521-536, October.
    14. Saeed-Ul Hassan & Mubashir Imran & Uzair Gillani & Naif Radi Aljohani & Timothy D. Bowman & Fereshteh Didegah, 2017. "Measuring social media activity of scientific literature: an exhaustive comparison of scopus and novel altmetrics big data," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(2), pages 1037-1057, November.
    15. Martín-Martín, Alberto & Orduna-Malea, Enrique & Delgado López-Cózar, Emilio, 2018. "Author-level metrics in the new academic profile platforms: The online behaviour of the Bibliometrics community," Journal of Informetrics, Elsevier, vol. 12(2), pages 494-509.
    16. Mousumi Karmakar & Sumit Kumar Banshal & Vivek Kumar Singh, 2020. "Does presence of social media plugins in a journal website result in higher social media attention of its research publications?," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(3), pages 2103-2143, September.
    17. Jyoti Paswan & Vivek Kumar Singh & Mousumi Karmakar & Prashasti Singh, 2022. "Does university–industry–government collaboration in research gets higher citation and altmetric impact? A case study from India," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(11), pages 6063-6082, November.
    18. Ying Guo & Xiantao Xiao, 2022. "Author-level altmetrics for the evaluation of Chinese scholars," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(2), pages 973-990, February.
    19. Jianhua Hou & Xiucai Yang & Yang Zhang, 2023. "The effect of social media knowledge cascade: an analysis of scientific papers diffusion," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(9), pages 5169-5195, September.
    20. Mingyang Wang & Zhenyu Wang & Guangsheng Chen, 2019. "Which can better predict the future success of articles? Bibliometric indices or alternative metrics," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(3), pages 1575-1595, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:129:y:2024:i:7:d:10.1007_s11192-024-05061-9. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.