IDEAS home Printed from https://ideas.repec.org/a/eee/infome/v16y2022i4s1751157722000748.html
   My bibliography  Save this article

On some properties of medians, percentiles, baselines, and thresholds in empirical bibliometric analysis

Author

Listed:
  • Pislyakov, Vladimir

Abstract

One of the most useful and correct methodological approaches in bibliometrics is ranking. In the context of highly skewed bibliometric distributions and severe distortions caused by outliers, it is often the preferable way of analysis. Ranking methodology strictly implies that “oranges should be compared with oranges, apples with apples”. We should make a “like with like” comparison. Ranks in different fields show how a unit under study is compared to others in its field. But do we always apply an “apples approach” appropriately? Is median really a 50%, quartile a 25%, 10th percentile a 10%? The paper considers theoretical definitions of such terms compared to their real sense in the course of bibliometric research. It is found that in many empirical cases quartiles are not quarters, medians are not halves, world baselines are not unity, and integer thresholds lead to inequality of performance evaluation in different science fields.

Suggested Citation

  • Pislyakov, Vladimir, 2022. "On some properties of medians, percentiles, baselines, and thresholds in empirical bibliometric analysis," Journal of Informetrics, Elsevier, vol. 16(4).
    • Handle: RePEc:eee:infome:v:16:y:2022:i:4:s1751157722000748
    DOI: 10.1016/j.joi.2022.101322
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S1751157722000748
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joi.2022.101322?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Chen, Shiji & Qiu, Junping & Arsenault, Clément & Larivière, Vincent, 2021. "Exploring the interdisciplinarity patterns of highly cited papers," Journal of Informetrics, Elsevier, vol. 15(1).
    2. Michael Schreiber, 2013. "How much do different ways of calculating percentiles influence the derived performance indicators? A case study," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(3), pages 821-829, December.
    3. Loet Leydesdorff & Lutz Bornmann & Rüdiger Mutz & Tobias Opthof, 2011. "Turning the tables on citation analysis one more time: Principles for comparing sets of documents," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 62(7), pages 1370-1381, July.
    4. Syed Hasan & Robert Breunig, 2021. "Article length and citation outcomes," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(9), pages 7583-7608, September.
    5. Christine Bertram & Martin Quaas & Thorsten B. H. Reusch & Athanasios T. Vafeidis & Claudia Wolff & Wilfried Rickels, 2021. "Author Correction: The blue carbon wealth of nations," Nature Climate Change, Nature, vol. 11(10), pages 886-886, October.
    6. Vladimir Pislyakov & Elena Shukshina, 2014. "Measuring excellence in Russia: Highly cited papers, leading institutions, patterns of national and international collaboration," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 65(11), pages 2321-2330, November.
    7. Antonoyiannakis, Manolis, 2018. "Impact Factors and the Central Limit Theorem: Why citation averages are scale dependent," Journal of Informetrics, Elsevier, vol. 12(4), pages 1072-1088.
    8. Martin Grančay & Jolita Vveinhardt & Ērika Šumilo, 2017. "Publish or perish: how Central and Eastern European economists have dealt with the ever-increasing academic publishing requirements 2000–2015," Scientometrics, Springer;Akadémiai Kiadó, vol. 111(3), pages 1813-1837, June.
    9. Juan Xie & Kaile Gong & Ying Cheng & Qing Ke, 2019. "The correlation between paper length and citations: a meta-analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(3), pages 763-786, March.
    10. Waltman, Ludo & van Eck, Nees Jan & van Leeuwen, Thed N. & Visser, Martijn S. & van Raan, Anthony F.J., 2011. "Towards a new crown indicator: Some theoretical considerations," Journal of Informetrics, Elsevier, vol. 5(1), pages 37-47.
    11. Laszlo A. Koczy & Martin Strobel, 2010. "The World Cup of Economics Journals: A Ranking by a Tournament Method," CERS-IE WORKING PAPERS 1018, Institute of Economics, Centre for Economic and Regional Studies.
    12. Opthof, Tobias & Leydesdorff, Loet, 2010. "Caveats for the journal and field normalizations in the CWTS (“Leiden”) evaluations of research performance," Journal of Informetrics, Elsevier, vol. 4(3), pages 423-430.
    13. Thed N. Van Leeuwen & Martijn S. Visser & Henk F. Moed & Ton J. Nederhof & Anthony F. J. Van Raan, 2003. "The Holy Grail of science policy: Exploring and combining bibliometric tools in search of scientific excellence," Scientometrics, Springer;Akadémiai Kiadó, vol. 57(2), pages 257-280, June.
    14. R E de Bruin & A Kint & M Luwel & H F Moed, 1993. "A study of research evaluation and planning: the University of Ghent," Research Evaluation, Oxford University Press, vol. 3(1), pages 25-41, April.
    15. W. M. Shaw, 1985. "Critical thresholds in co‐citation graphs," Journal of the American Society for Information Science, Association for Information Science & Technology, vol. 36(1), pages 38-43, January.
    16. Ludo Waltman & Michael Schreiber, 2013. "On the calculation of percentile-based bibliometric indicators," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(2), pages 372-379, February.
    17. Ruben Miranda & Esther Garcia-Carpintero, 2019. "Comparison of the share of documents and citations from different quartile journals in 25 research areas," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(1), pages 479-501, October.
    18. Alfredo Yegros-Yegros & Ismael Rafols & Pablo D’Este, 2015. "Does Interdisciplinary Research Lead to Higher Citation Impact? The Different Effect of Proximal and Distal Interdisciplinarity," PLOS ONE, Public Library of Science, vol. 10(8), pages 1-21, August.
    19. Waltman, Ludo & van Eck, Nees Jan, 2015. "Field-normalized citation impact indicators and the choice of an appropriate counting method," Journal of Informetrics, Elsevier, vol. 9(4), pages 872-894.
    20. Maia Chankseliani & Andrey Lovakov & Vladimir Pislyakov, 2021. "A big picture: bibliometric study of academic publications from post-Soviet countries," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(10), pages 8701-8730, October.
    21. Kwon, Seokbeom & Liu, Xiaoyu & Porter, Alan L. & Youtie, Jan, 2019. "Research addressing emerging technological ideas has greater scientific impact," Research Policy, Elsevier, vol. 48(9), pages 1-1.
    22. van Raan, Anthony F.J. & van Leeuwen, Thed N. & Visser, Martijn S. & van Eck, Nees Jan & Waltman, Ludo, 2010. "Rivals for the crown: Reply to Opthof and Leydesdorff," Journal of Informetrics, Elsevier, vol. 4(3), pages 431-435.
    23. Ludo Waltman & Nees Jan Eck & Thed N. Leeuwen & Martijn S. Visser & Anthony F. J. Raan, 2011. "Towards a new crown indicator: an empirical analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 87(3), pages 467-481, June.
    24. Ludo Waltman & Michael Schreiber, 2013. "On the calculation of percentile‐based bibliometric indicators," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(2), pages 372-379, February.
    25. Lutz Bornmann & Werner Marx, 2014. "How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 487-509, January.
    26. Tolga Yuret, 2016. "Is it easier to publish in journals that have low impact factors?," Applied Economics Letters, Taylor & Francis Journals, vol. 23(11), pages 801-803, July.
    27. Subochev, Andrey & Aleskerov, Fuad & Pislyakov, Vladimir, 2018. "Ranking journals using social choice theory methods: A novel approach in bibliometrics," Journal of Informetrics, Elsevier, vol. 12(2), pages 416-429.
    28. Colliander, Cristian & Ahlgren, Per, 2011. "The effects and their stability of field normalization baseline on relative performance with respect to citation impact: A case study of 20 natural science departments," Journal of Informetrics, Elsevier, vol. 5(1), pages 101-113.
    29. Ewen Callaway, 2016. "Beat it, impact factor! Publishing elite turns against controversial metric," Nature, Nature, vol. 535(7611), pages 210-211, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Kosyakov, Denis & Pislyakov, Vladimir, 2024. "“I'd like to publish in Q1, but there's no Q1 to be found”: Study of journal quartile distributions across subject categories and topics," Journal of Informetrics, Elsevier, vol. 18(1).
    2. Katchanov, Yurij L. & Markova, Yulia V. & Shmatko, Natalia A., 2023. "Uncited papers in the structure of scientific communication," Journal of Informetrics, Elsevier, vol. 17(2).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    2. Dunaiski, Marcel & Geldenhuys, Jaco & Visser, Willem, 2019. "Globalised vs averaged: Bias and ranking performance on the author level," Journal of Informetrics, Elsevier, vol. 13(1), pages 299-313.
    3. Bouyssou, Denis & Marchant, Thierry, 2016. "Ranking authors using fractional counting of citations: An axiomatic approach," Journal of Informetrics, Elsevier, vol. 10(1), pages 183-199.
    4. Schneider, Jesper W., 2013. "Caveats for using statistical significance tests in research assessments," Journal of Informetrics, Elsevier, vol. 7(1), pages 50-62.
    5. Thelwall, Mike, 2017. "Three practical field normalised alternative indicator formulae for research evaluation," Journal of Informetrics, Elsevier, vol. 11(1), pages 128-151.
    6. Tatiana Lisitskaya & Pavel Taranov & Ekaterina Ugnich & Vladimir Pislyakov, 2024. "Pillar Universities in Russia: Bibliometrics of ‘the second best’," Quality & Quantity: International Journal of Methodology, Springer, vol. 58(1), pages 365-383, February.
    7. Herranz, Neus & Ruiz-Castillo, Javier, 2012. "Sub-field normalization in the multiplicative case: Average-based citation indicators," Journal of Informetrics, Elsevier, vol. 6(4), pages 543-556.
    8. Bornmann, Lutz & Ganser, Christian & Tekles, Alexander, 2022. "Simulation of the h index use at university departments within the bibliometrics-based heuristics framework: Can the indicator be used to compare individual researchers?," Journal of Informetrics, Elsevier, vol. 16(1).
    9. Bornmann, Lutz & Leydesdorff, Loet & Wang, Jian, 2013. "Which percentile-based approach should be preferred for calculating normalized citation impact values? An empirical comparison of five approaches including a newly developed citation-rank approach (P1," Journal of Informetrics, Elsevier, vol. 7(4), pages 933-944.
    10. Dunaiski, Marcel & Geldenhuys, Jaco & Visser, Willem, 2019. "On the interplay between normalisation, bias, and performance of paper impact metrics," Journal of Informetrics, Elsevier, vol. 13(1), pages 270-290.
    11. Albarrán, Pedro & Herrero, Carmen & Ruiz-Castillo, Javier & Villar, Antonio, 2017. "The Herrero-Villar approach to citation impact," Journal of Informetrics, Elsevier, vol. 11(2), pages 625-640.
    12. Loet Leydesdorff & Lutz Bornmann & Jonathan Adams, 2019. "The integrated impact indicator revisited (I3*): a non-parametric alternative to the journal impact factor," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(3), pages 1669-1694, June.
    13. Ludo Waltman & Nees Jan Eck & Thed N. Leeuwen & Martijn S. Visser & Anthony F. J. Raan, 2011. "Towards a new crown indicator: an empirical analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 87(3), pages 467-481, June.
    14. Lutz Bornmann & Alexander Tekles & Loet Leydesdorff, 2019. "How well does I3 perform for impact measurement compared to other bibliometric indicators? The convergent validity of several (field-normalized) indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 119(2), pages 1187-1205, May.
    15. Mingers, John & Leydesdorff, Loet, 2015. "A review of theory and practice in scientometrics," European Journal of Operational Research, Elsevier, vol. 246(1), pages 1-19.
    16. Tol, Richard S.J., 2013. "Identifying excellent researchers: A new approach," Journal of Informetrics, Elsevier, vol. 7(4), pages 803-810.
    17. Rons, Nadine, 2012. "Partition-based Field Normalization: An approach to highly specialized publication records," Journal of Informetrics, Elsevier, vol. 6(1), pages 1-10.
    18. Marcello D’Agostino & Valentino Dardanoni & Roberto Ghiselli Ricci, 2017. "How to standardize (if you must)," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(2), pages 825-843, November.
    19. Wu, Jiang, 2013. "Investigating the universal distributions of normalized indicators and developing field-independent index," Journal of Informetrics, Elsevier, vol. 7(1), pages 63-71.
    20. Thelwall, Mike, 2016. "The precision of the arithmetic mean, geometric mean and percentiles for citation data: An experimental simulation modelling approach," Journal of Informetrics, Elsevier, vol. 10(1), pages 110-123.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:infome:v:16:y:2022:i:4:s1751157722000748. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joi .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.