IDEAS home Printed from https://ideas.repec.org/a/spr/qualqt/v49y2015i1p155-165.html
   My bibliography  Save this article

Research quality evaluation: comparing citation counts considering bibliometric database errors

Author

Listed:
  • Fiorenzo Franceschini
  • Domenico Maisano
  • Luca Mastrogiacomo

Abstract

When evaluating the research output of scientists, institutions or journals, different portfolios of publications are usually compared with each other. e.g., a typical problem is to select, between two scientists of interest, the one with the most cited portfolio. The total number of received citations is a very popular indicator, generally obtained by bibliometric databases. However, databases are not free from errors, which may affect the result of evaluations and comparisons; among these errors, one of the most significant is that of omitted citations. This paper presents a methodology for the pair-wise comparison of publication portfolios, which takes into account the database quality regarding omitted citations. In particular, it is defined a test for establishing if a citation count is (or not) significantly higher than one other. A statistical model for estimating the type-I error related to this test is also developed. Copyright Springer Science+Business Media Dordrecht 2015

Suggested Citation

  • Fiorenzo Franceschini & Domenico Maisano & Luca Mastrogiacomo, 2015. "Research quality evaluation: comparing citation counts considering bibliometric database errors," Quality & Quantity: International Journal of Methodology, Springer, vol. 49(1), pages 155-165, January.
    • Handle: RePEc:spr:qualqt:v:49:y:2015:i:1:p:155-165
    DOI: 10.1007/s11135-013-9979-1
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1007/s11135-013-9979-1
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1007/s11135-013-9979-1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Senator Jeong & Sungin Lee & Hong‐Gee Kim, 2009. "Are you an invited speaker? A bibliometric analysis of elite groups for scholarly events in bioinformatics," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(6), pages 1118-1131, June.
    2. Fiorenzo Franceschini & Maurizio Galetto & Domenico Maisano & Luca Mastrogiacomo, 2012. "The success-index: an alternative approach to the h-index for evaluating an individual’s research output," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(3), pages 621-641, September.
    3. Georgina Guilera & Juana Gómez-Benito & M. Hidalgo, 2010. "Citation analysis in research on differential item functioning," Quality & Quantity: International Journal of Methodology, Springer, vol. 44(6), pages 1249-1255, October.
    4. David Adam, 2002. "The counting house," Nature, Nature, vol. 415(6873), pages 726-729, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Matteo Pedrini & Valentina Langella & Mario Alberto Battaglia & Paola Zaratin, 2018. "Assessing the health research’s social impact: a systematic review," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 1227-1250, March.
    2. Shirley Ainsworth & Jane M. Russell, 2018. "Has hosting on science direct improved the visibility of Latin American scholarly journals? A preliminary analysis of data quality," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(3), pages 1463-1484, June.
    3. João Ricardo Faria & Peter F. Wanke & João J. Ferreira & Franklin G. Mixon, 2018. "Research and innovation in higher education: empirical evidence from research and patenting in Brazil," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(1), pages 487-504, July.
    4. Arlette Jappe, 2020. "Professional standards in bibliometric research evaluation? A meta-evaluation of European assessment practice 2005–2019," PLOS ONE, Public Library of Science, vol. 15(4), pages 1-23, April.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zhi Li & Qinke Peng & Che Liu, 2016. "Two citation-based indicators to measure latent referential value of papers," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(3), pages 1299-1313, September.
    2. Antonio Fernandez-Cano & Inés M. Fernández-Guerrero, 2017. "A multivariate model for evaluating emergency medicine journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(2), pages 991-1003, February.
    3. Lakshmi Balachandran Nair & Michael Gibbert, 2016. "What makes a ‘good’ title and (how) does it matter for citations? A review and general model of article title attributes in management science," Scientometrics, Springer;Akadémiai Kiadó, vol. 107(3), pages 1331-1359, June.
    4. Senator Jeong & Hong-Gee Kim, 2010. "Intellectual structure of biomedical informatics reflected in scholarly events," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(2), pages 541-551, November.
    5. J. A. García & Rosa Rodriguez-Sánchez & J. Fdez-Valdivia & J. Martinez-Baena, 2012. "On first quartile journals which are not of highest impact," Scientometrics, Springer;Akadémiai Kiadó, vol. 90(3), pages 925-943, March.
    6. Perc, Matjaž, 2010. "Zipf’s law and log-normal distributions in measures of scientific output across fields and institutions: 40 years of Slovenia’s research as an example," Journal of Informetrics, Elsevier, vol. 4(3), pages 358-364.
    7. Aurora A.C. Teixeira & José Sequeira, 2009. "Determinants of the international influence of a R&D organisation: a bibliometric approach," UITT Working Papers 2009-03-wp3, INESC Porto, Unidade de Inovação e Transferência de Tecnologia(UITT).
    8. Jochen Krauss, 2007. "Journal self-citation rates in ecological sciences," Scientometrics, Springer;Akadémiai Kiadó, vol. 73(1), pages 79-89, October.
    9. Lutz Bornmann & Werner Marx, 2014. "How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 487-509, January.
    10. Fiorenzo Franceschini & Domenico Maisano & Luca Mastrogiacomo, 2015. "Errors in DOI indexing by bibliometric databases," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(3), pages 2181-2186, March.
    11. Chieh Liu & Mu-Hsuan Huang, 2022. "Exploring the relationships between altmetric counts and citations of papers in different academic fields based on co-occurrence analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4939-4958, August.
    12. David Rodriguez, 2005. "Publishing Performance Of Spanish Academics: 1970-2004," UFAE and IAE Working Papers 642.05, Unitat de Fonaments de l'Anàlisi Econòmica (UAB) and Institut d'Anàlisi Econòmica (CSIC).
    13. Li, Feng & Miao, Yajun & Yang, Chenchen, 2015. "How do alumni faculty behave in research collaboration? An analysis of Chang Jiang Scholars in China," Research Policy, Elsevier, vol. 44(2), pages 438-450.
    14. Jean-Charles Billaut & Denis Bouyssou & Philippe Vincke, 2009. "Should you believe in the Shanghai ranking? An MCDM view," Working Papers hal-00877050, HAL.
    15. Fiorenzo Franceschini & Domenico Maisano & Luca Mastrogiacomo, 2014. "The citer-success-index: a citer-based indicator to select a subset of elite papers," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(2), pages 963-983, November.
    16. Barbosa, Fabiana G. & Schneck, Fabiana, 2015. "Characteristics of the top-cited papers in species distribution predictive models," Ecological Modelling, Elsevier, vol. 313(C), pages 77-83.
    17. John Rigby & Barbara Jones, 2020. "Bringing the doctoral thesis by published papers to the Social Sciences and the Humanities: A quantitative easing? A small study of doctoral thesis submission rules and practice in two disciplines in ," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(2), pages 1387-1409, August.
    18. Ana-Beatriz Hernández-Lara & Maria-Victoria Sánchez-Rebull & Angels Niñerola, 2021. "Six Sigma in Health Literature, What Matters?," IJERPH, MDPI, vol. 18(16), pages 1-13, August.
    19. Waltman, Ludo & van Eck, Nees Jan, 2008. "Some comments on the journal weighted impact factor proposed by Habibzadeh and Yadollahie," Journal of Informetrics, Elsevier, vol. 2(4), pages 369-372.
    20. Narongrit Sombatsompop & T. Markpin & N. Premkamolnetr, 2004. "A modified method for calculating the Impact Factors of journals in ISI Journal Citation Reports: Polymer Science Category in 1997–2001," Scientometrics, Springer;Akadémiai Kiadó, vol. 60(2), pages 217-235, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:qualqt:v:49:y:2015:i:1:p:155-165. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.