IDEAS home Printed from https://ideas.repec.org/a/spr/qualqt/v49y2015i1p155-165.html
   My bibliography  Save this article

Research quality evaluation: comparing citation counts considering bibliometric database errors

Author

Listed:
  • Fiorenzo Franceschini
  • Domenico Maisano
  • Luca Mastrogiacomo

Abstract

When evaluating the research output of scientists, institutions or journals, different portfolios of publications are usually compared with each other. e.g., a typical problem is to select, between two scientists of interest, the one with the most cited portfolio. The total number of received citations is a very popular indicator, generally obtained by bibliometric databases. However, databases are not free from errors, which may affect the result of evaluations and comparisons; among these errors, one of the most significant is that of omitted citations. This paper presents a methodology for the pair-wise comparison of publication portfolios, which takes into account the database quality regarding omitted citations. In particular, it is defined a test for establishing if a citation count is (or not) significantly higher than one other. A statistical model for estimating the type-I error related to this test is also developed. Copyright Springer Science+Business Media Dordrecht 2015

Suggested Citation

  • Fiorenzo Franceschini & Domenico Maisano & Luca Mastrogiacomo, 2015. "Research quality evaluation: comparing citation counts considering bibliometric database errors," Quality & Quantity: International Journal of Methodology, Springer, vol. 49(1), pages 155-165, January.
    • Handle: RePEc:spr:qualqt:v:49:y:2015:i:1:p:155-165
    DOI: 10.1007/s11135-013-9979-1
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1007/s11135-013-9979-1
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1007/s11135-013-9979-1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Senator Jeong & Sungin Lee & Hong‐Gee Kim, 2009. "Are you an invited speaker? A bibliometric analysis of elite groups for scholarly events in bioinformatics," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(6), pages 1118-1131, June.
    2. David Adam, 2002. "The counting house," Nature, Nature, vol. 415(6873), pages 726-729, February.
    3. Fiorenzo Franceschini & Maurizio Galetto & Domenico Maisano & Luca Mastrogiacomo, 2012. "The success-index: an alternative approach to the h-index for evaluating an individual’s research output," Scientometrics, Springer;Akadémiai Kiadó, vol. 92(3), pages 621-641, September.
    4. Georgina Guilera & Juana Gómez-Benito & M. Hidalgo, 2010. "Citation analysis in research on differential item functioning," Quality & Quantity: International Journal of Methodology, Springer, vol. 44(6), pages 1249-1255, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Shirley Ainsworth & Jane M. Russell, 2018. "Has hosting on science direct improved the visibility of Latin American scholarly journals? A preliminary analysis of data quality," Scientometrics, Springer;Akadémiai Kiadó, vol. 115(3), pages 1463-1484, June.
    2. Matteo Pedrini & Valentina Langella & Mario Alberto Battaglia & Paola Zaratin, 2018. "Assessing the health research’s social impact: a systematic review," Scientometrics, Springer;Akadémiai Kiadó, vol. 114(3), pages 1227-1250, March.
    3. João Ricardo Faria & Peter F. Wanke & João J. Ferreira & Franklin G. Mixon, 2018. "Research and innovation in higher education: empirical evidence from research and patenting in Brazil," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(1), pages 487-504, July.
    4. Arlette Jappe, 2020. "Professional standards in bibliometric research evaluation? A meta-evaluation of European assessment practice 2005–2019," PLOS ONE, Public Library of Science, vol. 15(4), pages 1-23, April.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zhi Li & Qinke Peng & Che Liu, 2016. "Two citation-based indicators to measure latent referential value of papers," Scientometrics, Springer;Akadémiai Kiadó, vol. 108(3), pages 1299-1313, September.
    2. Antonio Fernandez-Cano & Inés M. Fernández-Guerrero, 2017. "A multivariate model for evaluating emergency medicine journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(2), pages 991-1003, February.
    3. J. A. García & Rosa Rodriguez-Sánchez & J. Fdez-Valdivia & J. Martinez-Baena, 2012. "On first quartile journals which are not of highest impact," Scientometrics, Springer;Akadémiai Kiadó, vol. 90(3), pages 925-943, March.
    4. Perc, Matjaž, 2010. "Zipf’s law and log-normal distributions in measures of scientific output across fields and institutions: 40 years of Slovenia’s research as an example," Journal of Informetrics, Elsevier, vol. 4(3), pages 358-364.
    5. Aurora A.C. Teixeira & José Sequeira, 2009. "Determinants of the international influence of a R&D organisation: a bibliometric approach," UITT Working Papers 2009-03-wp3, INESC Porto, Unidade de Inovação e Transferência de Tecnologia(UITT).
    6. Jochen Krauss, 2007. "Journal self-citation rates in ecological sciences," Scientometrics, Springer;Akadémiai Kiadó, vol. 73(1), pages 79-89, October.
    7. Lutz Bornmann & Werner Marx, 2014. "How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 487-509, January.
    8. Li, Feng & Miao, Yajun & Yang, Chenchen, 2015. "How do alumni faculty behave in research collaboration? An analysis of Chang Jiang Scholars in China," Research Policy, Elsevier, vol. 44(2), pages 438-450.
    9. Fiorenzo Franceschini & Domenico Maisano & Luca Mastrogiacomo, 2014. "The citer-success-index: a citer-based indicator to select a subset of elite papers," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(2), pages 963-983, November.
    10. Barbosa, Fabiana G. & Schneck, Fabiana, 2015. "Characteristics of the top-cited papers in species distribution predictive models," Ecological Modelling, Elsevier, vol. 313(C), pages 77-83.
    11. John Rigby & Barbara Jones, 2020. "Bringing the doctoral thesis by published papers to the Social Sciences and the Humanities: A quantitative easing? A small study of doctoral thesis submission rules and practice in two disciplines in ," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(2), pages 1387-1409, August.
    12. Waltman, Ludo & van Eck, Nees Jan, 2008. "Some comments on the journal weighted impact factor proposed by Habibzadeh and Yadollahie," Journal of Informetrics, Elsevier, vol. 2(4), pages 369-372.
    13. Narongrit Sombatsompop & T. Markpin & N. Premkamolnetr, 2004. "A modified method for calculating the Impact Factors of journals in ISI Journal Citation Reports: Polymer Science Category in 1997–2001," Scientometrics, Springer;Akadémiai Kiadó, vol. 60(2), pages 217-235, June.
    14. José María Gómez-Sancho & María Jesús Mancebón-Torrubia, 2009. "The evaluation of scientific production: Towards a neutral impact factor," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(2), pages 435-458, November.
    15. Bar-Ilan, Judit, 2008. "Informetrics at the beginning of the 21st century—A review," Journal of Informetrics, Elsevier, vol. 2(1), pages 1-52.
    16. Wenjia Zhu & Jiancheng Guan, 2013. "A bibliometric study of service innovation research: based on complex network analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(3), pages 1195-1216, March.
    17. Min Song & Su Yeon Kim, 2013. "Detecting the knowledge structure of bioinformatics by mining full-text collections," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(1), pages 183-201, July.
    18. Guan Jiancheng & Wang Junxia, 2004. "Evaluation and interpretation of knowledge production efficiency," Scientometrics, Springer;Akadémiai Kiadó, vol. 59(1), pages 131-155, January.
    19. Santiago Salinas & Stephan B Munch, 2015. "Where Should I Send It? Optimizing the Submission Decision Process," PLOS ONE, Public Library of Science, vol. 10(1), pages 1-11, January.
    20. Matthew Francisco & Staša Milojevic & Selma Šabanovic, 2011. "Conference Models to Bridge Micro and Macro Studies of Science," Journal of Artificial Societies and Social Simulation, Journal of Artificial Societies and Social Simulation, vol. 14(4), pages 1-13.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:qualqt:v:49:y:2015:i:1:p:155-165. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.