IDEAS home Printed from https://ideas.repec.org/a/spr/busres/v10y2017i2d10.1007_s40685-017-0044-0.html
   My bibliography  Save this article

Citation metrics as an additional indicator for evaluating research performance? An analysis of their correlations and validity

Author

Listed:
  • Marcel Clermont

    (Technische Universität Braunschweig)

  • Alexander Dirksen

    (Sparkassenverband, Westfalen-Lippe)

  • Barbara Scheidt

    (Forschungszentrum Jülich GmbH, Central Library, Bibliometrics Team)

  • Dirk Tunger

    (Forschungszentrum Jülich GmbH, Central Library, Bibliometrics Team)

Abstract

This paper investigates the potential expansion of an indicator set for research performance evaluation to include citations for the mapping of research impact. To this end, we use research performance data of German business schools and consider the linear correlations and the rank correlations between publication-based, supportive, and citation-based indicators. Furthermore, we compare the business schools in partial ratings of the relative indicators amongst themselves and with those business schools that are classified in other studies as being strong in research and/or reputable. Only low correlations are found between the citation metrics and the other indicator types. Since citations map research outcome, this is an expected result in terms of divergent validity. Amongst themselves, the citation metrics display high correlations, which, in accordance with the convergent validity, shows that they can represent research outcome. However, this does not apply to the J-factor, which is a journal-based normalizing citation metric.

Suggested Citation

  • Marcel Clermont & Alexander Dirksen & Barbara Scheidt & Dirk Tunger, 2017. "Citation metrics as an additional indicator for evaluating research performance? An analysis of their correlations and validity," Business Research, Springer;German Academic Association for Business Research, vol. 10(2), pages 249-279, October.
    • Handle: RePEc:spr:busres:v:10:y:2017:i:2:d:10.1007_s40685-017-0044-0
    DOI: 10.1007/s40685-017-0044-0
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s40685-017-0044-0
    File Function: Abstract
    Download Restriction: no
    File URL: https://libkey.io/10.1007/s40685-017-0044-0?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Bornmann, Lutz & Mutz, Rüdiger & Hug, Sven E. & Daniel, Hans-Dieter, 2011. "A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants," Journal of Informetrics, Elsevier, vol. 5(3), pages 346-359.
    2. Rafael Ball & Bernhard Mittermaier & Dirk Tunger, 2009. "Creation of journal-based publication profiles of scientific institutions — A methodology for the interdisciplinary comparison of scientific research based on the J-factor," Scientometrics, Springer;Akadémiai Kiadó, vol. 81(2), pages 381-392, November.
    3. Marcel Clermont & Alexander Dirksen & Harald Dyckhoff, 2015. "Returns to scale of Business Administration research in Germany," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(2), pages 583-614, May.
    4. Leo Egghe, 2006. "Theory and practise of the g-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 69(1), pages 131-152, October.
    5. Heinrich W. Ursprung, 2003. "Schneewittchen im Land der Klapperschlangen: Evaluation eines Evaluators," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 4(2), pages 177-190, May.
    6. Stefan Hornbostel, 2001. "Third party funding of German universities. An indicator of research activity?," Scientometrics, Springer;Akadémiai Kiadó, vol. 50(3), pages 523-537, March.
    7. Philip Hans Franses, 2014. "Trends in three decades of rankings of Dutch economists," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(2), pages 1257-1268, February.
    8. Hauke Jan & Kossowski Tomasz, 2011. "Comparison of Values of Pearson's and Spearman's Correlation Coefficients on the Same Sets of Data," Quaestiones Geographicae, Sciendo, vol. 30(2), pages 87-93, June.
    9. Dilger, Alexander & Müller, Harry, 2011. "Ein Forschungsleistungsranking auf der Grundlage von Google Scholar," Discussion Papers of the Institute for Organisational Economics 12/2011, University of Münster, Institute for Organisational Economics.
    10. Cyrenne, Philippe & Grant, Hugh, 2009. "University decision making and prestige: An empirical study," Economics of Education Review, Elsevier, vol. 28(2), pages 237-248, April.
    11. Dilger, Alexander, 2009. "Rankings von Zeitschriften und Personen in der BWL," IÖB-Diskussionspapiere 5/09, University of Münster, Institute for Economic Education.
    12. Rolf Ketzler & Klaus F. Zimmermann, 2013. "A citation-analysis of economic research institutes," Scientometrics, Springer;Akadémiai Kiadó, vol. 95(3), pages 1095-1112, June.
    13. Michel Zitt & Suzy Ramanana-Rahary & Elise Bassecoulard, 2005. "Relativity of citation performance and excellence measures: From cross-field to cross-scale effects of field-normalisation," Scientometrics, Springer;Akadémiai Kiadó, vol. 63(2), pages 373-401, April.
    14. Giovanni Abramo & Ciriaco Andrea D’Angelo & Fulvio Viel, 2013. "The suitability of h and g indexes for measuring the research performance of institutions," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(3), pages 555-570, December.
    15. Karol Paludkiewicz & Klaus Wohlrabe, 2010. "Qualitätsanalyse von Zeitschriften in den Wirtschaftswissenschaften – über Zitationsdatenbanken und Impaktfaktoren im Online-Zeitalter," ifo Schnelldienst, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, vol. 63(21), pages 18-28, November.
    16. Anne‐Wil Harzing & Ron van der Wal, 2009. "A Google Scholar h‐index for journals: An alternative metric to measure journal impact in economics and business," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 60(1), pages 41-46, January.
    17. Pablo Jensen & Jean-Baptiste Rouquier & Yves Croissant, 2009. "Testing bibliometric indicators by their prediction of scientists promotions," Scientometrics, Springer;Akadémiai Kiadó, vol. 78(3), pages 467-479, March.
    18. Rolf Sternberg & Timo Litzenberger, 2005. "The publication and citation output of German Faculties of Economics and Social Sciences - a comparison of faculties and disciplines based upon SSCI data," Scientometrics, Springer;Akadémiai Kiadó, vol. 65(1), pages 29-53, October.
    19. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2012. "A sensitivity analysis of researchers’ productivity rankings to the time of citation observation," Journal of Informetrics, Elsevier, vol. 6(2), pages 192-201.
    20. Bielecki, Andre & Albers, Sönke, 2012. "Eine Analyse der Forschungseffizienz deutscher betriebswirtschaftlicher Fachbereiche basierend auf den Daten des Centrums für Hochschulentwicklung (CHE)," EconStor Preprints 57429, ZBW - Leibniz Information Centre for Economics.
    21. Felix Schläpfer & Friedrich Schneider, 2010. "Messung der akademischen Forschungsleistung in den Wirtschaftswissenschaften: Reputation vs. Zitierhäufigkeiten," Perspektiven der Wirtschaftspolitik, Verein für Socialpolitik, vol. 11(4), pages 325-339, November.
    22. Keeney,Ralph L. & Raiffa,Howard, 1993. "Decisions with Multiple Objectives," Cambridge Books, Cambridge University Press, number 9780521438834, September.
    23. Lutz Bornmann & Rüdiger Mutz & Hans-Dieter Daniel, 2013. "Multilevel-statistical reformulation of citation-based university rankings: The Leiden ranking 2011/2012," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(8), pages 1649-1658, August.
    24. Rafael Ball & Dirk Tunger, 2006. "Bibliometric analysis - A new business area for information professionals in libraries?," Scientometrics, Springer;Akadémiai Kiadó, vol. 66(3), pages 561-577, March.
    25. Porter, Stephen R. & Toutkoushian, Robert K., 2006. "Institutional research productivity and the connection to average student quality and overall reputation," Economics of Education Review, Elsevier, vol. 25(6), pages 605-617, December.
    26. Chun-Ting Zhang, 2009. "The e-Index, Complementing the h-Index for Excess Citations," PLOS ONE, Public Library of Science, vol. 4(5), pages 1-4, May.
    27. Abramo, Giovanni & Cicero, Tindaro & D’Angelo, Ciriaco Andrea, 2012. "A sensitivity analysis of research institutions’ productivity rankings to the time of citation observation," Journal of Informetrics, Elsevier, vol. 6(2), pages 298-306.
    28. Albers, Sönke & Bielecki, Andre, 2012. "Wovon hängt die Leistung in Forschung und Lehre ab? Eine Analyse deutscher betriebswirtschaftlicher Fachbereiche basierend auf den Daten des Centrums für Hochschulentwicklung," EconStor Preprints 57428, ZBW - Leibniz Information Centre for Economics.
    29. Johannes Hönekopp & Julie Khan, 2012. "Future publication success in science is better predicted by traditional measures than by the h index," Scientometrics, Springer;Akadémiai Kiadó, vol. 90(3), pages 843-853, March.
    30. Costas, Rodrigo & Bordons, María, 2007. "The h-index: Advantages, limitations and its relation with other bibliometric indicators at the micro level," Journal of Informetrics, Elsevier, vol. 1(3), pages 193-203.
    31. Anthony F. J. Raan, 2006. "Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups," Scientometrics, Springer;Akadémiai Kiadó, vol. 67(3), pages 491-502, June.
    32. Davinia Palomares-Montero & Adela García-Aracil, 2011. "What are the key indicators for evaluating the activities of universities?," Research Evaluation, Oxford University Press, vol. 20(5), pages 353-363, December.
    33. Ursprung Heinrich W. & Zimmer Markus, 2007. "Who is the ”Platz-Hirsch“ of the German Economics Profession?: A Citation Analysis," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 227(2), pages 187-208, April.
    34. Daniela Filippo & Fernando Casani & Carlos García-Zorita & Preiddy Efraín-García & Elías Sanz-Casado, 2012. "Visibility in international rankings. Strategies for enhancing the competitiveness of Spanish universities," Scientometrics, Springer;Akadémiai Kiadó, vol. 93(3), pages 949-966, December.
    35. Jia Zhu & Saeed-Ul Hassan & Hamid Turab Mirza & Qing Xie, 2014. "Measuring recent research performance for Chinese universities using bibliometric methods," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 429-443, October.
    36. Colliander, Cristian & Ahlgren, Per, 2011. "The effects and their stability of field normalization baseline on relative performance with respect to citation impact: A case study of 20 natural science departments," Journal of Informetrics, Elsevier, vol. 5(1), pages 101-113.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Marcel Clermont & Johanna Krolak & Dirk Tunger, 2021. "Does the citation period have any effect on the informative value of selected citation indicators in research evaluations?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1019-1047, February.
    2. Heinz Ahn & Marcel Clermont & Julia Langner, 2022. "The impact of selected input and output factors on measuring research efficiency of university research fields: insights from a purpose-, field-, and method-specific perspective," Journal of Business Economics, Springer, vol. 92(8), pages 1303-1335, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Giovanni Abramo & Ciriaco Andrea D’Angelo & Fulvio Viel, 2013. "The suitability of h and g indexes for measuring the research performance of institutions," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(3), pages 555-570, December.
    2. Waltman, Ludo, 2016. "A review of the literature on citation impact indicators," Journal of Informetrics, Elsevier, vol. 10(2), pages 365-391.
    3. Marcel Clermont & Johanna Krolak & Dirk Tunger, 2021. "Does the citation period have any effect on the informative value of selected citation indicators in research evaluations?," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(2), pages 1019-1047, February.
    4. Zhang, Lin & Thijs, Bart & Glänzel, Wolfgang, 2011. "The diffusion of H-related literature," Journal of Informetrics, Elsevier, vol. 5(4), pages 583-593.
    5. Vîiu, Gabriel-Alexandru, 2016. "A theoretical evaluation of Hirsch-type bibliometric indicators confronted with extreme self-citation," Journal of Informetrics, Elsevier, vol. 10(2), pages 552-566.
    6. Judit Bar-Ilan & Mark Levene, 2015. "The hw-rank: an h-index variant for ranking web pages," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(3), pages 2247-2253, March.
    7. Bornmann, Lutz & Marx, Werner, 2012. "HistCite analysis of papers constituting the h index research front," Journal of Informetrics, Elsevier, vol. 6(2), pages 285-288.
    8. Lorna Wildgaard & Jesper W. Schneider & Birger Larsen, 2014. "A review of the characteristics of 108 author-level bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 125-158, October.
    9. Franceschini, Fiorenzo & Maisano, Domenico, 2010. "The citation triad: An overview of a scientist's publication output based on Ferrers diagrams," Journal of Informetrics, Elsevier, vol. 4(4), pages 503-511.
    10. Ana Paula dos Santos Rubem & Ariane Lima Moura & João Carlos Correia Baptista Soares de Mello, 2015. "Comparative analysis of some individual bibliometric indices when applied to groups of researchers," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(1), pages 1019-1035, January.
    11. Jingda Ding & Chao Liu & Goodluck Asobenie Kandonga, 2020. "Exploring the limitations of the h-index and h-type indexes in measuring the research performance of authors," Scientometrics, Springer;Akadémiai Kiadó, vol. 122(3), pages 1303-1322, March.
    12. Heinz Ahn & Marcel Clermont & Julia Langner, 2022. "The impact of selected input and output factors on measuring research efficiency of university research fields: insights from a purpose-, field-, and method-specific perspective," Journal of Business Economics, Springer, vol. 92(8), pages 1303-1335, October.
    13. Yves Fassin, 2020. "The HF-rating as a universal complement to the h-index," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 965-990, November.
    14. Lorna Wildgaard, 2015. "A comparison of 17 author-level bibliometric indicators for researchers in Astronomy, Environmental Science, Philosophy and Public Health in Web of Science and Google Scholar," Scientometrics, Springer;Akadémiai Kiadó, vol. 104(3), pages 873-906, September.
    15. Hui-Zhen Fu & Yuh-Shan Ho, 2013. "Comparison of independent research of China’s top universities using bibliometric indicators," Scientometrics, Springer;Akadémiai Kiadó, vol. 96(1), pages 259-276, July.
    16. Marcin Kozak & Lutz Bornmann, 2012. "A New Family of Cumulative Indexes for Measuring Scientific Performance," PLOS ONE, Public Library of Science, vol. 7(10), pages 1-4, October.
    17. Shahryar Rahnamayan & Sedigheh Mahdavi & Kalyanmoy Deb & Azam Asilian Bidgoli, 2020. "Ranking Multi-Metric Scientific Achievements Using a Concept of Pareto Optimality," Mathematics, MDPI, vol. 8(6), pages 1-46, June.
    18. Corey J A Bradshaw & Justin M Chalker & Stefani A Crabtree & Bart A Eijkelkamp & John A Long & Justine R Smith & Kate Trinajstic & Vera Weisbecker, 2021. "A fairer way to compare researchers at any career stage and in any discipline using open-access citation data," PLOS ONE, Public Library of Science, vol. 16(9), pages 1-15, September.
    19. Abramo, Giovanni & D’Angelo, Ciriaco Andrea, 2015. "Ranking research institutions by the number of highly-cited articles per scientist," Journal of Informetrics, Elsevier, vol. 9(4), pages 915-923.
    20. Parul Khurana & Kiran Sharma, 2022. "Impact of h-index on author’s rankings: an improvement to the h-index for lower-ranked authors," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(8), pages 4483-4498, August.

    More about this item

    Keywords

    Citation metrics; German business schools; Research performance evaluation; Research ranking; Validity;
    All these keywords.

    JEL classification:

    • H83 - Public Economics - - Miscellaneous Issues - - - Public Administration
    • I23 - Health, Education, and Welfare - - Education - - - Higher Education; Research Institutions

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:busres:v:10:y:2017:i:2:d:10.1007_s40685-017-0044-0. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.