IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v125y2020i3d10.1007_s11192-020-03722-z.html
   My bibliography  Save this article

Analyzing the impact of reputational bias on global university rankings based on objective research performance data: the case of the Shanghai Ranking (ARWU)

Author

Listed:
  • Vicente Safón

    (University of Valencia)

  • Domingo Docampo

    (University of Vigo)

Abstract

This paper analyzes the potential impact of reputational bias on academic classifications based upon current research performance measures. Empirical evidence supports the existence of a noticeable effect of reputational bias in the Academic Ranking of World Universities (ARWU), informally known as the Shanghai Ranking. Despite using reliable, objective data, the results of the ranking are partially contaminated by the halo effect that pervades peer review processes and citation practices behind two of ARWU’s main indicators: highly cited researchers (HiCi), and papers published in Nature and Science (N&S). ARWU results may contribute to reinforcing reputational bias through a vicious feedback loop. The study describes a method for quantifying the bias present in the aforementioned indicators. Our findings reveal that bias exists in N&S but not in HiCi, and that while it benefits top universities and Japanese universities, it stifles Australian counterparts. The paper paves the way for a new, interesting debate that will require future research to identify and cope with shortcomings derived from the impact of reputational bias in journal procedures both when evaluating and accepting manuscripts, as well as in the methodology of global university rankings.

Suggested Citation

  • Vicente Safón & Domingo Docampo, 2020. "Analyzing the impact of reputational bias on global university rankings based on objective research performance data: the case of the Shanghai Ranking (ARWU)," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(3), pages 2199-2227, December.
    • Handle: RePEc:spr:scient:v:125:y:2020:i:3:d:10.1007_s11192-020-03722-z
    DOI: 10.1007/s11192-020-03722-z
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-020-03722-z
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-020-03722-z?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Catherine Dehon & Alice McCathie & Vincenzo Verardi, 2010. "Uncovering excellence in academic rankings: a closer look at the Shanghai ranking," Scientometrics, Springer;Akadémiai Kiadó, vol. 83(2), pages 515-524, May.
    2. Petersen, Alexander M., 2019. "Megajournal mismanagement: Manuscript decision bias and anomalous editor activity at PLOS ONE," Journal of Informetrics, Elsevier, vol. 13(4).
    3. Nicky Phillips, 2017. "Japanese research leaders warn about national science decline," Nature, Nature, vol. 550(7676), pages 310-311, October.
    4. Jacek Bogocz & Andrzej Bak & Jaroslaw Polanski, 2014. "No free lunches in nature? An analysis of the regional distribution of the affiliations of Nature publications," Scientometrics, Springer;Akadémiai Kiadó, vol. 101(1), pages 547-568, October.
    5. Mingers, John & Xu, Fang, 2010. "The drivers of citations in management science journals," European Journal of Operational Research, Elsevier, vol. 205(2), pages 422-430, September.
    6. Ellis, Larry V. & Durden, Garey C., 1991. "Why economists rank their journals the way they do," Journal of Economics and Business, Elsevier, vol. 43(3), pages 265-270, August.
    7. Domingo Docampo, 2013. "Reproducibility of the Shanghai academic ranking of world universities results," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(2), pages 567-587, February.
    8. Daraio, Cinzia & Bonaccorsi, Andrea & Simar, Léopold, 2015. "Rankings and university performance: A conditional multidimensional approach," European Journal of Operational Research, Elsevier, vol. 244(3), pages 918-930.
    9. Mei Li & Sriram Shankar & Kam Ki Tang, 2009. "Why does the US dominate university league tables?," Discussion Papers Series 391, School of Economics, University of Queensland, Australia.
    10. repec:fth:middec:63 is not listed on IDEAS
    11. Carole J. Lee & Cassidy R. Sugimoto & Guo Zhang & Blaise Cronin, 2013. "Bias in peer review," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 64(1), pages 2-17, January.
    12. Domingo Docampo & Lawrence Cram, 2015. "On the effects of institutional size in university classifications: the case of the Shanghai ranking," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(2), pages 1325-1346, February.
    13. Marshall Medoff, 2006. "Evidence of a Harvard and Chicago Matthew Effect," Journal of Economic Methodology, Taylor & Francis Journals, vol. 13(4), pages 485-506.
    14. van den Besselaar, Peter & Heyman, Ulf & Sandström, Ulf, 2017. "Perverse effects of output-based research funding? Butler’s Australian case revisited," Journal of Informetrics, Elsevier, vol. 11(3), pages 905-918.
    15. Vogel, Rick & Hattke, Fabian & Petersen, Jessica, 2017. "Journal rankings in management and business studies: What rules do we play by?," Research Policy, Elsevier, vol. 46(10), pages 1707-1722.
    16. Carl Shapiro, 1983. "Premiums for High Quality Products as Returns to Reputations," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 98(4), pages 659-679.
    17. Fredrik Niclas Piro & Gunnar Sivertsen, 2016. "How can differences in international university rankings be explained?," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 2263-2278, December.
    18. Carole J. Lee & Cassidy R. Sugimoto & Guo Zhang & Blaise Cronin, 2013. "Bias in peer review," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 64(1), pages 2-17, January.
    19. Anthony F. J. van Raan, 2005. "Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods," Scientometrics, Springer;Akadémiai Kiadó, vol. 62(1), pages 133-143, January.
    20. J. Scott Armstrong & Tad Sperry, 1994. "The Ombudsman: Business School Prestige—Research versus Teaching," Interfaces, INFORMS, vol. 24(2), pages 13-43, April.
    21. Michael Spence, 1973. "Job Market Signaling," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 87(3), pages 355-374.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Christoph Burmann & Fernando García & Francisco Guijarro & Javier Oliver, 2021. "Ranking the Performance of Universities: The Role of Sustainability," Sustainability, MDPI, vol. 13(23), pages 1-16, November.
    2. Elena De la Poza & Paloma Merello & Antonio Barberá & Alberto Celani, 2021. "Universities’ Reporting on SDGs: Using THE Impact Rankings to Model and Measure Their Contribution to Sustainability," Sustainability, MDPI, vol. 13(4), pages 1-28, February.
    3. Sergey N. Kukushkin & Aleksei V. Bogoviz & Yuliya V. Chutcheva & Larisa S. Korobeinikova & Svetlana V. Lobova, 2024. "Target SAP-LAP Model for Sustainable Development of Russian Universities in the Decade of Science and Technology," Global Journal of Flexible Systems Management, Springer;Global Institute of Flexible Systems Management, vol. 25(1), pages 123-138, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Vicente Safón, 2019. "Inter-ranking reputational effects: an analysis of the Academic Ranking of World Universities (ARWU) and the Times Higher Education World University Rankings (THE) reputational relationship," Scientometrics, Springer;Akadémiai Kiadó, vol. 121(2), pages 897-915, November.
    2. Mehdi Rhaiem & Nabil Amara, 2020. "Determinants of research efficiency in Canadian business schools: evidence from scholar-level data," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(1), pages 53-99, October.
    3. Antonio Fernández-Cano & Elvira Curiel-Marin & Manuel Torralbo-Rodríguez & Mónica Vallejo-Ruiz, 2018. "Questioning the Shanghai Ranking methodology as a tool for the evaluation of universities: an integrative review," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 2069-2083, September.
    4. Meyer, Matthias & Waldkirch, Rüdiger W. & Duscher, Irina & Just, Alexander, 2018. "Drivers of citations: An analysis of publications in “top” accounting journals," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 51(C), pages 24-46.
    5. Elena Veretennik & Maria Yudkevich, 2023. "Inconsistent quality signals: evidence from the regional journals," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(6), pages 3675-3701, June.
    6. D. Docampo & D. Egret & L. Cram, 2015. "The effect of university mergers on the Shanghai ranking," Scientometrics, Springer;Akadémiai Kiadó, vol. 104(1), pages 175-191, July.
    7. David Card & Stefano DellaVigna, 2017. "What do Editors Maximize? Evidence from Four Leading Economics Journals," NBER Working Papers 23282, National Bureau of Economic Research, Inc.
    8. David Card & Stefano DellaVigna, 2020. "What Do Editors Maximize? Evidence from Four Economics Journals," The Review of Economics and Statistics, MIT Press, vol. 102(1), pages 195-217, March.
    9. Dag W. Aksnes & Liv Langfeldt & Paul Wouters, 2019. "Citations, Citation Indicators, and Research Quality: An Overview of Basic Concepts and Theories," SAGE Open, , vol. 9(1), pages 21582440198, February.
    10. Frenken, Koen & Heimeriks, Gaston J. & Hoekman, Jarno, 2017. "What drives university research performance? An analysis using the CWTS Leiden Ranking data," Journal of Informetrics, Elsevier, vol. 11(3), pages 859-872.
    11. Domingo Docampo & Lawrence Cram, 2015. "On the effects of institutional size in university classifications: the case of the Shanghai ranking," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(2), pages 1325-1346, February.
    12. Nabil Amara & Réjean Landry & Norrin Halilem, 2015. "What can university administrators do to increase the publication and citation scores of their faculty members?," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(2), pages 489-530, May.
    13. Domingo Docampo & Lawrence Cram, 2019. "Highly cited researchers: a moving target," Scientometrics, Springer;Akadémiai Kiadó, vol. 118(3), pages 1011-1025, March.
    14. Anna Nagurney & Dong Li, 2014. "Equilibria and dynamics of supply chain network competition with information asymmetry in quality and minimum quality standards," Computational Management Science, Springer, vol. 11(3), pages 285-315, July.
    15. Mahenc, Philippe & Meunier, Valérie, 2006. "Early Sales of Bordeaux grands crus," Journal of Wine Economics, Cambridge University Press, vol. 1(1), pages 57-74, April.
    16. Thams, Yannick & Alvarado-Vargas, Marcelo J. & Newburry, William, 2016. "Geographical diversification as a predictor of MNC reputations in their home nations," Journal of Business Research, Elsevier, vol. 69(8), pages 2882-2889.
    17. Jürgen Janger & Nicole Schmidt-Padickakudy & Anna Strauss-Kollin, 2019. "International Differences in Basic Research Grant Funding. A Systematic Comparison," WIFO Studies, WIFO, number 61664.
    18. Colombelli, Alessandra & Grilli, Luca & Minola, Tommaso & Mrkajic, Boris, 2020. "To what extent do young innovative companies take advantage of policy support to enact innovation appropriation mechanisms?," Research Policy, Elsevier, vol. 49(10).
    19. Rodríguez Sánchez, Isabel & Makkonen, Teemu & Williams, Allan M., 2019. "Peer review assessment of originality in tourism journals: critical perspective of key gatekeepers," Annals of Tourism Research, Elsevier, vol. 77(C), pages 1-11.
    20. Zhentao Liang & Jin Mao & Gang Li, 2023. "Bias against scientific novelty: A prepublication perspective," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 74(1), pages 99-114, January.

    More about this item

    Keywords

    ARWU; Ranking; University; Reputation; Regression;
    All these keywords.

    JEL classification:

    • I23 - Health, Education, and Welfare - - Education - - - Higher Education; Research Institutions
    • L14 - Industrial Organization - - Market Structure, Firm Strategy, and Market Performance - - - Transactional Relationships; Contracts and Reputation

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:125:y:2020:i:3:d:10.1007_s11192-020-03722-z. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.