IDEAS home Printed from https://ideas.repec.org/a/eee/respol/v40y2011i1p165-177.html
   My bibliography  Save this article

Rickety numbers: Volatility of university rankings and policy implications

Author

Listed:
  • Saisana, Michaela
  • d'Hombres, Béatrice
  • Saltelli, Andrea

Abstract

The most popular world university rankings are routinely taken at face value by media and social actors. While these rankings are politically influential, they are sensitive to both the conceptual framework (the set of indicators) and the modelling choices made in their construction (e.g., weighting or type of aggregation). A robustness analysis, based on a multi-modelling approach, aims to test the validity of the inference about the rankings produced in the Academic Ranking of World Universities (ARWU) of Shanghai Jiao Tong University and those produced by the UK's Times Higher Education Supplement (THES). Conclusions are drawn on the reliability of individual university ranks and on relative country or macro regional performance (e.g., Europe versus USA versus China) in terms of the number of top performing institutions. We find that while university and country level statistical inferences are unsound, the inference on macro regions is more robust. The paper also aims to propose an alternative ranking which is more dependant on the framework than on the methodological choices.

Suggested Citation

  • Saisana, Michaela & d'Hombres, Béatrice & Saltelli, Andrea, 2011. "Rickety numbers: Volatility of university rankings and policy implications," Research Policy, Elsevier, vol. 40(1), pages 165-177, February.
    • Handle: RePEc:eee:respol:v:40:y:2011:i:1:p:165-177
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0048-7333(10)00181-2
    Download Restriction: Full text for ScienceDirect subscribers only
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Philippe Aghion & Mathias Dewatripont & Caroline Hoxby & Andreu Mas-Colell & André Sapir, . "Higher aspirations- an agenda for reforming European universities," Blueprints, Bruegel, number 1, June.
    2. Giuseppe Nicoletti & Stefano Scarpetta & Olivier Boylaud, 2000. "Summary Indicators of Product Market Regulation with an Extension to Employment Protection Legislation," OECD Economics Department Working Papers 226, OECD Publishing.
    3. Sapir, Andre & Aghion, Philippe & Bertola, Giuseppe & Hellwig, Martin & Pisani-Ferry, Jean & Rosati, Dariusz & Vinals, Jose & Wallace, Helen, 2004. "An Agenda for a Growing Europe: The Sapir Report," OUP Catalogue, Oxford University Press, number 9780199271498.
    4. Giuseppe Munda, 2003. "Social Multi-Criteria Evaluation (SMCE)," UHE Working papers 2003_04, Universitat Autònoma de Barcelona, Departament d'Economia i Història Econòmica, Unitat d'Història Econòmica.
    5. Jean-Charles Billaut & Denis Bouyssou & Philippe Vincke, 2009. "Should you believe in the Shanghai ranking? An MCDM view," Working Papers hal-00877050, HAL.
    6. Ellen Hazelkorn, 2007. "The Impact of League Tables and Ranking Systems on Higher Education Decision Making," Higher Education Management and Policy, OECD Publishing, vol. 19(2), pages 1-24.
    7. Daron Acemoglu & Philippe Aghion & Fabrizio Zilibotti, 2006. "Distance to Frontier, Selection, and Economic Growth," Journal of the European Economic Association, MIT Press, vol. 4(1), pages 37-74, March.
    8. Jean-Charles Billaut & Denis Bouyssou & Philippe Vincke, 2010. "Should you believe in the Shanghai ranking?," Scientometrics, Springer;Akadémiai Kiadó, vol. 84(1), pages 237-263, July.
    9. Frederick van der Ploeg, 2008. "Towards Evidence-based Reform of European Universities," CESifo Economic Studies, CESifo Group, vol. 54(2), pages 99-120, June.
    10. Cherchye, Laurens & Knox Lovell, C.A. & Moesen, Wim & Van Puyenbroeck, Tom, 2007. "One market, one number? A composite indicator assessment of EU internal market dynamics," European Economic Review, Elsevier, vol. 51(3), pages 749-779, April.
    11. L Cherchye & W Moesen & N Rogge & T Van Puyenbroeck & M Saisana & A Saltelli & R Liska & S Tarantola, 2008. "Creating composite indicators with DEA and robustness analysis: the case of the Technology Achievement Index," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 59(2), pages 239-251, February.
    12. Bas Jacobs & Frederick van der Ploeg, 2006. "Guide to reform of higher education: a European perspective [‘The efficiency of Australian universities: a data envelopment analysis’]," Economic Policy, CEPR, CESifo, Sciences Po;CES;MSH, vol. 21(47), pages 536-592.
    13. Donald A Brand & Michaela Saisana & Lisa A Rynn & Fulvia Pennoni & Albert B Lowenfels, 2007. "Comparative Analysis of Alcohol Control Policies in 30 Countries," PLOS Medicine, Public Library of Science, vol. 4(4), pages 1-8, April.
    14. Jamil Salmi, 2009. "The Challenge of Establishing World-Class Universities," World Bank Publications - Books, The World Bank Group, number 2600.
    15. Frederik Booysen, 2002. "An Overview and Evaluation of Composite Indices of Development," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 59(2), pages 115-151, August.
    16. Anthony F. J. van Raan, 2005. "Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods," Scientometrics, Springer;Akadémiai Kiadó, vol. 62(1), pages 133-143, January.
    17. Andrea Saltelli, 2007. "Composite Indicators between Analysis and Advocacy," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 81(1), pages 65-77, March.
    18. Giuseppe Munda, 2008. "Social Multi-Criteria Evaluation for a Sustainable Economy," Springer Books, Springer, number 978-3-540-73703-2, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Salvatore Greco & Alessio Ishizaka & Menelaos Tasiou & Gianpiero Torrisi, 2019. "On the Methodological Framework of Composite Indices: A Review of the Issues of Weighting, Aggregation, and Robustness," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 141(1), pages 61-94, January.
    2. Daraio, Cinzia & Bonaccorsi, Andrea & Geuna, Aldo & Lepori, Benedetto & Bach, Laurent & Bogetoft, Peter & F. Cardoso, Margarida & Castro-Martinez, Elena & Crespi, Gustavo & de Lucio, Ignacio Fernandez, 2011. "The European university landscape: A micro characterization based on evidence from the Aquameth project," Research Policy, Elsevier, vol. 40(1), pages 148-164, February.
    3. Annalina Sarra & Eugenia Nissi, 2020. "A Spatial Composite Indicator for Human and Ecosystem Well-Being in the Italian Urban Areas," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 148(2), pages 353-377, April.
    4. P. Zhou & B. Ang, 2009. "Comparing MCDA Aggregation Methods in Constructing Composite Indicators Using the Shannon-Spearman Measure," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 94(1), pages 83-96, October.
    5. Cantone, Giulio Giacomo & Tomaselli, Venera, 2024. "On the Coherence of Composite Indexes: Multiversal Model and Specification Analysis for an Index of Well-Being," MetaArXiv d5y26, Center for Open Science.
    6. Paolo Paruolo & Michaela Saisana & Andrea Saltelli, 2013. "Ratings and rankings: voodoo or science?," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 176(3), pages 609-634, June.
    7. Samira El Gibari & Trinidad Gómez & Francisco Ruiz, 2019. "Building composite indicators using multicriteria methods: a review," Journal of Business Economics, Springer, vol. 89(1), pages 1-24, February.
    8. Andrea Saltelli, 2007. "Composite Indicators between Analysis and Advocacy," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 81(1), pages 65-77, March.
    9. Khatab Alqararah, 2023. "Assessing the robustness of composite indicators: the case of the Global Innovation Index," Journal of Innovation and Entrepreneurship, Springer, vol. 12(1), pages 1-22, December.
    10. Samy Katumba & Julia Kadt & Mark Orkin & Paul Fatti, 2022. "Construction of a Reflective Quality of Life Index for Gauteng Province in South Africa," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 164(1), pages 373-408, November.
    11. Daraio, Cinzia & Bonaccorsi, Andrea & Simar, Léopold, 2015. "Rankings and university performance: A conditional multidimensional approach," European Journal of Operational Research, Elsevier, vol. 244(3), pages 918-930.
    12. Laurens CHERCHYE & Willem MOESEN & Nicky ROGGE & Tom VAN PUYENBROECK, 2009. "Constructing a knowledge economy composite indicator with imprecise data," Working Papers of Department of Economics, Leuven ces09.15, KU Leuven, Faculty of Economics and Business (FEB), Department of Economics, Leuven.
    13. D. Docampo & D. Egret & L. Cram, 2015. "The effect of university mergers on the Shanghai ranking," Scientometrics, Springer;Akadémiai Kiadó, vol. 104(1), pages 175-191, July.
    14. Giménez, Víctor & Thieme, Claudio & Prior, Diego & Tortosa-Ausina, Emili, 2022. "Evaluation and determinants of preschool effectiveness in Chile," Socio-Economic Planning Sciences, Elsevier, vol. 81(C).
    15. Giuseppe Munda, 2015. "Beyond Gdp: An Overview Of Measurement Issues In Redefining ‘Wealth’," Journal of Economic Surveys, Wiley Blackwell, vol. 29(3), pages 403-422, July.
    16. Giuseppe Munda, 2012. "Choosing Aggregation Rules for Composite Indicators," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 109(3), pages 337-354, December.
    17. Micael Castanheira De Moura & Reinhilde Veugelers, 2009. "When and how does education lead to economic growth? A survey with policy suggestions," ULB Institutional Repository 2013/159857, ULB -- Universite Libre de Bruxelles.
    18. P. Zhou & B. Ang & D. Zhou, 2010. "Weighting and Aggregation in Composite Indicator Construction: a Multiplicative Optimization Approach," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 96(1), pages 169-181, March.
    19. Walheer, Barnabé, 2024. "A sequential benefit-of-the-doubt composite indicator," European Journal of Operational Research, Elsevier, vol. 316(1), pages 228-239.
    20. Milica Maricic & Jose A. Egea & Veljko Jeremic, 2019. "A Hybrid Enhanced Scatter Search—Composite I-Distance Indicator (eSS-CIDI) Optimization Approach for Determining Weights Within Composite Indicators," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 144(2), pages 497-537, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:respol:v:40:y:2011:i:1:p:165-177. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/respol .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.