IDEAS home Printed from https://ideas.repec.org/a/spr/scient/v89y2011i1d10.1007_s11192-011-0419-5.html
   My bibliography  Save this article

Improving quality assessment of composite indicators in university rankings: a case study of French and German universities of excellence

Author

Listed:
  • M. Benito

    (Universidad Carlos III de Madrid)

  • R. Romera

    (Universidad Carlos III de Madrid)

Abstract

Composite indicators play an essential role for benchmarking higher education institutions. One of the main sources of uncertainty building composite indicators and, undoubtedly, the most debated problem in building composite indicators is the weighting schemes (assigning weights to the simple indicators or subindicators) together with the aggregation schemes (final composite indicator formula). Except the ideal situation where weights are provided by the theory, there clearly is a need for improving quality assessment of the final rank linked with a fixed vector of weights. We propose to use simulation techniques to generate random perturbations around any initial vector of weights to obtain robust and reliable ranks allowing to rank universities in a range bracket. The proposed methodology is general enough to be applied no matter the weighting scheme used for the composite indicator. The immediate benefit achieved is a reduction of the uncertainty associated with the assessment of a specific rank which is not representative of the real performance of the university, and an improvement of the quality assessment of composite indicators used to rank. To illustrate the proposed methodology we rank the French and the German universities involved in their respective 2008 Excellence Initiatives.

Suggested Citation

  • M. Benito & R. Romera, 2011. "Improving quality assessment of composite indicators in university rankings: a case study of French and German universities of excellence," Scientometrics, Springer;Akadémiai Kiadó, vol. 89(1), pages 153-176, October.
    • Handle: RePEc:spr:scient:v:89:y:2011:i:1:d:10.1007_s11192-011-0419-5
    DOI: 10.1007/s11192-011-0419-5
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11192-011-0419-5
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11192-011-0419-5?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Marco Geraci & M. Degli Esposti, 2011. "Where do Italian universities stand? An in-depth statistical analysis of national and international rankings," Scientometrics, Springer;Akadémiai Kiadó, vol. 87(3), pages 667-681, June.
    2. Jingda Ding & Junping Qiu, 2011. "An approach to improve the indicator weights of scientific and technological competitiveness evaluation of Chinese universities," Scientometrics, Springer;Akadémiai Kiadó, vol. 86(2), pages 285-297, February.
    3. M. Saisana & A. Saltelli & S. Tarantola, 2005. "Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 168(2), pages 307-323, March.
    4. Isidro F. Aguillo & Judit Bar-Ilan & Mark Levene & José Luis Ortega, 2010. "Comparing university rankings," Scientometrics, Springer;Akadémiai Kiadó, vol. 85(1), pages 243-256, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Giménez, Víctor & Thieme, Claudio & Prior, Diego & Tortosa-Ausina, Emili, 2022. "Evaluation and determinants of preschool effectiveness in Chile," Socio-Economic Planning Sciences, Elsevier, vol. 81(C).
    2. M. M. Segovia-González & I. Contreras, 2023. "A Composite Indicator to Compare the Performance of Male and Female Students in Educational Systems," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 165(1), pages 181-212, January.
    3. Alfio Ferrara & Silvia Salini, 2012. "Ten challenges in modeling bibliographic data for bibliometric analysis," Scientometrics, Springer;Akadémiai Kiadó, vol. 93(3), pages 765-785, December.
    4. M. Ryan Haley, 2020. "Combining the weighted and unweighted Euclidean indices: a graphical approach," Scientometrics, Springer;Akadémiai Kiadó, vol. 123(1), pages 103-111, April.
    5. Rosalia Castellano & Antonella Rocca, 2015. "Assessing the gender gap in labour market index: volatility of results and reliability," International Journal of Social Economics, Emerald Group Publishing Limited, vol. 42(8), pages 749-772, August.
    6. ZAHARIEV, Boyan & BOYADJIEVA, Pepka, 2012. "The Impact Of Weighting Preferences On University Rankings: The Example Of Bulgaria," Regional and Sectoral Economic Studies, Euro-American Association of Economic Development, vol. 12(3).
    7. Ruiz, Francisco & El Gibari, Samira & Cabello, José M. & Gómez, Trinidad, 2020. "MRP-WSCI: Multiple reference point based weak and strong composite indicators," Omega, Elsevier, vol. 95(C).
    8. Elio Atenógenes Villaseñor & Ricardo Arencibia-Jorge & Humberto Carrillo-Calvet, 2017. "Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher education institutions," Scientometrics, Springer;Akadémiai Kiadó, vol. 110(1), pages 77-104, January.
    9. Yelin Fu & Kong Xiangtianrui & Hao Luo & Lean Yu, 2020. "Constructing Composite Indicators with Collective Choice and Interval-Valued TOPSIS: The Case of Value Measure," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 152(1), pages 117-135, November.
    10. Yang Ding & Yelin Fu & Kin Keung Lai & W. K. John Leung, 2018. "Using Ranked Weights and Acceptability Analysis to Construct Composite Indicators: A Case Study of Regional Sustainable Society Index," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 139(3), pages 871-885, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. repec:cte:wsrepe:ws112015 is not listed on IDEAS
    2. Antonio Fernández-Cano & Elvira Curiel-Marin & Manuel Torralbo-Rodríguez & Mónica Vallejo-Ruiz, 2018. "Questioning the Shanghai Ranking methodology as a tool for the evaluation of universities: an integrative review," Scientometrics, Springer;Akadémiai Kiadó, vol. 116(3), pages 2069-2083, September.
    3. Matthias Firgo & Fabian Gabelberger & Andreas Reinstaller & Yvonne Wolfmayr, 2024. "Assessing Regional Production Potential to Strengthen the Security of Supply in Strategic Products," WIFO Working Papers 670, WIFO.
    4. Andrea Saltelli, 2007. "Composite Indicators between Analysis and Advocacy," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 81(1), pages 65-77, March.
    5. Rajko Tomaš, 2022. "Measurement of the Concentration of Potential Quality of Life in Local Communities," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 163(1), pages 79-109, August.
    6. F. Cugnata & G. Perucca & S. Salini, 2017. "Bayesian networks and the assessment of universities' value added," Journal of Applied Statistics, Taylor & Francis Journals, vol. 44(10), pages 1785-1806, July.
    7. Aleksandra Maksimovska & Aleksandar Stojkov, 2019. "Composite Indicator of Social Responsiveness of Local Governments: An Empirical Mapping of the Networked Community Governance Paradigm," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 144(2), pages 669-706, July.
    8. Cherchye, Laurens & Knox Lovell, C.A. & Moesen, Wim & Van Puyenbroeck, Tom, 2007. "One market, one number? A composite indicator assessment of EU internal market dynamics," European Economic Review, Elsevier, vol. 51(3), pages 749-779, April.
    9. Janina Isabel Steinert & Lucie Dale Cluver & G. J. Melendez-Torres & Sebastian Vollmer, 2018. "One Size Fits All? The Validity of a Composite Poverty Index Across Urban and Rural Households in South Africa," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 136(1), pages 51-72, February.
    10. Greco, Salvatore & Ishizaka, Alessio & Tasiou, Menelaos & Torrisi, Gianpiero, 2018. "σ-µ efficiency analysis: A new methodology for evaluating units through composite indices," MPRA Paper 83569, University Library of Munich, Germany.
    11. Marco Marozzi & Mario Bolzan, 2018. "An Index of Household Accessibility to Basic Services: A Study of Italian Regions," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 136(3), pages 1237-1250, April.
    12. Di Zio, Simone & Bolzan, Mario & Marozzi, Marco, 2021. "Classification of Delphi outputs through robust ranking and fuzzy clustering for Delphi-based scenarios," Technological Forecasting and Social Change, Elsevier, vol. 173(C).
    13. Teja Koler-Povh & Primož Južnič & Goran Turk, 2014. "Impact of open access on citation of scholarly publications in the field of civil engineering," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(2), pages 1033-1045, February.
    14. El Gibari, Samira & Gómez, Trinidad & Ruiz, Francisco, 2018. "Evaluating university performance using reference point based composite indicators," Journal of Informetrics, Elsevier, vol. 12(4), pages 1235-1250.
    15. Drago, Carlo & Gatto, Andrea, 2022. "Policy, regulation effectiveness, and sustainability in the energy sector: A worldwide interval-based composite indicator," Energy Policy, Elsevier, vol. 167(C).
    16. Hyejin Park & Han Woo Park, 2018. "Two-side face of knowledge building using scientometric analysis," Quality & Quantity: International Journal of Methodology, Springer, vol. 52(6), pages 2815-2836, November.
    17. Mike Thelwall, 2017. "Judit Bar-Ilan: information scientist, computer scientist, scientometrician," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(3), pages 1235-1244, December.
    18. Rosalia Castellano & Antonella Rocca, 2017. "The dynamic of the gender gap in the European labour market in the years of economic crisis," Quality & Quantity: International Journal of Methodology, Springer, vol. 51(3), pages 1337-1357, May.
    19. Brad Carter & Claus Rinner, 2014. "Locally weighted linear combination in a vector geographic information system," Journal of Geographical Systems, Springer, vol. 16(3), pages 343-361, July.
    20. Paola Costantini & Marielle Linting & Giovanni C. Porzio, 2010. "Mining performance data through nonlinear PCA with optimal scaling," Applied Stochastic Models in Business and Industry, John Wiley & Sons, vol. 26(1), pages 85-101, January.
    21. Murat Perit Çakır & Cengiz Acartürk & Oğuzhan Alaşehir & Canan Çilingir, 2015. "A comparative analysis of global and national university ranking systems," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(3), pages 813-848, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:scient:v:89:y:2011:i:1:d:10.1007_s11192-011-0419-5. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.