IDEAS home Printed from https://ideas.repec.org/a/spr/soinre/v129y2016i2d10.1007_s11205-015-1116-1.html
   My bibliography  Save this article

Measuring University Performance by Means of Composite Indicators: A Robustness Analysis of the Composite Measure Used for the Benchmark of Italian Universities

Author

Listed:
  • Michela Gnaldi

    (University of Perugia)

  • M. Giovanna Ranalli

    (University of Perugia)

Abstract

Composite indicators (CIs) are a useful communication tool for conveying summary performance information in a relatively simple way. They are used widely in various sectors in public services. In Italy, they are also employed to allocate a quota of public funding to the universities. Despite increasing use, CIs value and benefit are still under debate and their suitability for the benchmark of universities is challenged by methodological issues and the multi-dimensional facet of higher education institutions. The issue addressed in the present work is whether the composite measure employed by the Italian Government is an accurate tool for evaluating, ranking and funding universities. For this purpose—after a revision of the indicators, weighting and aggregation methods put forward at the national level—CIs have been computed for the overall university performance, employing the indicators chosen at the national level and combining various weighting and normalisation schemes. The variation in the rankings assigned by the CIs to the given universities has been then explored to gauge its robustness. In the light of the results, some technical and analytical issues involved in the design of CIs and their policy implications are also discussed.

Suggested Citation

  • Michela Gnaldi & M. Giovanna Ranalli, 2016. "Measuring University Performance by Means of Composite Indicators: A Robustness Analysis of the Composite Measure Used for the Benchmark of Italian Universities," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 129(2), pages 659-675, November.
    • Handle: RePEc:spr:soinre:v:129:y:2016:i:2:d:10.1007_s11205-015-1116-1
    DOI: 10.1007/s11205-015-1116-1
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11205-015-1116-1
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11205-015-1116-1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. P. Zhou & B. Ang & D. Zhou, 2010. "Weighting and Aggregation in Composite Indicator Construction: a Multiplicative Optimization Approach," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 96(1), pages 169-181, March.
    2. Marco Marozzi, 2014. "Construction, dimension reduction and uncertainty analysis of an index of trust in public institutions," Quality & Quantity: International Journal of Methodology, Springer, vol. 48(2), pages 939-953, March.
    3. Andrea Saltelli, 2007. "Composite Indicators between Analysis and Advocacy," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 81(1), pages 65-77, March.
    4. L Cherchye & W Moesen & N Rogge & T Van Puyenbroeck & M Saisana & A Saltelli & R Liska & S Tarantola, 2008. "Creating composite indicators with DEA and robustness analysis: the case of the Technology Achievement Index," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 59(2), pages 239-251, February.
    5. Saisana, Michaela & d'Hombres, Béatrice & Saltelli, Andrea, 2011. "Rickety numbers: Volatility of university rankings and policy implications," Research Policy, Elsevier, vol. 40(1), pages 165-177, February.
    6. Pilar Murias & José Miguel & David Rodríguez, 2008. "A Composite Indicator for University Quality Assesment: The Case of Spanish Higher Education System," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 89(1), pages 129-146, October.
    7. Marco Marozzi, 2009. "A composite indicator dimension reduction procedure with application to university student satisfaction," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 63(3), pages 258-268, August.
    8. Jorge Guardiola & Andrés Picazo-Tadeo, 2014. "Building Weighted-Domain Composite Indices of Life Satisfaction with Data Envelopment Analysis," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 117(1), pages 257-274, May.
    9. Hussein Sayed & Ramadan Hamed & Mohamed Ramadan & Samaa Hosny, 2015. "Using Meta-goal Programming for a New Human Development Indicator with Distinguishable Country Ranks," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 123(1), pages 1-27, August.
    10. Giuseppe Munda & Michela Nardo & Michaela Saisana & Tanja Srebotnjak, 2009. "Measuring uncertainties in composite indicators of sustainability," International Journal of Environmental Technology and Management, Inderscience Enterprises Ltd, vol. 11(1/2/3), pages 7-26.
    11. M. Saisana & A. Saltelli & S. Tarantola, 2005. "Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 168(2), pages 307-323, March.
    12. P. Zhou & B. Ang, 2009. "Comparing MCDA Aggregation Methods in Constructing Composite Indicators Using the Shannon-Spearman Measure," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 94(1), pages 83-96, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. M. M. Segovia-González & I. Contreras, 2023. "A Composite Indicator to Compare the Performance of Male and Female Students in Educational Systems," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 165(1), pages 181-212, January.
    2. Michela Gnaldi & Simone Del Sarto, 2018. "Variable Weighting via Multidimensional IRT Models in Composite Indicators Construction," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 136(3), pages 1139-1156, April.
    3. Marco Centoni & Vieri Del Panta & Antonello Maruotti & Valentina Raponi, 2019. "Concomitant-Variable Latent-Class Beta Inflated Models to Assess Students’ Performance: An Italian Case Study," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 146(1), pages 7-18, November.
    4. Edson Kogachi & Adonias Ferreira & Carlos Cavalcante & Marcelo Embiruçu, 2021. "Development of Performance Evaluation Indicators for Table Grape Packaging Units. 2. Global Indexes," Sustainability, MDPI, vol. 13(11), pages 1-16, June.
    5. Sandra E. Parada & Olga Blasco-Blasco & Vicente Liern, 2019. "Adequacy Indicators Based on Pre-established Goals: An Implementation in a Colombian University," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 143(1), pages 1-24, May.
    6. Giorgio E. Montanari & Marco Doretti, 2019. "Ranking Nursing Homes’ Performances Through a Latent Markov Model with Fixed and Random Effects," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 146(1), pages 307-326, November.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Marco Marozzi, 2015. "Measuring Trust in European Public Institutions," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 123(3), pages 879-895, September.
    2. Milica Maricic & Jose A. Egea & Veljko Jeremic, 2019. "A Hybrid Enhanced Scatter Search—Composite I-Distance Indicator (eSS-CIDI) Optimization Approach for Determining Weights Within Composite Indicators," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 144(2), pages 497-537, July.
    3. Sandra E. Parada & Olga Blasco-Blasco & Vicente Liern, 2019. "Adequacy Indicators Based on Pre-established Goals: An Implementation in a Colombian University," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 143(1), pages 1-24, May.
    4. Marco Marozzi, 2016. "Construction, Robustness Assessment and Application of an Index of Perceived Level of Socio-Economic Threat from Immigrants: A Study of 47 European Countries and Regions," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 128(1), pages 413-437, August.
    5. Olga Blasco-Blasco & Marina Liern-García & Aarón López-García & Sandra E. Parada-Rico, 2021. "An Academic Performance Indicator Using Flexible Multi-Criteria Methods," Mathematics, MDPI, vol. 9(19), pages 1-19, September.
    6. Salvatore Greco & Alessio Ishizaka & Menelaos Tasiou & Gianpiero Torrisi, 2019. "On the Methodological Framework of Composite Indices: A Review of the Issues of Weighting, Aggregation, and Robustness," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 141(1), pages 61-94, January.
    7. Cantone, Giulio Giacomo & Tomaselli, Venera, 2024. "On the Coherence of Composite Indexes: Multiversal Model and Specification Analysis for an Index of Well-Being," MetaArXiv d5y26, Center for Open Science.
    8. P. Zhou & B. Ang & D. Zhou, 2010. "Weighting and Aggregation in Composite Indicator Construction: a Multiplicative Optimization Approach," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 96(1), pages 169-181, March.
    9. Marco Marozzi, 2021. "Perceived Justifiability Towards Morally Debatable Behaviors Across Europe," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 153(2), pages 759-778, January.
    10. Marco Marozzi & Mario Bolzan, 2018. "An Index of Household Accessibility to Basic Services: A Study of Italian Regions," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 136(3), pages 1237-1250, April.
    11. Chao Shi & Kenneth C. Land, 2021. "The Data Envelopment Analysis and Equal Weights/Minimax Methods of Composite Social Indicator Construction: a Methodological Study of Data Sensitivity and Robustness," Applied Research in Quality of Life, Springer;International Society for Quality-of-Life Studies, vol. 16(4), pages 1689-1716, August.
    12. Yang Ding & Yelin Fu & Kin Keung Lai & W. K. John Leung, 2018. "Using Ranked Weights and Acceptability Analysis to Construct Composite Indicators: A Case Study of Regional Sustainable Society Index," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 139(3), pages 871-885, October.
    13. Riccardo Natoli & Segu Zuhair, 2011. "Measuring Progress: A Comparison of the GDP, HDI, GS and the RIE," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 103(1), pages 33-56, August.
    14. Drago, Carlo & Gatto, Andrea, 2023. "Gauging energy poverty in developing countries with a composite metric of electricity access," Utilities Policy, Elsevier, vol. 81(C).
    15. Giménez, Víctor & Thieme, Claudio & Prior, Diego & Tortosa-Ausina, Emili, 2022. "Evaluation and determinants of preschool effectiveness in Chile," Socio-Economic Planning Sciences, Elsevier, vol. 81(C).
    16. Annalina Sarra & Eugenia Nissi, 2020. "A Spatial Composite Indicator for Human and Ecosystem Well-Being in the Italian Urban Areas," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 148(2), pages 353-377, April.
    17. P. Zhou & B. Ang, 2009. "Comparing MCDA Aggregation Methods in Constructing Composite Indicators Using the Shannon-Spearman Measure," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 94(1), pages 83-96, October.
    18. Tianjiao Wang & Yelin Fu, 2020. "Constructing Composite Indicators with Individual Judgements and Best–Worst Method: An Illustration of Value Measure," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 149(1), pages 1-14, May.
    19. Yelin Fu & Kong Xiangtianrui & Hao Luo & Lean Yu, 2020. "Constructing Composite Indicators with Collective Choice and Interval-Valued TOPSIS: The Case of Value Measure," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 152(1), pages 117-135, November.
    20. Marozzi Marco & Chowdhury Shovan, 2020. "An index of teaching performance based on students’ feedback," Monte Carlo Methods and Applications, De Gruyter, vol. 26(2), pages 83-91, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:soinre:v:129:y:2016:i:2:d:10.1007_s11205-015-1116-1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.