IDEAS home Printed from https://ideas.repec.org/a/spr/soinre/v131y2017i3d10.1007_s11205-016-1297-2.html
   My bibliography  Save this article

Sensitivity Analysis: A Necessary Ingredient for Measuring the Quality of a Teaching Activity Index

Author

Listed:
  • María del Carmen Bas

    (Universitat Politècnica de Valencia)

  • Stefano Tarantola

    (EC Joint Research Centre, Econometrics and Applied Statistics)

  • José Miguel Carot

    (Universitat Politècnica de Valencia)

  • Andrea Conchado

    (Universitat Politècnica de Valencia)

Abstract

In recent years, and following the introduction of the European Higher Education Area, universities have developed measurement mechanisms to ensure improvement in the quality of their teaching and teaching staff. One of the measurement tools increasingly used in Higher Education to implement continuous improvement policies for university teaching is composite indicators, which are a mathematical aggregation of a selected set of suitably weighted indicators. Composite indicator building should be accompanied by sensitivity analysis to ensure good practice. However, this is rarely done. Sensitivity analysis helps to improve the understanding and, ultimately, the soundness of the composite. In most cases, sensitivity analysis shows that the weights assigned to indicators do not reflect the actual importance of those indicators in the aggregation to the composite because of the heteroskedasticity of, and correlation between the underlying indicators. This paper proposes a composite indicator for the teaching activity of academic staff in a Spanish university. As we shall see in the paper, the desired weights stated by developers rarely represent the effective importance of the components. Hence, we propose sensitivity analysis as a necessary tool for re-adjusting weights in order to achieve the desired level of importance for each component indicator.

Suggested Citation

  • María del Carmen Bas & Stefano Tarantola & José Miguel Carot & Andrea Conchado, 2017. "Sensitivity Analysis: A Necessary Ingredient for Measuring the Quality of a Teaching Activity Index," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 131(3), pages 931-946, April.
    • Handle: RePEc:spr:soinre:v:131:y:2017:i:3:d:10.1007_s11205-016-1297-2
    DOI: 10.1007/s11205-016-1297-2
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11205-016-1297-2
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11205-016-1297-2?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Paolo Paruolo & Michaela Saisana & Andrea Saltelli, 2013. "Ratings and rankings: voodoo or science?," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 176(3), pages 609-634, June.
    2. Andrea Saltelli, 2007. "Composite Indicators between Analysis and Advocacy," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 81(1), pages 65-77, March.
    3. De Witte, Kristof & Rogge, Nicky & Cherchye, Laurens & Van Puyenbroeck, Tom, 2013. "Economies of scope in research and teaching: A non-parametric investigation," Omega, Elsevier, vol. 41(2), pages 305-314.
    4. Sheila M. Bird & Cox Sir David & Vern T. Farewell & Goldstein Harvey & Holt Tim & Smith Peter C., 2005. "Performance indicators: good, bad, and ugly," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 168(1), pages 1-27, January.
    5. Pilar Murias & José Miguel & David Rodríguez, 2008. "A Composite Indicator for University Quality Assesment: The Case of Spanish Higher Education System," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 89(1), pages 129-146, October.
    6. Tarantola, S. & Gatelli, D. & Mara, T.A., 2006. "Random balance designs for the estimation of first order global sensitivity indices," Reliability Engineering and System Safety, Elsevier, vol. 91(6), pages 717-727.
    7. M. Saisana & A. Saltelli & S. Tarantola, 2005. "Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 168(2), pages 307-323, March.
    8. Bana e Costa, Carlos A. & Oliveira, Mónica D., 2012. "A multicriteria decision analysis model for faculty evaluation," Omega, Elsevier, vol. 40(4), pages 424-436.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Nitika Mundetia & Devesh Sharma & Swatantra Kumar Dubey & Madhu Priya, 2018. "Development and Poverty Assessment Using an Alternate Non-compensatory Composite Index of Rajasthan State in India," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 140(1), pages 1-34, November.
    2. Marozzi Marco & Chowdhury Shovan, 2020. "An index of teaching performance based on students’ feedback," Monte Carlo Methods and Applications, De Gruyter, vol. 26(2), pages 83-91, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Milica Maricic & Jose A. Egea & Veljko Jeremic, 2019. "A Hybrid Enhanced Scatter Search—Composite I-Distance Indicator (eSS-CIDI) Optimization Approach for Determining Weights Within Composite Indicators," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 144(2), pages 497-537, July.
    2. Paolo Paruolo & Michaela Saisana & Andrea Saltelli, 2013. "Ratings and rankings: voodoo or science?," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 176(3), pages 609-634, June.
    3. Greco, Salvatore & Ishizaka, Alessio & Tasiou, Menelaos & Torrisi, Gianpiero, 2018. "σ-µ efficiency analysis: A new methodology for evaluating units through composite indices," MPRA Paper 83569, University Library of Munich, Germany.
    4. Rachel M. Gisselquist, 2013. "Evaluating Governance Indexes: Critical and Less Critical Questions," WIDER Working Paper Series wp-2013-068, World Institute for Development Economic Research (UNU-WIDER).
    5. Greco, Salvatore & Ishizaka, Alessio & Tasiou, Menelaos & Torrisi, Gianpiero, 2019. "Sigma-Mu efficiency analysis: A methodology for evaluating units through composite indicators," European Journal of Operational Research, Elsevier, vol. 278(3), pages 942-960.
    6. Sandra E. Parada & Olga Blasco-Blasco & Vicente Liern, 2019. "Adequacy Indicators Based on Pre-established Goals: An Implementation in a Colombian University," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 143(1), pages 1-24, May.
    7. Michela Gnaldi & M. Giovanna Ranalli, 2016. "Measuring University Performance by Means of Composite Indicators: A Robustness Analysis of the Composite Measure Used for the Benchmark of Italian Universities," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 129(2), pages 659-675, November.
    8. Biggeri, Mario & Clark, David A. & Ferrannini, Andrea & Mauro, Vincenzo, 2019. "Tracking the SDGs in an ‘integrated’ manner: A proposal for a new index to capture synergies and trade-offs between and within goals," World Development, Elsevier, vol. 122(C), pages 628-647.
    9. Gisselquist, Rachel M., 2013. "Evaluating Governance Indexes: Critical and Less Critical Questions," WIDER Working Paper Series 068, World Institute for Development Economic Research (UNU-WIDER).
    10. Salvatore Greco & Alessio Ishizaka & Menelaos Tasiou & Gianpiero Torrisi, 2019. "On the Methodological Framework of Composite Indices: A Review of the Issues of Weighting, Aggregation, and Robustness," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 141(1), pages 61-94, January.
    11. P. Zhou & B. Ang, 2009. "Comparing MCDA Aggregation Methods in Constructing Composite Indicators Using the Shannon-Spearman Measure," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 94(1), pages 83-96, October.
    12. Cantone, Giulio Giacomo & Tomaselli, Venera, 2024. "On the Coherence of Composite Indexes: Multiversal Model and Specification Analysis for an Index of Well-Being," MetaArXiv d5y26, Center for Open Science.
    13. Ziwei Shu & Ramón Alberto Carrasco & Javier Portela García-Miguel & Manuel Sánchez-Montañés, 2022. "Multiple Scenarios of Quality of Life Index Using Fuzzy Linguistic Quantifiers: The Case of 85 Countries in Numbeo," Mathematics, MDPI, vol. 10(12), pages 1-28, June.
    14. See, Kok Fong & Ng, Ying Chu & Yu, Ming-Miin, 2022. "An alternative assessment approach to national higher education system evaluation," Evaluation and Program Planning, Elsevier, vol. 94(C).
    15. P. Zhou & B. Ang & D. Zhou, 2010. "Weighting and Aggregation in Composite Indicator Construction: a Multiplicative Optimization Approach," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 96(1), pages 169-181, March.
    16. Giuseppina Guagnano & Maria Rita Sebastiani, 2018. "Away from Dissatisfaction, Closer to Well-Being: A Multidimensional Synthetic Measure," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 136(3), pages 977-997, April.
    17. Van Puyenbroeck, Tom & Montalto, Valentina & Saisana, Michaela, 2021. "Benchmarking culture in Europe: A data envelopment analysis approach to identify city-specific strengths," European Journal of Operational Research, Elsevier, vol. 288(2), pages 584-597.
    18. Giovanna Boccuzzo & Licia Maron, 2017. "Proposal of a composite indicator of job quality based on a measure of weighted distances," Quality & Quantity: International Journal of Methodology, Springer, vol. 51(5), pages 2357-2374, September.
    19. Khatab Alqararah, 2023. "Assessing the robustness of composite indicators: the case of the Global Innovation Index," Journal of Innovation and Entrepreneurship, Springer, vol. 12(1), pages 1-22, December.
    20. Olga Blasco-Blasco & Marina Liern-García & Aarón López-García & Sandra E. Parada-Rico, 2021. "An Academic Performance Indicator Using Flexible Multi-Criteria Methods," Mathematics, MDPI, vol. 9(19), pages 1-19, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:soinre:v:131:y:2017:i:3:d:10.1007_s11205-016-1297-2. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.