The Impact Of Weighting Preferences On University Rankings: The Example Of Bulgaria

On the basis of data from the official Bulgarian University Ranking System (BURS launched in 2011 and updated in 2012), the selection of weights for different indicators in a university ranking system is discussed. Weights represent the importance assigned to the different dimensions of performance measured through ranking, and can therefore be regarded as quantified value statements. Weight preferences differ across groups of stakeholders or even from one agent to another. In this paper a number of stochastic simulations using 4 initial sets of weights are made. The actual weights applied to BURS —simple uniform weights— are derived from a survey encompassing 15,000 university students as well as weights determined by university rectors. The results show that rankings, whose value is relatively neutral —with a set of weights closer to uniformity— are more vulnerable to small random fluctuations. Value laden rankings with asymmetric weights as in the case of BURS are more resilient to slight disturbance(s) however are very unstable in the case of the rearrangement of priorities. A consistent ranking system demands asymmetric weighting, which means strong value statements and prioritisation. The problem however with such a ranking system is that if heavily weighted preferences do not capture important elements of performance, the whole system can become not just marginally imprecise but also very erroneous.