Improving quality assessment of composite indicators in university rankings: a case study of French and German universities of excellence

Abstract

Composite indicators play an essential role for benchmarking higher education institutions. One of the main sources of uncertainty building composite indicators and, undoubtedly, the most debated problem in building composite indicators is the weighting schemes (assigning weights to the simple indicators or subindicators) together with the aggregation schemes (final composite indicator formula). Except the ideal situation where weights are provided by the theory, there clearly is a need for improving quality assessment of the final rank linked with a fixed vector of weights. We propose to use simulation techniques to generate random perturbations around any initial vector of weights to obtain robust and reliable ranks allowing to rank universities in a range bracket. The proposed methodology is general enough to be applied no matter the weighting scheme used for the composite indicator. The immediate benefit achieved is a reduction of the uncertainty associated with the assessment of a specific rank which is not representative of the real performance of the university, and an improvement of the quality assessment of composite indicators used to rank. To illustrate the proposed methodology we rank the French and the German universities involved in their respective 2008 Excellence Initiatives.

Appendices

Appendix 1. Monte Carlo schemes

Algorithm 1

• (a) Let w ⁽⁰⁾ = (w ₁,…,w _p ). Fix the radius s and the sample size m.

• (b) Generate p − 1 uniform values w ⁽¹⁾₁ ,…, w ⁽¹⁾ _p−1 on (w ₁ − s,w ₁ + s), (w ₂ − s,w ₂ + s),…, (w _p−1 − s,w _p−1 + s), respectively.

• (c) If (1 − (w ⁽¹⁾₁ ,…, w ⁽¹⁾ _p−1 )) belongs to the interval (w _p  − s,w _p  + s) then w ⁽¹⁾ = (w ⁽¹⁾₁ ,…, w ⁽¹⁾ _p−1 , 1 − (w ⁽¹⁾₁ ,…, w ⁽¹⁾ _p−1 )), otherwise reject.

• (d) Iterate steps (b) and (c) to get w ⁽¹⁾… w ^(m).

Figure 5 shows the corresponding surface in ℜ³ for the initial vector of weights w ⁽⁰⁾ = (1/3,1/3,1/3) when the perturbations are generated around this point with s = 0.2w _j following Algorithm 1.

Fig. 5

Points randomly generated for the initial vector of weights w ⁽⁰⁾ = (1/3,1/3,1/3) such as they live in the intersection of the 3-dimensional hypercube and the 2-simplex in R³

Algorithm 2

• (a) Let w ⁽⁰⁾ = (w ₁,…,w _p ). Fix the radius s and the sample size m.

• (b) Generate p uniform values w ⁽¹⁾₁ ,…, w ⁽¹⁾ _p on (w ₁ − s,w ₁ + s), (w ₂ − s,w ₂ + s), …, (w _p  − s,w _p  + s), respectively.

• (c) If (w ⁽¹⁾₁ )² +···+ (w ⁽¹⁾ _p )² ≤ s ², and w ₁ ⁽¹⁾ +…+ w _p ⁽¹⁾ = 1, then w ⁽¹⁾ = (w ⁽¹⁾₁ ,…, w ⁽¹⁾ _p ), otherwise reject and re-select p uniform values following step b.

• (d) Iterate steps (b) and (c) to get w ⁽¹⁾… w ^(m).

Appendix 2. Robust ranking using different values of the level of perturbation

See Tables 9 and 10.

Table 9 Robust ranking for evaluating the Academic Profile performance of the excellence French universities using different values of the level of perturbation s

Table 10 Robust ranking for evaluating the Academic Profile performance of the excellence German universities using different values of the perturbations