Page 120 - Journal of Library Science in China 2020 Vol.46

Shelia X. WEI, Ronda J. ZHANG, Howell Y. WANG, CAO CONG & Fred Y. YE / 119
Rankings never really quantify contributions: A quantitative and qualitative study on universities and their libraries

Table 1. Three simple synthetic criterion systems: Designed indicators and weights
Overall Score T=0.50A+0.50B
Criterion Different indicator weights
Criterion Indicator
weight S1 S2 S3
Papers in ESI 0.20 0.30 0.10
Quantity 0.50 Papers in nature 0.40 0.35 0.45
(A)
Papers in science 0.40 0.35 0.45
Highly cited papers in ESI 0.30 0.35 0.25
Contribution (B) 0.50 Citations in ESI 0.30 0.35 0.25
Nobel Laureates 0.40 0.30 0.50

However, the contribution (B) in Table 1 is only false appearance, where most indicators
represent quantitative statistics, not real academic or knowledge contributions. Meanwhile,
research has shown that assigning different weights to the same indicators could lead to
different ranking outcomes. Dehon, Mccathie, & Verardi (2010) show such changes in the top
10 universities and some European universities, and Saisana’s research (Saisana, d’Hombres, &

Saltelli, 2011) includes the combination changes of weights, normalization rule, and the number of
indicators, both of which indicate that the top universities are less affected by the changes and that
most universities’ rankings fluctuate greatly. We try to present the mechanism based on a single
aspect (weight) change. Figure 1 provides a mechanism of different rankings, where we compute
different synthetic criteria (S) according to different design criterion systems (S1, S2 and S3) that
consist of a few indictors with variable weights. Supposing S1 gives a linear result, S2 and S3 will
be non-linear fluctuation around S1. Consequently, a university (Ui) could be located in different
positions given different Sj as Rj, so that its ranking position could change in different systems.
Except for the top one institution, others’ rankings are changeable. Farther the original point left
away, larger the differences would exist.

Figure 1. A ranking mechanism diagram in different criterion systems