Page 139 - JOURNAL OF LIBRARY SCIENCE IN CHINA 2018 Vol. 42

138 Journal of Library Science in China, Vol. 8, 2016

are the three most important evaluation criteria. The expert in human computer interaction Nielsen
divided usability into five dimensions: learnability, memorability, interaction efficiency, error rate,
and user satisfaction (Nielsen, 1993). In addition, ease of use, aesthetic appearance, terminology
etc. are also common evaluation criteria (Tsakonas & Papatheodorou, 2006). On the basis of the
above, we defined the usability evaluation criteria (including four top criteria and five sub-criteria)
for the usage modes of terminology services in information retrieval systems (as shown in Table 2).

Table 2. Usability evaluation criteria of terminology service usage modes
No. Usability evaluation criteria
Effectiveness：the accuracy and completeness of the system for a user to achieve a specific goal
- Representation degree of retrieval requirement：how much the system can help a user to better represent his/her
1 retrieval requirement
- Improvement degree of retrieval results：how much the system can help a user to improve the final retrieval result
- Freedom of term selection：the freedom for a user to select among the terms recommended by the system
Efficiency：consumed resources for a user to complete a specific task with the system
2 - Operation time： the required time when a user optimizes the query term with the system
- Clicking times：the required clicking times when a user optimizes the query term with the system
3 User Satisfaction：a user’s subjective experience while interacting with the system
4 Learnability：how easy to learn and handle the operation and use of the system

After determining the evaluation criteria, we determined the weight of each criterion using
Analytic Hierarchy Process (AHP). AHP is multi-criteria decision-making approach which
decomposes a decision problem into decision goal, criteria, sub-criteria, and alternatives to do
quantitive and qualitative analysis. It was developed by Thomas L. Saaty, an expert in operational
research and a professor in University of Pittsburg, in the 1970s and has been extensively studied
and refined since then because of its advantage of systematicness , flexibility and simplicity
(Saaty, 1980). The rationale of AHP is: firstly find the key elements related to problem solving,
build a hierarchy according to the correlation and subordination of these elements, determine the
relative importance by comparing the same-level criteria two by two, and make a comprehensive
judgement to determine the overall rank of the relative importance of all the criteria. For the
objective of selecting a usage mode of terminology service, the hierarchy is shown in Figure 13.
Once the hierarchy is built, it is easy to construct a decision matrix of the criteria at each level
and then compare the criteria in the matrix two by two at a time based on their relative importance
in respect to the decision goal. The comparisons are performed by human experts and the relative
importance is graded using a scale of relative importance from 1 to 9 (as shown in Table 3). Nielsen
deemed that 5 human evaluators can find 75% usability problems in expert-based evaluation
(Nielsen, 1993). Thus we selected five experts to do judgments, averaged their grades and rounded
up and down to get the final importance decision matrixes of criteria at each level (as shown in
Table 4-Table 6). There are many weight calculation methods, such as sum product method, latent
root method, geometric average method, least square method etc. in AHP, but their calculation