Introduction

Comprehensive, generalizable evaluations of digital libraries (DLs) are rare. Where evaluation does occur, it is generally minimal. Saracevic (2004) analysed around 80 evaluations to conclude that, both in scientific research and in practice, thorough evaluations of DLs are rather the exception than the rule. The complexity of DL systems can be identified as one reason for this. Examining them in their entirety is not straightforward. Even when attempting to do so, we lack scientifically accepted concepts, approaches and models. Another reason is the allocated funding in DL projects. Evaluation is always a ‘must have’ stated by the funder, but is rarely supported by adequate resources. These reasons are, unfortunately, still valid at the time of writing.

In this chapter the method of expert evaluation is presented and shown to be one possible way of addressing such problems. Expert evaluations are heuristic or qualitative in nature, as opposed to quantitative evaluations, which aim to provide statistically significant results. Heuristic evaluations (Nielsen, 1994) are common in usability engineering, where user interfaces are evaluated by a small group of experts on the basis of their conformity to certain usability principles (heuristics). Expert evaluations differ from other types of heuristic evaluation in that they lack predefined heuristics. The experts are free to provide any comment, on the assumption that their views will be informed ones. Some examples of the 10 general heuristics defined by Nielsen are visibility of the system status, error recognition, error prevention, user control (support undo and redo), aesthetics and minimalistic design.

The advantages of such an expert evaluation are fast and cost-effective results, in contrast to the more expensive types of qualitative user study, which require a larger number of evaluators in order to reflect a representative result. We can distinguish further between two types of expert evaluation: in the first case, the experts themselves are the evaluators, conducting the evaluation and providing the results. In the second, the experts are monitored by evaluators, who lead the evaluation and assess the results.