Introduction

Evaluation of online systems, services and resources, and latterly digital libraries (DLs), has a long tradition of improving the state of the art in information retrieval (IR) technology. The traditional systemoriented experiments for IR system evaluation provide a measure of system performance in terms of items retrieved and relevant to a given query. Yet today's enduser interactive DL environments demand useroriented measures of usability and user behaviour, perceptions and preferences (Harter and Hert, 1997; Su, 2003). User satisfaction is often used to evaluate a system from the users’ perspective and, as Gluck (1996) notes, ‘in spite of the definitional and operational inadequacies these two traditional measures [user satisfaction and system performance] have led to considerable research and notable improvements’ (p. 92). Controversy surrounding the user satisfaction approach arises mostly from the very nature of ‘satisfaction’, a user state determined by the influences of many factors – be they elements or characteristics of the system, the users or the task situation (Griffiths et al., 2007; Al-Maskari and Sanderson, 2010). Melone (1990) argued that it is this lack of attention to the theoretical underpinnings of the user satisfaction construct that has led to disagreements amongst researchers regarding its definition, method of measurement and research findings. In spite of the debate, user satisfaction holds considerable value and appeal as a construct for usercentred system evaluation within both the IR and the information systems literature.

This chapter identifies practical applications of methods that elicit user engagement with the process of evaluation, using experience with the DiSCmap project for illustration.

The DiSCmap project

The ‘Digitisation in Special Collections: mapping, assessment and prioritisation’ (DiSCmap) project, funded by the Joint Information Systems Committee (JISC) and the Research Information Network (RIN) and run jointly by the Centre for Digital Library Research (CDLR) and the Centre for Research in Libraries, Information and Media (CeRLIM), took a collaborative approach to the creation of a userdriven digitization prioritization framework, encouraging participation and collective engagement between LIS professionals and patrons (Birrell et al., 2011). Between September 2008 and March 2009 the DiSCmap project team asked over 1200 intermediaries and endusers a variety of questions about which physical and digital special collections they made use of and what criteria they felt must be considered when selecting materials for digitization.