Applications of speech recognition are now widespread, but user-centred evaluation methods
are necessary to ensure their success. Objective evaluation techniques are fairly well established,
but previous subjective techniques have been unstructured and unproven. This paper reports
on the first stage in the development of a questionnaire measure for the Subjective Assessment
of Speech System Interfaces (SASSI). The aim of the research programme is to produce a
valid, reliable and sensitive measure of users' subjective experiences with speech recognition
systems. Such a technique could make an important contribution to theory and practice in the
design and evaluation of speech recognition systems according to best human factors practice.
A prototype questionnaire was designed, based on established measures for evaluating the
usability of other kinds of user interface, and on a review of the research literature into speech
system design. This consisted of 50 statements with which respondents rated their level of
agreement. The questionnaire was given to users of four different speech applications, and
Exploratory Factor Analysis of 214 completed questionnaires was conducted. This suggested
the presence of six main factors in users' perceptions of speech systems: System Response
Accuracy, Likeability, Cognitive Demand, Annoyance, Habitability and Speed. The six factors
have face validity, and a reasonable level of statistical reliability. The findings form a useful
theoretical and practical basis for the subjective evaluation of any speech recognition interface.
However, further work is recommended, to establish the validity and sensitivity of the
approach, before a final tool can be produced which warrants general use.