Abstract

In this chapter I describe ongoing research that seeks to provide a common framework for the generation and interpretation of spontaneous gesture in the context of speech. I present a testbed for this framework in the form of a program that generates speech, gesture, and facial expression from underlying rules specifying (a) what speech and gesture are generated on the basis of a given communicative intent, (b) how communicative intent is distributed across communicative modalities, and (c) where one can expect to find gestures with respect to the other communicative acts. Finally, I describe a system that has the capacity to interpret communicative facial, gestural, intonational, and verbal behaviors.

Introduction

I am addressing in this chapter one very particular use of the term “gesture” – that is, hand gestures that co-occur with spoken language. Why such a narrow focus, given that so much of the work on gesture in the human-computer interface community has focused on gestures as their own language – gestures that might replace the keyboard or mouse or speech as a direct command language? Because I don't believe that everyday human users have any more experience with, or natural affinity for, a “gestural language” than they have with DOS commands. We have plenty of experience with actions and the manipulation of objects. But the type of gestures defined as (Väänänen & Böhm, 1993) “body movements which are used to convey some information from one person to another” are in fact primarily found in association with spoken language (90% of gestures are found in the context of speech, according to McNeill, 1992).