Introduction

Social signal processing (SSP) is the computing domain aimed at modeling, analysis, and synthesis of social signals in human–human and human–machine interactions (Pentland, 2007; Vinciarelli et al., 2008, 2012; Vinciarelli, Pantic, & Bourlard, 2009). According to different theoretic orientations, social signals can be defined in different ways, for example, “acts or structures that influence the behavior or internal state of other individuals” (Mehu & Scherer, 2012; italics in original), “communicative or informative signals which … provide information about social facts” (Poggi & D'Errico, 2012; italics in original), or “actions whose function is to bring about some reaction or to engage in some process” (Brunet & Cowie, 2012; italics in original). The definitions might appear different, but there seems to be consensus on at least three points.

1. • Social signals are observable behaviors that people display during social interactions.

2. • The social signals of an individual A produce changes in others (e.g., the others develop an impression or a belief about A, react to A with appropriate social signals, or coordinate their social signals with those of A).

3. • The changes produced by the social signals of A in others are not random, but follow principles and laws.

In a computing perspective, the observations above lead to the key idea that shapes the field of Social Signal Processing, namely that social signals are the physical, machine detectable trace of social and psychological phenomena not otherwise accessible to direct observation. In fact, SSP addresses the following three main problems.

1. • Modeling: identification of principles and laws that govern the use of social signals.

2. • Analysis: automatic detection and interpretation of social signals in terms of the principles and laws above.

3. • Synthesis: automatic generation of artificial social signals following the principles and laws above.

Correspondingly, this book is organized into four main sections of which the first three focus on the three problems outlined above while the fourth one introduces current applications of SSP technologies.