Introduction

Social and behavioral signals carry invaluable information regarding how audiences perceive the multimedia content. Assessing the responses from the audience, we can generate tags, summaries, and other forms of metadata for multimedia representation and indexing. Tags are a form of metadata which enables a retrieval system to find and re-find the content of interest (Larson et al., 2011). Unlike classic tagging schemes where users’ direct input is needed, implicit human-centered tagging (IHCT) was proposed (Pantic & Vinciarelli, 2009) to generate tags without any specific input or effort from users. Translating the behavioral responses into tags results in “implicit” tags since there is no need for users’ direct input as reactions to multimedia are displayed spontaneously (Soleymani & Pantic, 2012).

User generated explicit tags are not always assigned with the intention of describing the content and might be given to promote the users themselves (Pantic & Vinciarelli, 2009). Implicit tags have the advantage of being detected for a certain goal relevant to a given application. For example, an online radio interested in the mood of its songs can assess listeners’ emotions; a marketing company is interested in assessing the success of its video advertisements.

It is also worth mentioning that implicit tags can be a complementary source of information in addition to the existing explicit tags. They can also be used to filter out the tags which are not relevant to the content (Soleymani & Pantic, 2013; Soleymani, Kaltwang, & Pantic, 2013). A scheme of implicit tagging versus explicit tagging is shown in Figure 26.1. Recently, we have been witnessing a growing interest from industry on this topic (Klinghult, 2012; McDuff, El Kaliouby, & Picard, 2012; Fleureau, Guillotel, & Orlac, 2013; Silveira et al., 2013) which is a sign of its significance.

Analyzing spontaneous reactions to multimedia content can assist multimedia indexing with the following scenarios: (i) direct translation to tags – users’ spontaneous reactions will be translated into emotions or preference, e.g., interesting, funny, disgusting, scary (Kierkels, Soleymani, & Pun, 2009; Soleymani, Pantic, & Pun, 2012; Petridis & Pantic, 2009; Koelstra et al., 2010; Silveira et al., 2013; Kurdyukova, Hammer, & Andr, 2012);

According to a recent survey on social signal processing (Vinciarelli, Pantic, & Bourlard, 2009), next-generation computing needs to implement the essence of social intelligence including the ability to recognize human social signals and social behaviors, such as turn taking, politeness, and disagreement, in order to become more effective and more efficient. Social signals and social behaviors are the expression of one's attitude towards social situation and interplay, and they are manifested through a multiplicity of nonverbal behavioral cues, including facial expressions, body postures and gestures, and vocal outbursts like laughter. Of the many social signals, only face, eye, and posture cues are capable of informing us about all identified social behaviors. During social interaction, it is a social norm that one looks their dyadic partner in the eyes, clearly focusing one's vision on the face. Facial expressions thus make for very powerful social signals. As one of the most comprehensive and objective ways to describe facial expressions, the facial action coding system (FACS) has recently received significant attention. Automating FACS coding would greatly benefit social signal processing, opening up new avenues to understanding how we communicate through facial expressions. In this chapter we provide a comprehensive overview of research into machine analysis of facial actions. We systematically review all components of such systems: pre-processing, feature extraction, and machine coding of facial actions. In addition, the existing FACS-coded facial expression databases are summarized. Finally, challenges that have to be addressed to make automatic facial action analysis applicable in real-life situations are extensively discussed.

Introduction

Scientific work on facial expressions can be traced back to at least 1872 when Charles Darwin published The Expression of the Emotions in Man and Animals (1872). He explored the importance of facial expressions for communication and described variations in facial expressions of emotions. Today, it is widely acknowledged that facial expressions serve as the primary nonverbal social signal for human beings, and are responsible for a large part to regulate interactions with each other (Ekman & Ronsenberg, 2005). They communicate emotions, clarify and emphasize what is being said, and signal comprehension, disagreement, and intentions (Pantic, 2009).