Skip to main content Accessibility help
Hostname: page-component-8bbf57454-kkbsd Total loading time: 0.579 Render date: 2022-01-26T06:46:29.939Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "metricsAbstractViews": false, "figures": true, "newCiteModal": false, "newCitedByModal": true, "newEcommerce": true, "newUsageEvents": true }

12 - Automatic Analysis of Bodily Social Signals

from Part II - Machine Analysis of Social Signals

Published online by Cambridge University Press:  13 July 2017

Ronald Poppe
University of Twente
Judee K. Burgoon
University of Arizona
Nadia Magnenat-Thalmann
Université de Genève
Maja Pantic
Imperial College London
Alessandro Vinciarelli
University of Glasgow
Get access


The human body plays an important role in face-to-face interactions (Knapp & Hall, 2010; McNeill, 1992). We use our bodies to regulate turns, to display attitudes and to signal attention (Scheflen, 1964). Unconsciously, the body also reflects our affective and mental states (Ekman & Friesen, 1969). There is a long history of research into the bodily behaviors that correlate with the social and affective state of a person, in particular in interaction with others (Argyle, 2010; Dittmann, 1987; Mehrabian, 1968). We will refer to these behaviors as bodily social signals. These social and affective cues can be detected and interpreted by observing the human body's posture and movement (Harrigan, 2008; Kleinsmith & Bianchi-Berthouze, 2013). Automatic observation and analysis has applications such as the detection of driver fatigue and deception, the analysis of interest and mood in interactions with robot companions, and in the interpretation of higher-level phenomena such as mimicry and turn-taking.

In this chapter, we will discuss various bodily social signals, and how to analyze and recognize them automatically. Human motion can be studied on many levels, from the physical level involving muscles and joints, to the level of interpreting a person's fullbody actions and intentions (Poppe, 2007, 2010; Jiang et al., 2013). We will focus on automatically analyzing movements with a relatively short time scale, such as a gesture or posture shift. In the first section, we will discuss the different ways of measurement and coding, both from motion capture data and images and video. The recorded data can subsequently be interpreted in terms of social signals. In the second section, we address the automatic recognition of several bodily social signals. We will conclude the chapter with a discussion of challenges and directions of future work

Measurement of Body Motion

Body movement can be observed and described quantitatively, for example, in terms of joint rotations or qualitatively with movement labels. While social signals are typically detected and identified as belonging to a certain category, body motion is typically described quantitatively. Therefore, the detection of bodily social signals is often based on a quantitative representation of the movement. From the perspective of computation, body motion is most conveniently recorded and measured using motion capture (mocap) devices. However, their obtrusive nature, cost, and the fact that they typically cannot be used outside the laboratory has limited their employment.

Social Signal Processing , pp. 155 - 167
Publisher: Cambridge University Press
Print publication year: 2017

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)


Argyle, Michael (2010). Bodily Communication (2nd rev. edn). New York: Routledge.
Baesler, E. James & Burgoon, Judee K. (1987). Measurement and reliability of nonverbal behavior. Journal of Nonverbal Behavior, 11(4), 205–233.Google Scholar
Bazzani, Loris, Cristani, Marco, Tosato, Diego, et al. (2013). Social interactions by visual focus of attention in a three-dimensional environment. Expert Systems, 30(2), 115–127.Google Scholar
Bente, Gary (1989). Facilities for the graphical computer simulation of head and body movements. Behavior Research Methods, Instruments, & Computers, 21(4), 455–462.Google Scholar
Bente, Gary, Petersen, Anita, Krämer, Nicole C., & De Ruiter, Jan Peter (2001). Transcript-based computer animation of movement: Evaluating a new tool for nonverbal behavior research. Behavior Research Methods, Instruments, & Computers, 33(3), 303–310.Google Scholar
Birdwhistell, Ray L. (1952). Introduction to Kinesics: An Annotation System for Analysis of Body Motion and Gesture. Louisville, KY: University of Louisville.
Bo, Liefeng & Sminchisescu, Cristian (2010). Twin Gaussian processes for structured prediction. International Journal of Computer Vision, 87(1–2), 28–52.Google Scholar
Bousmalis, Konstantinos, Mehu, Marc, & Pantic, Maja. (2013). Towards the automatic detection of spontaneous agreement and disagreement based on nonverbal behaviour: A survey of related cues, databases, and tools. Image and Vision Computing, 31(2), 203–221.Google Scholar
Bull, Peter E. (1987). Posture and Gesture. Oxford: Pergamon Press.
Burba, Nathan, Bolas, Mark, Krum, David M., & Suma, Evan A. (2012). Unobtrusive measurement of subtle nonverbal behaviors with the Microsoft Kinect. In Proceedings of IEEE Virtual Reality Short Papers and Posters March 4–8, 2012, Costa Mesa, CA.
Condon, William S. & Ogston, William D. (1966). Sound film analysis of normal and pathological behavior patterns. Journal of Nervous and Mental Disease, 143(4), 338–347.Google Scholar
Dael, Nele, Mortillaro, Marcello, & Scherer, Klaus R. (2012). The body action and posture coding system (BAP): Development and reliability. Journal of Nonverbal Behavior, 36(2), 97–121.Google Scholar
Deutscher, Jonathan, & Reid, Ian (2005). Articulated body motion capture by stochastic search. International Journal of Computer Vision, 61(2), 185–205.Google Scholar
Dittmann, Allen T. (1987). The role of body movement in communication. In A.W., Siegman & S., Feldstein (Eds), Nonverbal Behavior and Communication (pp. 37–64). Hillsdale, NJ: Lawrence Erlbaum.
Eichner, M., Marin-Jimenez, M., Zisserman, A., & Ferrari, V. (2012). 2D articulated human pose estimation and retrieval in (almost) unconstrained still images. International Journal of Computer Vision, 99(2), 190–214.Google Scholar
Eisler, Richard M., Hersen, Michel, & Agras, W. Stewart (1973). Videotape: A method for the controlled observation of nonverbal interpersonal behavior. Behavior Therapy, 4(3), 420–425.Google Scholar
Ekman, Paul (1965). Communication through nonverbal behavior: A source of information about an interpersonal relationship. In S. S., Tomkins & C. E., Izard (Eds), Affect, Cognition, and Personality (pp. 390–442). New York: Springer.
Ekman, Paul & Friesen, Wallace V. (1969). The repertoire of nonverbal behavior: Categories, origins, usage and coding. Semiotica, 1(1), 49–98.Google Scholar
Felzenszwalb, Pedro F., Girshick, Ross B., McAllester, David, & Ramanan, Deva (2010). Object detection with discriminatively trained part-based models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(9), 1627–1645.Google Scholar
Fragkiadaki, Katerina, Hu, Han, & Shi, Jianbo (2013). Pose from flow and flow from pose. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 2059–2066).
Frey, Siegfried & von Cranach, Mario (1973). A method for the assessment of body movement variability. In M. von, Cranach & I., Vine (Eds), Social Communication and Movement (pp. 389–418). New York: Academic Press.
Ganapathi, Varun, Plagemann, Christian, Koller, Daphne, & Thrun, Sebastian (2010). Real time motion capture using a single time-of-flight camera. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 755–762).
Griffin, Harry J., Aung, Min S. H., Romera-Paredes, Bernardino, et al. (2013). Laughter type recognition from whole body motion. In Proceedings of the International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 349–355).
Groh, Georg, Lehmann, Alexander, Reimers, Jonas, Friess, Marc Rene, & Schwarz, Loren (2010). Detecting social situations from interaction geometry. In Proceedings of the International Conference on Social Computing (SocialCom). (pp. 1–8).
Guan, Peng, Weiss, Alexander, Blan, Alexandru O., & Black, Michael J. (2009). Estimating human shape and pose from a single image. In Proceedings of the International Conference On Computer Vision (ICCV).
Hall, Edward T. (1966). The Hidden Dimension. New York: Doubleday.
Harrigan, Jinni A. (2008). Proxemics, kinesics, and gaze. In J. A., Harrigan & R., Rosenthal (Eds), New Handbook of Methods in Nonverbal Behavior Research (pp. 137–198). Oxford: Oxford University Press.
Heylen, Dirk (2006). Head gestures, gaze and the principles of conversational structure. International Journal of Humanoid Robotics, 3(3), 241–267.Google Scholar
Hirsbrunner, Hans-Peter, Frey, Siegfried, & Crawford, Robert (1987). Movement in human interaction: Description, parameter formation, and analysis. In A.W., Siegman & S., Feldstein (Eds), Nonverbal Behavior and Communication (pp. 99–140). Hillsdale, NJ: Lawrence Erlbaum.
Hutchinson Guest, Ann (2005). Labanotation: The System of Analyzing and Recording Movement (4th edn). New York: Routledge.
Jiang, Yu-Gang, Bhattacharya, Subhabrata, Chang, Shih-Fu, & Shah, Mubarak (2013). High-level event recognition in unconstrained videos. International Journal of Multimedia Information Retrieval, 2(2), 73–101.Google Scholar
Kendon, Adam (1990). Conducting Interaction: Patterns of Behavior in Focused Encounters. Cambridge: Cambridge University Press.
Kleinsmith, Andrea, & Bianchi-Berthouze, Nadia (2013). Affective body expression perception and recognition: A survey. IEEE Transactions on Affective Computing, 4(1), 15–33.Google Scholar
Kleinsmith, Andrea, Bianchi-Berthouze, Nadia, & Steed, Anthony (2011). Automatic recognition of non-acted affective postures. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 41(4), 1027–1038.Google Scholar
Klette, Reinhard, & Tee, Garry (2007). Understanding human motion: A historic review. In B., Rosenhahn, R., Klette, & D., Metaxas (Eds), Human Motion: Understanding, Modelling, Capture and Animation (pp. 1–22). New York: Springer.
Knapp, Mark L., & Hall, Judith A. (2010). Nonverbal Communication in Human Interaction (7th edn). Andover, UK: Cengage Learning.
Lausberg, Hedda & Sloetjes, Han (2009). Coding gestural behavior with the NEUROGES-ELAN system. Behavior Research Methods, 41(3), 841–849.Google Scholar
Marcos-Ramiro, Alvaro, Pizarro-Perez, Daniel, Romera, Marta Marrón, Nguyen, Laurent, & Gatica-Perez, Daniel (2013). Body communicative cue extraction for conversational analysis. In Proceedings of the International Conference on Automatic Face and Gesture Recognition (FG) (pp. 1–8).
McNeill, David (1985). So you think gestures are nonverbal? Psychological Review, 92(3), 350– 371.Google Scholar
McNeill, David (1992). Hand and Mind: What Gestures Reveal About Thought. Chicago: University of Chicago Press.
Mead, Ross, Atrash, Amin, & Matarić, Maja J. (2013). Automated proxemic feature extraction and behavior recognition: Applications in human-robot interaction. International Journal of Social Robotics, 5(3), 367–378.Google Scholar
Mehrabian, Albert (1968). Some referents and measures of nonverbal behavior. Behavior Research Methods, 1(6), 203–207.Google Scholar
Okwechime, Dumebi, Ong, Eng-Jon, Gilbert, Andrew, & Bowden, Richard (2011). Visualisation and prediction of conversation interest through mined social signals. Pages 951–956 of: Proceedings of the International Conference on Automatic Face and Gesture Recognition (FG).
Park, Sunghyun, Scherer, Stefan, Gratch, Jonathan, Carnevale, Peter, & Morency, Louis-Philippe (2013). Mutual behaviors during dyadic negotiation: Automatic prediction of respondent reactions. In Proceedings of the International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 423–428).
Paxton, Alexandra, & Dale, Rick (2013). Frame-differencing methods for measuring bodily synchrony in conversation. Behavior Research Methods, 45(2), 329–343.Google Scholar
Poppe, Ronald (2007). Vision-based human motion analysis: An overview. Computer Vision and Image Understanding, 108(1–2), 4–18.Google Scholar
Poppe, Ronald (2010). A survey on vision-based human action recognition. Image and Vision Computing, 28(6), 976–990.Google Scholar
Poppe, Ronald, Van Der Zee, Sophie, Heylen, Dirk K. J., & Taylor, Paul J. (2014). AMAB: Automated measurement and analysis of body motion. Behavior ResearchMethods, 46(3), 625–633.Google Scholar
Romera-Paredes, Bernardino, Aung, Hane, Pontil, Massimiliano, et al. (2013). Transfer learning to account for idiosyncrasy in face and body expressions. In Proceedings of the International Conference on Automatic Face and Gesture Recognition (FG) (pp. 1–8).
Rozensky, Ronald H., & Honor, Laurie Feldman (1982). Notation systems for coding nonverbal behavior: A review. Journal of Behavioral Assessment, 4(2), 119–132.Google Scholar
Sacks, Harvey, Schegloff, Emanuel A., & Jefferson, Gail (1974). A simplest systematics for the organisation of turn-taking for conversation. Language, 50(4), 696–735.Google Scholar
Scheflen, Albert E. (1964). The significance of posture in communicational systems. Psychiatry, 27(4), 316–331.Google Scholar
Scherer, Klaus R., & Ekman, Paul (2008). Methodological issues in studying nonverbal behavior. In J. A., Harrigan & R., Rosenthal (Eds), New Handbook of Methods in Nonverbal Behavior Research (pp. 471–504). Oxford: Oxford University Press.
Scherer, Stefan, Stratou, Giota, Mahmoud, Marwa, et al. (2013). Automatic behavior descriptors for psychological disorder analysis. Proceedings of the International Conference on Automatic Face and Gesture Recognition (FG) (pp. 1–8).
Shotton, Jamie, Fitzgibbon, Andrew, Cook, Mat, et al. (2011). Real-time human pose recognition in parts from single depth images. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 1297–1304).
Veenstra, Arno, & Hung, Hayley (2011). Do they like me? Using video cues to predict desires during speed-dates. In Proceedings of the International Conference on Computer Vision (ICCV) Workshops (pp. 838–845).
Velloso, Eduardo, Bulling, Andreas, & Gellersen, Hans (2013). AutoBAP: Automatic coding of body action and posture units from wearable sensors. In Proceedings of the International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 135–140).
Vinciarelli, Alessandro, Pantic, Maja, & Bourlard, Hervé (2009). Social signal processing: Survey of an emerging domain. Image and Vision Computing, 27(12), 1743–1759.Google Scholar
Vondrak, Marek, Sigal, Leonid, & Jenkins, Odest Chadwicke (2013). Dynamical simulation priors for human motion tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(1), 52–65.Google Scholar
von Laban, Rudolf (1975). Laban's Principles of Dance and Movement Notation (2nd edn). London: MacDonald and Evans.
Wallbott, Harald G. (1998). Bodily expression of emotion. European Journal of Social Psychology, 28(6), 879–896.Google Scholar
Cited by

Send book to Kindle

To send this book to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle.

Note you can select to send to either the or variations. ‘’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats

Send book to Dropbox

To send content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about sending content to Dropbox.

Available formats

Send book to Google Drive

To send content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about sending content to Google Drive.

Available formats