Skip to main content Accessibility help
  • Get access
    Check if you have access via personal or institutional login
  • Cited by 1
  • Print publication year: 2011
  • Online publication date: June 2012

10 - Embodiment and expressive communication on the internet


Overview: Human brains are basically social, and use communication mechanisms that have evolved during our evolutionary past. Thus, we suggest that even in communication with and by machines, humans will tend to react socially and use communication mechanisms that are primarily social and embodied. One of these mechanisms is communicative feedback, which refers to unobtrusive (usually short) vocal or bodily expressions, whereby a recipient of information can inform a contributor of information about whether he or she is able and willing to communicate, perceive the information, and understand the information. We will show how feedback can be modeled in virtual agents on facial expressions of a virtual agent or verbot and thus contribute to human–human communication over the internet. We will present a simple model based on a pleasure, arousal, and dominance space, which allows a complex stimulus generation program to be driven with only a few parameters.

Humans are social – but what about human–machine communication?

Internet communication consists of two major domains: communication with a machine and human–human communication through a machine. Both processes involve different but comparable elements in order to be efficient, as we will outline here.

In its early years, the internet was used by a rather small group of scientists for communication via email and bulletin boards. As compared to phone calls and direct face-to-face communication, it seemed to be missing a social component, thus leading to the introduction of emoticons such as the well-known smiley, which constituted a first attempt to fill this gap.

Allwood, J., Nivre, J., and Ahlsén, E. (1992). On the semantics and pragmatics of linguistic feedback. Journal of Semantics, 9(1), 1–26.
Allwood, J., Kopp, S., Grammer, K., Ahlsén, E., Oberzaucher, E., and Koppensteiner, M. (2008). The analysis of embodied communicative feedback in multimodal corpora – a prerequisite for behavior simulation. Journal of Language Resources and Evaluation (Special Issue on Multimodal Corpora), 41(3–4), 255–272.
Becker, C., Kopp, S., and Wachsmuth, I. (2004). Simulating the emotion dynamics of a multimodal conversational agent. In André, E. (ed.), Proceedings of Affective Dialogue Systems Conference (ADS '04), LNAI 3068 (pp. 154–165). Berlin: Springer.
Beun, R. J. and Eijk, R. M. (2004). A cooperative dialogue game for resolving ontological discrepancies. In Dignum, F., , F. (ed.), Advances in Agent Communication (pp. 349–363). Berlin: Springer.
Bulbulia, J. (2004). Religious costs as adaptations that signal altruistic intention. Evolution and Cognition, 10(1), 19–42.
Cassell, J. (2004). Towards a model of technology and literacy development: story listening systems. Journal of Applied Developmental Psychology, 25(1), 75–105.
Cassell, J. and Thórisson, K. R. (1999). The power of a nod and a glance: envelope vs. emotional feedback in animated conversational agents. Applied Artificial Intelligence, 13, 519–538.
Cassell, J., Bickmore, T., Billinghurst, M., Campbell, L., Chang, K., Vilhjálmsson, H., and Yan, H. (1999). Embodiment in conversational interfaces: Rea. In CHI '99 Conference Proceedings (pp. 520–527). New York: ACM Press.
Cassell, J., Vilhjálmsson, H., and Bickmore, T. (2001). BEAT: the Behavior Expression Animation Toolkit. In Proceedings of SIGGRAPH '01 (pp. 477–486). New York: ACM Press.
Cleveland, W. S. (1979). Robust locally weighted regression and smoothing scatterplots. Journal of the American Statistical Association, 74, 829–836.
Cleveland, W. S. and Devlin, S. J. (1988). Locally weighted regression: an approach to regression analysis by local fitting. Journal of the American Statistical Association, 83, 596–610.
Cosmides, L. and Tooby, J. (1992). Cognitive adaptations for social exchange. In Barkow, J., Cosmides, L., and Tooby, J. (eds), The Adapted Mind: Evolutionary Psychology and the Generation of Culture (pp. 163–228). New York: Oxford University Press.
Cosmides, L., Tooby, J., and Barkow, J. H. (1992). Evolutionary Psychology and Conceptual Integration. Oxford University Press.
Coulson, M. (2004). Attributing emotion to static body postures: recognition accuracy, confusions, and viewpoint dependence. Journal of Nonverbal Behavior, 28, 117–139.
Darwin, C. (1872). The Expression of the Emotions in Man and Animals. New York: D. Appleton and Company.
Dawkins, R. and Krebs, J. R. (1978). Animal signals: information or manipulation? In Krebs, J. R. and Davies, N. B. (eds), Behavioural Ecology: An Evolutionary Approach (pp. 282–309). Sunderland, MA: Sinauer.
Rosis, F., Pelachaud, C., Poggi, I., Carofiglio, V., and Carolis, B. (2003). From Greta's mind to her face: modeling the dynamics of affective states in a conversational embodied agent. International Journal of Human–Computer Studies, 59, 81–118.
Ehlich, K. (1986). Interjektionen. Tübingen: Niemeyer.
Eibl-Eibesfeldt, I. and Hass, H. (1967). Neue Wege der Humanethologie[New Ways in Human Ethology], Homo, 18, 13–23.
Ekman, P. (1969). The repertoire of nonverbal behavior – categories, origins, usage and coding. Semiotica, 1, 49–98.
Ekman, P. (1971). Universals and cultural differences in facial expressions of emotion. Nebraska Symposium on Motivation, 19, 207–283.
Ekman, P. (1980). Biological and cultural contributions to body and facial movement in the expression of emotions. In Rorty, A. O. (ed.), Explaining Emotions (pp. 73–101). Berkeley, CA: University of California Press.
Ekman, P. (1984). Expression and the nature of emotion. In Scherer, K. and Ekman, P. (eds), Approaches to Emotion (pp. 319–343). Hillsdale, NJ: Lawrence Erlbaum.
Ekman, P. (1994). Strong evidence for universals in facial expressions: a reply to Russell's mistaken critique. Psychological Bulletin, 115, 268–287.
Ekman, P. and Friesen, W. V. (1978). Investigator's Guide: Facial Action Coding System. Palo Alto, CA: Consulting Psychologists Press.
Fahlmann, S. E. (2007). Smiley lore :-).
Fridlund, A. J. (1991). Sociality and solitary smiling: potentiation by an implicit audience. Journal of Personality and Social Psychology, 60, 229–240.
Fridlund, A. J. (1994). Human facial expression: An Evolutionary View. San Diego, CA: Academic Press.
Fried, I. (2007). Warning sounded over ‘flirting robots.’
Frijda, N. H. (1986). The Emotions. Cambridge University Press.
Fujie, S., Fukushima, K., and Kobayashi, T. (2004). A conversation robot with back-channel feedback function based on linguistic and nonlinguistic information. In Proceedings of the International Conference on Autonomous Robots and Agents (pp. 379–384).
Gallese, V. and Goldmann, A. (1998). Mirror neurons and the simulation theory of mind-reading. Trends in Cognitive Science, 2, 493–501.
Gillenson, M. L. (1974). The Interactive Generation of Facial Images on a CRT Using a Heuristic Strategy. Ohio State University, Computer Graphics Research Group.
Grammer, K., Schiefenhovel, W., Schleidt, M., Lorenz, B., and Eibl-Eibesfeldt, I. (1988). Patterns on the face: the eyebrow flash in crosscultural comparison. Ethology, 77, 279–299.
Grammer, K. and Fieder, M. (1997). A neural network approach for the classification of body movements (Abstract). In Schmitt, A., Atzwanger, K., Grammer, K., and Schäfer, K. (eds), New Aspects of Human Ethology (pp. 202–203). New York: Plenum Press.
Grammer, K., Filova, V., and Fieder, M. (1997). The communication paradox and possible solutions: towards a radical empiricism. In Schmitt, A., Atzwanger, K., Grammer, K., and Schäfer, K. (eds), New Aspects of Human Ethology (pp. 91–120). New York: Plenum Press.
Grammer, K., Fink, B., and Renninger, L. (2002). Dynamic systems and inferential information processing in human communication. Neuro Endocrinological Letters (Special Issue), 23(4), 15–22.
Grammer, K., Fink, B., Møller, A. P., and Thornhill, R. (2003). Darwinian aesthetics: sexual selection and the biology of beauty. Biological Reviews, 78(3), 385–340.
Grammer, K., Fink, B., Oberzaucher, E., Atzmueller, M., Blantar, I., and Mitteroecker, P. (2004). The representation of self-reported affect in body posture and body posture simulation. Collegium Anthropologicum, 28(2), 159–173.
Grammer, K. and Oberzaucher, E. (2006). The reconstruction of facial expressions in embodied systems: new approaches to an old problem. ZIF Mitteilungen, 2, 14–31.
Griffiths, P. E. (1990). Modularity and the psychoevolutionary theory of emotion. Biology & Philosophy, 5, 175–196.
Guthrie, S. (1993). Faces in the Clouds: A New Theory of Religion. Oxford University Press.
Hess, U., Adams, R. S., and Kleck, R. E. (2004). Facial appearance, gender, and emotion expression. Emotion, 4, 378–388.
Houck, N. and Gass, S. M. (1997). Cross-cultural back channels in English refusals: a source of trouble. In Jaworski, A. (ed.), Silence – Interdisciplinary Perspectives (pp. 285–308). Berlin: Mouton de Gruyter.
Izard, C. E. (1971). The Face of Emotion. New York: Appleton-Century-Crofts.
Izard, C. E. (1977). Human Emotions. New York: Plenum.
Izard, C. E. (1991). The Psychology of Emotions. New York: Plenum.
James, W. (1950 [1890]). The Principles of Psychology. New York: Dover.
Johansson, G. (1973). Visual perception of biological motion and a model of its analysis. Perception & Psychophysics, 14, 201–211.
Johansson, G. (1976). Spatio-temporal differentiation and integration in visual motion perception. Psychological Research, 38, 379–398.
Kipp, M. (2001). Anvil – a generic annotation tool for multimodal dialogue. In Proceedings of Eurospeech 2001, Aalborg (pp. 1367–1370).
Kleinginna, P. R. and Kleinginna, A. M. (1981). A categorized list of emotion definitions with suggestions for a consensual definition. Motivation and Emotion, 5, 345–379.
Kopp, S., Gesellensetter, L., Krämer, N.C., and Wachsmuth, I. (2005). A conversational agent as a museum guide. Design and evaluation of a real-world application. In Panayiotopoulos, al. (eds), Intelligent Virtual Agents 2005 (pp. 329–343). Hamburg: Springer.
Kopp, S., Allwood, J., Ahlsén, E., Grammer, K., and Stocksmeier, T. (2008). Modeling embodied feedback with virtual humans. In Wachsmuth, I. and Knoblich, G. (eds), Modeling Communication with Robots and Virtual Humans (pp. 18–37). Berlin: Springer.
Krämer, N. C. (2001). Bewegende Bewegung. Sozio-emotionale Wirkungen nonverbalen Verhaltens und deren experimentelle Untersuchung mittels Computeranimation. [Moving Movements. Socio-emotional Effects of Nonverbal Behavior and Their Experimental Analysis by Means of Computer Animation] Lengerich: Pabst Science Publishers.
Krebs, J. R. and Dawkins, R. (1984). Animal signals: mind reading and manipulation. In Krebs, J. R. and Davies, N. B. (eds), Behavioural Ecology: An Evolutionary Approach (pp. 380–402). Oxford: Blackwell Scientific.
Krumhuber, E. and Kappas, A. (2005). Moving smiles: the role of dynamic components for the perception of the genuineness of smiles. Journal of Nonverbal Behavior, 29, 3–24.
Lang, P. J. (1995). The emotion probe. American Psychologist, 50, 372–385.
Lazarus, R. S. (1991). Emotion and Adaption. New York: Oxford University Press.
Lee, E., Kang, J. I., Park, I. H., Kim, J.-J., and An, S. K. (2008). Is a neutral face really evaluated as being emotionally neutral?Psychiatry Research, 157, 77–85.
Mehrabian, A. and Russell, J. A. (1974). An Approach to Environmental Psychology. Cambridge, MA: MIT Press.
,Microsoft (2003). Microsoft Agent product information. (
Moser, E., Derntl, B., Robinson, S., Fink, B., Gur, R. C., and Grammer, K. (2006). Amygdala activation at 3T in response to human and avatar facial expressions of emotions. Journal of Neuroscience Methods, 161(1), 126–133.
Ortony, A. and Turner, T. J. (1990). What's basic about basic emotions?Psychological Review, 97, 315–331.
Osgood, C. E. (1966). Dimensionality of the semantic space for communication via facial expressions. Scandinavian Journal of Psychology, 7, 1–30.
Panksepp, J. (1992). A critical role for “affective neuroscience” in resolving what is basic about basic emotions. Psychological Review, 99(3), 554–560.
Parke, F. I. (1972). Computer-generated animation of faces. Proceedings of ACM National Conference (vol. I, pp. 451–457).
Petta, P., Staller, A., Trappl, R., Mantler, S., Szalavari, Z., Psik, T., and Gervautz, M. (1999). Towards engaging full-body interaction. In Bullinger, H.-J. and Vossen, P. H. (eds), Adjunct Conference Proceedings, HCI International '99, 8th International Conference on Human–Computer Interaction (pp. 280–281). Stuttgart: Fraunhofer IRB Verlag.
Platt, S. M. and Badler, N. (1981). Animating facial expression. Computer Graphics, 15(3), 245–252.
Poggi, I., Pelachaud, C., Rosis, F., Carofiglio, V., and Carolis, B. (2005). GRETA. A believable embodied conversational agent. In Stock, O. and Zancarano, M. (eds), Multimodal Intelligent Information Presentation. (pp. 3–25). Dordrecht: Kluwer.
Reeves, B. and Nass, C. (1996). The Media Equation. How People Treat Computers, Television, and New Media like Real People and Places. New York: Cambridge University Press.
Russell, J. A. (1978). Evidence of convergent validity on the dimensions of affect. Journal of Personality and Social Psychology, 36, 1152–1168.
Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39, 1161–1178.
Russell, J. A. (1991). In defense of a prototype approach to emotion concepts. Journal of Personality and Social Psychology, 60, 425–438.
Russell, J. A. (1995). Facial expression of emotion: what lies beyond minimal universality?Psychological Bulletin, 118, 379–391.
Russell, J. A. and Mehrabian, A. (1977). Evidence for a three-factor theory of emotions. Journal of Research in Personality, 11, 273–294.
Scherer, K. R. (1984). On the nature and function of emotion: a component process approach. In Scherer, K. R. and Ekman, P. (eds), Approaches to Emotion (pp. 293–317). Hillsdale, NJ: Erlbaum.
Scherer, K. R. (1994). Affect bursts. In Goozen, S., Poll, N. E., and Sergeant, J. A. (eds), Emotions: Essays on Emotion Theory (pp. 161–193). Hillsdale, NJ: Lawrence Erlbaum.
Scherer, K. R. (1997). Profiles of emotion-antecedent appraisal: testing theoretical predictions across cultures. Cognition and Emotion, 11, 113–150.
Scherer, K. R. (1999). Appraisal theory. In Dalgleish, T. and Power, M. J. (eds), Handbook of Emotion and Cognition (pp. 637–663). New York: Wiley.
Scherer, K. R. (2001). Appraisal considered as a process of multi-level sequential checking. In Scherer, R., Schorr, A., and Johnstone, T. (eds), Appraisal Processes in Emotion: Theory, Methods, Research (pp. 92–120). New York: Oxford University Press.
Schmidt, K. L. and Cohen, J. F. (2002). Human facial expressions as adaptations: evolutionary questions in facial expression research. Yearbook of Physical Anthropology, 44, 3–24.
Shneiderman, B. (1989). Social and individual impact. Educational Media International, 26(2), 101–106.
Smith, C. A. and Scott, H. S. (1997). Spontaneous facial behavior during intense emotional episodes: artistic truth and optical truth. In Russell, J. A. and Fernández-Dols, J. M. (eds), The Psychology of Facial Expression (pp. 229–254). Cambridge University Press.
Snodgrass, J. (1992). Judgment of Feeling States from Facial Behavior: A Bottom-up Approach. Unpublished doctoral dissertation, University of British Columbia.
Spencer-Smith, J., Wild, H., Innes-Ker, A. H., Townsend, J. T., Duffy, C., Edwards, C., Ervin, K., Merritt, N., and Paik, J. W. (2001). Making faces: creating three-dimensional parameterized models of facial expression. Behavior Research Methods, Instruments and Computers, 33, 115–123.
Staller, A. and Petta, P. (1998). Towards a tractable appraisal-based architecture for situated cognizers. In Canamero, D., Numaoka, C., and Petta, P. (eds), Grounding Emotions in Adaptive Systems, Workshop Notes, 5th International Conference of the Society for Adaptive Behavior (pp. 56–61). Zurich: Society for Adaptive Behavior.
Takeuchi, M., Kitaoka, N., and Nakagawa, S. (2004). Timing detection for realtime dialog systems using prosodic and linguistic information. In Proceedings of the International Conference on Speech Prosody (pp. 529–532).
Thórisson, K. R. (1996). Communicative Humanoids – A Computational Model of Psychological Dialogue Skills. PhD thesis, School of Architecture and Planning, Massachusetts Institute of Technology.
Tomkins, S. S. (1962). Affect Imagery Consciousness: Vol. I. The Positive Affects. New York: Springer.
Ward, N. and Tsukahara, W. (2000). Prosodic features which cue back-channel responses in English and Japanese. Journal of Pragmatics, 32(8), 1177–1207.
Watson, D., Clark, L. A., and Tellegen, A. (1988). Development and validation of brief measures of positive and negative affect: the PANAS scales. Journal of Personality and Social Psychology, 54(6), 1063–1070.
Wehrle, T. and Scherer, K. R. (2001). Towards computational modeling of appraisal theories. In Scherer, K. R., Schorr, A., and Johnstone, T. (eds), Appraisal Processes in Emotion: Theory, Methods, Research (pp. 350–365). New York: Oxford University Press.
Wiener, N. (1984). Cybernetics and Control and Communication in the Animal and the Machine. Cambridge, MA: MIT Press.
Woodworth, R. S. (1938). Experimental Psychology. New York: Holt.
Wundt, W. (1924 [1912]). An Introduction to Psychology (Pinter, R., Trans.). London: Allen and Unwin.
Xiao, J. (2001). Understanding the Use and Utility of Anthropomorphic Interface Agents. Student Poster at the CHI 2001 in Seattle, WA, USA.
Yngve, V. H. (1970). On getting a word in edgewise. In Papers from the Sixth Regional Meeting of the Chicago Linguistics Society (pp. 567–578). Chicago: Chicago Linguistics Society.