Hostname: page-component-77c89778f8-vsgnj Total loading time: 0 Render date: 2024-07-20T07:48:47.090Z Has data issue: false hasContentIssue false

Multimodality in webconference-based language tutoring: An ecological approach integrating eye tracking

Published online by Cambridge University Press:  23 May 2022

Marco Cappellini
Aix-Marseille University, Laboratoire Parole & Langage UMR 7309 CNRS, France (
Yu-Yin Hsu
The Hong Kong Polytechnic University, Hong Kong, China (


Drawing on existing research with a holistic stance toward multimodal meaning-making, this paper takes an analytic approach to integrating eye-tracking data to study the perception and use of multimodality by teachers and learners. To illustrate this approach, we analyse two webconference tutoring sessions from a telecollaborative project involving pre-service teachers and learners of Mandarin Chinese. The tutoring sessions were recorded and transcribed multimodally, and our analysis of two types of conversational side sequences shows that the integration of eye-tracking data into an ecological approach provides richer results. Specifically, our proposed approach provided a window on the participants’ cognitive management of graphic and visual affordances during interaction and uncovered episodes of joint attention.

Research Article
© The Author(s), 2022. Published by Cambridge University Press on behalf of European Association for Computer Assisted Language Learning

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)


Bavelas, J. B., Chovil, N., Coates, L. & Roe, L. (1995) Gestures specialized for dialogue. Personality and Social Psychology Bulletin, 21(4): 394405. CrossRefGoogle Scholar
Bezemer, J. & Kress, G. (2016) Multimodality, learning and communication: A social semiotic frame. London: Routledge. Google Scholar
Blin, F. (2016) The theory of affordances. In Caws, C. & Hamel, M.-J. (eds.), Language-learner computer interactions. Theory, methodology and CALL applications. Amsterdam: John Benjamins, 4164. Google Scholar
Bronfenbrenner, U. (1979) The ecology of human development: Experiments by nature and design. Cambridge MA: Harvard University Press.Google Scholar
Cappellini, M. (2016) Roles and scaffolding in teletandem interactions: A study of the relations between the sociocultural and the language learning dimensions in a French–Chinese teletandem. Innovation in Language Learning and Teaching, 10(1): 620. CrossRefGoogle Scholar
Cappellini, M. (2021) Une approche multimodale intégrant l’oculométrie pour l’étude des interactions télécollaboratives par visioconférence. Les Cahiers de l’ASDIFLE, 31: 99120.Google Scholar
Cappellini, M. & Azaoui, B. (2017) Sequences of normative evaluation in two telecollaboration projects: A comparative study of multimodal feedback through desktop videoconference. Language Learning in Higher Education, 7(1): 5580. CrossRefGoogle Scholar
Cappellini, M. & Hsu, Y.-Y. (2018) Ce que l’oculométrie peut apporter dans une approche écologique des échanges en ligne. Une discussion épistémologique et une étude de cas. In Dejean-Thircuir, C., Mangenot, F., Nissen, E. & Soubrié, T. (eds.), Actes du colloque Échanger Pour Apprendre en Ligne 2018. Google Scholar
Cappellini, M. & Hsu, Y.-Y. (2020) When future teachers meet real learners through telecollaboration: An experiential approach to learn how to teach languages online. Journal of Virtual Exchange, 3: 111. Google Scholar
Cohen, C. (2017) Former à l’enseignement en ligne. In Guichon, N. & Tellier, M. (eds.), Enseigner l’oral en ligne: Une approche multimodale. Paris: Didier, 215242.Google Scholar
Conklin, K., Pellicer-Sánchez, A. & Carrol, G. (2018) Eye-tracking: A guide for applied linguistics research. Cambridge: Cambridge University Press. CrossRefGoogle Scholar
Council of Europe (2001) Common European framework of reference for languages: Learning, teaching, assessment. Strasbourg: Language Policy Unit. Google Scholar
de Pietro, J.-F., Matthey, M. & Py, B. (1989) Acquisition et contrat didactique: les séquences potentiellement acquisitionnelles dans la conversation exolingue. In Weil, D. & Fugier, H. (eds.), Actes du troisième Colloque Régional de Linguistique. Strasbourg: Université des Sciences Humaines et Université Louis Pasteur, 99124.Google Scholar
Develotte, C., Guichon, N. & Vincent, C. (2010) The use of the webcam for teaching a foreign language in a desktop videoconferencing environment. ReCALL, 22(3): 293312. CrossRefGoogle Scholar
Develotte, C., Kern, R. & Lamy, M.-N. (eds.) (2011) Décrire la conversation en ligne: Le face à face distanciel. Lyon: ENS Editions.CrossRefGoogle Scholar
Dooly, M. (2010) The teacher 2.0. In Guth, S. & Helm, F. (eds.), Telecollaboration 2.0: Language, literacies and intercultural learning in the 21st century. Bern: Peter Lang, 277303.Google Scholar
Dooly, M. (2016) ‘Please remove your avatar from my personal space’: Competences of the telecollaboratively efficient person. In O’Dowd, R. & Lewis, T. (eds.), Online intercultural exchange: Policy, pedagogy, practice. Abingdon: Routledge, 192208.Google Scholar
Fuchs, C., Hauck, M. & Müller-Hartmann, A. (2012) Promoting learner autonomy through multiliteracy skills development in cross-institutional exchanges. Language Learning & Technology, 16(3): 82102.Google Scholar
Gibson, J. J. (1979) The ecological approach to visual perception. London: Lawrence Erlbaum Associates.Google Scholar
Gillespie, J. (2020) CALL research: Where are we now? ReCALL, 32(2): 127144. CrossRefGoogle Scholar
Goodwin, C. (2000) Action and embodiment within situated human interaction. Journal of Pragmatics, 32(10): 14891522. CrossRefGoogle Scholar
Guichon, N. (2012) Vers l’intégration des TIC dans l’enseignement des langues. Paris: Didier.Google Scholar
Guichon, N. & Cohen, C. (2016) Multimodality and CALL. In Farr, F. & Murray, L. (eds.), The Routledge handbook of language learning and technology. London: Routledge, 509521.Google Scholar
Guichon, N. & Tellier, M. (eds.) (2017) Enseigner l’oral en ligne: Une approche multimodale. Paris: Didier.Google Scholar
Guichon, N. & Wigham, C. R. (2016) A semiotic perspective on webconferencing-supported language teaching. ReCALL, 28(1): 6282. CrossRefGoogle Scholar
Gullberg, M. & Holmqvist, K. (2006) What speakers do and what addressees look at: Visual attention to gestures in human interaction live and on video. Pragmatics & Cognition, 14(1): 5382. CrossRefGoogle Scholar
Hampel, R. & Stickler, U. (2005) New skills for new classrooms: Training tutors to teach languages online. Computer Assisted Language Learning, 18(4): 311326. CrossRefGoogle Scholar
Hampel, R. & Stickler, U. (2012) The use of videoconference to support multimodal interaction in an online language classroom. ReCALL, 24(2): 116137. CrossRefGoogle Scholar
Hampel, R. & Stickler, U. (eds.) (2015) Developing online language teaching: Research-based pedagogies and reflective practices. New York: Palgrave Macmillan.CrossRefGoogle Scholar
Hauck, M. (2010) Telecollaboration: At the interface between multimodal and intercultural communicative competence. In Guth, S. & Helm, F. (eds.), Telecollaboration 2.0: Language, literacies and intercultural learning in the 21st century. Bern: Peter Lang, 219244.Google Scholar
Helm, F. & Dooly, M. (2017) Challenges in transcribing multimodal data: A case study. Language Learning & Technology, 21(1): 166185.Google Scholar
Holt, B. & Tellier, M. (2017) Conduire des explications lexicales. In Guichon, N. & Tellier, M. (eds.), Enseigner l'oral en ligne. Paris: Didier, 5990.Google Scholar
Hubbard, P. (2017) An invitation to CALL: Foundations of computer-assisted language learning. In Son, J.-B. & Windeatt, S. (eds.), Language teacher education and technology: Approaches and practices. London: Bloomsbury, 153168 Google Scholar
Jefferson, G. (1972) Side sequences. In Sudnow, D. (ed.), Studies in social interaction. New York: The Free Press, 294338.Google Scholar
Kendon, A. (1982) The study of gesture: Some observations on its history. Recherches sémiotiques/Semiotic Inquiry, 2(1): 2562.Google Scholar
Kessler, G. (2016) Technology standards for language teacher preparation. In Farr, F. & Murray, L. (eds.), The Routledge handbook of language learning and technology. Abingdon: Routledge, 5770.Google Scholar
Kozar, O. (2016) Perceptions of webcam use by experienced online teachers and learners: A seeming disconnect between research and practice. Computer Assisted Language Learning, 29(4): 779789. CrossRefGoogle Scholar
Lamy, M.-N. & Hampel, R. (2007) Online communication in language learning and teaching. New York: Palgrave Macmillan. CrossRefGoogle Scholar
Mercier, H. & Sperber, D. (2017) The enigma of reason. London: Penguin.Google Scholar
Mondada, L. (2016) Challenges of multimodality: Language and the body in social interaction. Journal of Sociolinguistics, 20(3): 336366. CrossRefGoogle Scholar
Mondada, L. & Pekarek Doehler, S. (2004) Second language acquisition as situated practice: Task accomplishment in the French second language classroom. The Modern Language Journal, 88(4): 501518. CrossRefGoogle Scholar
O’Dowd, R. (2018) From telecollaboration to virtual exchange: State-of-the-art and the role of UNICollaboration in moving forward. Journal of Virtual Exchange, 1: 123. CrossRefGoogle Scholar
O’Rourke, B. (2012) Using eye-tracking to investigate gaze behavior in synchronous computer-mediated communication for language learning. In Dooly, M. & O’Dowd, R. (eds.), Researching online foreign language interaction and exchange: Theories, methods and challenges. Bern: Peter Lang, 305342.Google Scholar
Pekarek Doehler, S. (ed.) (2000) Approches interactionnistes de l’acquisition des langues étrangères: concepts, recherches, perspectives. Acquisition et Interaction en Langue Étrangère, 12: 119. Google Scholar
Py, B. (2000) La construction interactive de la norme comme pratique et comme représentation. Acquisition et Interaction en Langue Étrangère, 12: 7797. CrossRefGoogle Scholar
Rivens Mompean, A. & Cappellini, M. (2015) Teletandem as a complex learning environment: Looking for a model. DELTA, 31(3): 633663. CrossRefGoogle Scholar
Satar, H. M. (2016) Meaning-making in online language learner interactions via desktop videoconferencing. ReCALL, 28(3): 305325. CrossRefGoogle Scholar
Satar, H. M. & Wigham, C. R. (2017) Multimodal instruction-giving practices in webconferencing-supported language teaching. System, 70: 6380. CrossRefGoogle Scholar
Satar, H. M. & Wigham, C. R. (2020) Delivering task instructions in multimodal synchronous online language teaching. Alsic, 23(1). Google Scholar
Sert, O. & Balaman, U. (2018) Orientations to negotiated language and task rules in online L2 interaction. ReCALL, 30(3): 355374. CrossRefGoogle Scholar
Sinclair, J. M. & Coulthard, R. M. (1975) Towards an analysis of discourse: The English used by teachers and pupils. Oxford: Oxford University Press.Google Scholar
Sloetjes, H. & Wittenburg, P. (2008) Annotation by category - ELAN and ISO DCR. In Calzolari, N., Choukri, K., Maegaard, B., Mariani, J., Odijk, J., Piperidis, S. & Tapias, D. (eds.), Proceedings of the Sixth International Conference on Language Resources and Evaluation. Marrakech: European Language Resources Association, 816820.Google Scholar
Sperber, D. & Wilson, D. (1986) Relevance: Communication and cognition. Oxford: Blackwell Publishing.Google Scholar
Stickler, U., Smith, B. & Shi, L. (2016) Using eye-tracking technology to explore online learner interactions. In Caws, C. & Hamel, M.-J. (eds.), Language-learner computer interactions: Theory, methodology and CALL applications. Amsterdam: John Benjamins, 163186. Google Scholar
van Lier, L. (2004) The ecology and semiotics of language learning: A sociocultural perspective. Dordrecht: Kluwer Academic. CrossRefGoogle Scholar
Wigham, C. R. (2017) A multimodal analysis of lexical explanation sequences in webconferencing-supported language teaching. Language Learning in Higher Education, 7(1): 81108. CrossRefGoogle Scholar
Yamada, M. & Akahori, K. (2010) Awareness and performance through self- and partner’s image in videoconferencing. CALICO Journal, 27(1): 125. CrossRefGoogle Scholar
Supplementary material: File

Cappellini and Hsu supplementary material

Cappellini and Hsu supplementary material

Download Cappellini and Hsu supplementary material(File)
File 26.6 KB