Hostname: page-component-848d4c4894-2pzkn Total loading time: 0 Render date: 2024-05-05T17:55:51.304Z Has data issue: false hasContentIssue false
  • English
  • Français

Why subtitle speed matters: Evidence from word skipping and rereading

Published online by Cambridge University Press:  03 December 2021

Jan-Louis Kruger Open the ORCID record for Jan-Louis Kruger [Opens in a new window] ,
Natalia Wisniewska  and
Sixin Liao Open the ORCID record for Sixin Liao [Opens in a new window]
Jan-Louis Kruger*
Affiliation:
Department of Linguistics, Macquarie University, Sydney, Australia Macquarie University Centre for Reading, Sydney, Australia UPSET Focus area, North-West University, Vanderbijlpark, South Africa
Natalia Wisniewska
Affiliation:
Department of Linguistics, Macquarie University, Sydney, Australia Macquarie University Centre for Reading, Sydney, Australia
Sixin Liao
Affiliation:
Department of Linguistics, Macquarie University, Sydney, Australia Macquarie University Centre for Reading, Sydney, Australia
*
*Corresponding author. Email: janlouis.kruger@mq.ed.au
Get access Rights & Permissions [Opens in a new window]

Abstract

High subtitle speed undoubtedly impacts the viewer experience. However, little is known about how fast subtitles might impact the reading of individual words. This article presents new findings on the effect of subtitle speed on viewers’ reading behavior using word-based eye-tracking measures with specific attention to word skipping and rereading. In multimodal reading situations such as reading subtitles in video, rereading allows people to correct for oculomotor error or comprehension failure during linguistic processing or integrate words with elements of the image to build a situation model of the video. However, the opportunity to reread words, to read the majority of the words in the subtitle and to read subtitles to completion, is likely to be compromised when subtitles are too fast. Participants watched videos with subtitles at 12, 20, and 28 characters per second (cps) while their eye movements were recorded. It was found that comprehension declined as speed increased. Eye movement records also showed that faster subtitles resulted in more incomplete reading of subtitles. Furthermore, increased speed also caused fewer words to be reread following both horizontal eye movements (likely resulting in reduced lexical processing) and vertical eye movements (which would likely reduce higher-level comprehension and integration).

Keywords

eye movementslinguistic processingrereadingsubtitle readingsubtitle speedword skipping
Type
Original Article
Information
Applied Psycholinguistics , Volume 43 , Issue 1 , January 2022 , pp. 211 - 236
Check for updates
Copyright
© The Author(s), 2021. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Angele, B., Laishley, A. E., Rayner, K., & Liversedge, S. P. (2014). The effect of high-and low-frequency previews and sentential fit on word skipping during reading. Journal of Experimental Psychology: Learning, Memory, and Cognition, 40(4), 1181–1203.Google ScholarPubMed
Bates, D., Kliegl, R., Vasishth, S., & Baayen, H. (2015). Parsimonious mixed models. arXiv preprint arXiv:1506.04967.Google Scholar
Cook, A. E., & Wei, W. (2019). What can eye movements tell us about higher level comprehension? Vision, 3(3), 45–61.CrossRefGoogle ScholarPubMed
d’Ydewalle, G., & De Bruycker, W. (2007). Eye movements of children and adults while reading television subtitles. European Psychologist, 12, 196–205.CrossRefGoogle Scholar
d’Ydewalle, G., & Gielen, I. (1992). Attention allocation with overlapping sound, image, and text. In Rayner, K., (Eds.), Eye movements and visual cognition: Scene perception and reading (pp. 415–427). New York: Springer-Verlag.CrossRefGoogle Scholar
d’Ydewalle, G., Muylle, P., & van Rensbergen, J. (1985). Attention shifts in partially redundant information situations. In Groner, R., McConkie, G. W. & Menz, C., (Eds.), Eye Movements and Human Information Processing (pp. 375–384). Amsterdam: Elsevier.Google Scholar
d’Ydewalle, G., van Rensbergen, J. & Pollet, J. (1987). Reading a message when the same message is available auditorily in another language: The case of subtitling. In O’Regan, J. K. & Levy-Schoen, A. (Eds.) Eye movements: from physiology to cognition (pp. 313–321). Amsterdam/New York: Elsevier.CrossRefGoogle Scholar
Drieghe, D. (2008). Foveal processing and word skipping during reading. Psychonomic Bulletin & Review, 15(4), 856–860.CrossRefGoogle ScholarPubMed
Drieghe, D., Rayner, K., & Pollatsek, A. (2005). Eye movements and word skipping during reading revisited. Journal of Experimental Psychology: Human Perception and Performance, 31(5), 954–969.Google ScholarPubMed
Eskenazi, M. A., & Folk, J. R. (2017). Regressions during reading: The cost depends on the cause. Psychonomic Bulletin & Review, 24(4), 1211–1216.CrossRefGoogle ScholarPubMed
Fothergill, A. (Director). (2006). Planet Earth. Burbank, CA: BBC Video.Google Scholar
Fresno, N. & Sepielak, K. (2020). Subtitling speed in media accessibility research: some methodological considerations. Perspectives. Advanced online publication. DOI: 10.1080/0907676X.2020.1761841.CrossRefGoogle Scholar
Graesser, A. C., McNamara, D. S., Cai, Z., Conley, M., Li, H., & Pennebaker, J. (2014). Coh-metrix measures text characteristics at multiple levels of language and discourse. The Elementary School Journal, 115, 210–228.CrossRefGoogle Scholar
Hyönä, J., Lorch, R. F. Jr., & Kaakinen, J. K. (2002). Individual differences in reading to summarize expository text: Evidence from eye fixation patterns. Journal of Educational Psychology, 94, 44–55.CrossRefGoogle Scholar
Hyönä, J., Lorch, R. F. Jr., & Rinck, M. (2003). Eye movement measures to study global text processing. In Hyönä, J., Radach, R. & Deubel, H. (Eds.), The mind’s eye: Cognitive and applied aspects of eye movement research (pp. 313–334). Amsterdam: Elsevier Science.CrossRefGoogle Scholar
Inhoff, A. W., Kim, A., & Radach, R. (2019). Regressions during reading. Vision, 3(3), 35–49.CrossRefGoogle ScholarPubMed
Jensema, C. J., McCann, R., & Ramsey, S. (1996). Closed-captioned television presentation speed and vocabulary. American Annals of the Deaf, 141(4), 284–292.CrossRefGoogle ScholarPubMed
Jensen, M. S., Yao, R., Street, W. N., & Simons, D. J. (2011). Change blindness and inattentional blindness. Wiley Interdisciplinary Reviews: Cognitive Science, 2(5), 529–546.Google ScholarPubMed
Koolstra, C. M., Van Der Voort, T. H. A., & d’Ydewalle, G. (1999). Lengthening the Presentation Time of Subtitles on Television: Effects on Children’s Reading Time and Recognition. Communications, 24(4), 407–422.CrossRefGoogle Scholar
Krejtz, I., Szarkowska, A., & Krejtz, K. (2013). The effects of shot changes on eye movements in subtitling. Journal of Eye Movement Research, 6(5), 1–12.CrossRefGoogle Scholar
Kruger, J. L., & Steyn, F. (2014). Subtitles and eye tracking: Reading and performance. Reading Research Quarterly, 49, 105–120.CrossRefGoogle Scholar
Laeng, B., & Teodorescu, D. S. (2002). Eye scanpaths during visual imagery reenact those of perception of the same visual scene. Cognitive Science. 26, 207–231.CrossRefGoogle Scholar
Liao, S., Kruger, J. L., & Doherty, S. (2020). The impact of monolingual and bilingual subtitles on visual attention, cognitive load, and comprehension. The Journal of Specialised Translation, 33, 70–89.Google Scholar
Liao, S., Yu, L., Reichel, E. D., & Kruger, J. L. (2021). Using eye movements to study the reading of subtitles in video. Scientific Studies of Reading, 25, 417–435.CrossRefGoogle Scholar
Metzner, P., von der Malsburg, T., Vasishth, S., & Rösler, F. (2017). The importance of reading naturally: Evidence from combined recordings of eye movements and electric brain potentials. Cognitive Science, 41, 1232–1263.CrossRefGoogle ScholarPubMed
Netflix. (2020). English timed text style guide. Retrieved 15 August, 2020, from https://partnerhelp.netflixstudios.com/hc/en-us/articles/217350977 Google Scholar
Perego, E., Del Missier, F., Porta, M. & Mosconi, M. 2010. The Cognitive Effectiveness of Subtitle Processing. Media Psychology, 13(3), 243–272.CrossRefGoogle Scholar
Rayner, K. & Duffy, S. A. (1986). Lexical complexity and fixation times in reading: Effects of word frequency, verb complexity, and lexical ambiguity. Memory & Cognition, 14(3), 191–201.CrossRefGoogle ScholarPubMed
Rayner, K. & Morrison, R. M. (1981). Eye movements and identifying words in parafoveal vision. Bulletin of the Psychonomic Society, 17, 135–138.CrossRefGoogle Scholar
Rayner, K. & Sereno, S. C. (1994). Eye movements in reading. Psycholinguistic studies. In Gernsbacher, M. A., (Ed.) Handbook of psycholinguistics (pp. 57–81). San Diego, CA: Academic Press.Google Scholar
Rayner, K. (1978). Eye movements in reading and information processing. Psychological Bulletin, 85(3), 618–660.CrossRefGoogle ScholarPubMed
Rayner, K. (1986). Eye movements and the perceptual span in beginning and skilled readers. Journal of Experimental Child Psychology, 41(2), 211–236.CrossRefGoogle ScholarPubMed
Rayner, K. (1995). Eye movements and cognitive processes in reading, visual search, and scene perception. Studies in Visual Information Processing, 6, 3–22.CrossRefGoogle Scholar
Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124, 372–422.CrossRefGoogle ScholarPubMed
Rayner, K. (2009). Eye movements in reading: Models and data. Journal of eye movement research, 2(5), 1–10.CrossRefGoogle ScholarPubMed
Rayner, K., Ashby, J., Pollatsek, A., & Reichle, E. D. (2004). The effects of frequency and predictability on eye fixations in reading: Implications for the E-Z Reader model. Journal of Experimental Psychology: Human Perception and Performance, 30, 720–732.Google ScholarPubMed
Rayner, K., Pollatsek, A., Ashby, J., & Clifton, C. Jr. (2012). Psychology of reading, 2nd Ed. New York: Psychology Press.CrossRefGoogle Scholar
Rayner, K., Sereno, S. C., Morris, R. K., Schmauder, A. R., & Clifton, C. (1989). Eye movements and on-line language comprehension processes. Language and Cognitive Processes, 4, 21–50.CrossRefGoogle Scholar
Rayner, K., Slattery, T. J., Drieghe, D., & Liversedge, S. P. (2011). Eye movements and word skipping during reading: effects of word length and predictability. Journal of Experimental Psychology: Human Perception and Performance, 37(2), 514–528.Google ScholarPubMed
Rayner, K., Smith, T. J., Malcolm, G. L., & Henderson, J. M. (2009). Eye movements and visual encoding during scene perception. Psychological Science, 20(1), 6–10.CrossRefGoogle ScholarPubMed
Reichle, E. D. (2021). Computational models of reading: A handbook. Oxford: Oxford University Press.CrossRefGoogle Scholar
Reichle, E. D., Pollatsek, A., Fisher, D. L., & Rayner, K. (1998). Toward a model of eye-movement control in reading. Psychological Review, 105, 125–157.CrossRefGoogle Scholar
Reichle, E. D., Rayner, K., & Pollatsek, A. (2003). The EZ Reader model of eye-movement control in reading: Comparisons to other models. Behavioral and Brain Sciences, 26(4), 445–526.CrossRefGoogle Scholar
Romero-Fresco, P. (2009). More haste less speed: Edited versus verbatim respoken subtitles. Vigo International Journal of Applied Linguistics, 6, 109–133.Google Scholar
Romero-Fresco, P. (2019). Accessible filmmaking: Integrating translation and accessibility into the filmmaking process. London: Routledge.CrossRefGoogle Scholar
Schotter, E. R., Tran, R., & Rayner, K. (2014). Don’t believe what you read (only once) comprehension is supported by regressions during reading. Psychological Science, 25(6), 1218–1226.CrossRefGoogle ScholarPubMed
Szarkowska, A. (2016). Report on the results of an online survey on subtitle presentation times and line breaks in interlingual subtitling. Part 1: Subtitlers. London: University College London.Google Scholar
Szarkowska, A., & Bogucka, L. (2019). Six-second rule revisited: An eye-tracking study on the impact of speech rate and language proficiency on subtitle reading. Translation, Cognition & Behavior, 2(1), 101–124.CrossRefGoogle Scholar
Szarkowska, A., & Gerber-Morón, O. (2018). Viewers can keep up with fast subtitles: Evidence from eye movements. Plos One, 13(6).CrossRefGoogle ScholarPubMed
Szarkowska, A., Krejtz, I., Pilipczuk, O., Dutka, Ł., & Kruger, J.L. (2016). The effects of text editing and subtitle presentation rate on the comprehension and reading patterns of interlingual and intralingual subtitles among deaf, hard of hearing and hearing viewers. Across Languages and Cultures, 17(2), 183–204.CrossRefGoogle Scholar
Tyler, M. D., Jones, C., Grebennikov, L., Leigh, G., Noble, W., & Burnham, D. (2009). Effect of caption rate on the comprehension of educational television programmes by deaf school students. Deafness & Education International, 11(3), 152–162.CrossRefGoogle Scholar
Veldre, A., & Andrews, S. (2017). Parafoveal preview benefit in sentence reading: Independent effects of plausibility and orthographic relatedness. Psychonomic Bulletin & Review, 24(2), 519–28.CrossRefGoogle ScholarPubMed
Warren, T., White, S. J., & Reichle, E. D. (2009). Investigating the causes of wrap-up effects: Evidence from eye movements and E–Z Reader. Cognition, 111(1), 132–137.CrossRefGoogle ScholarPubMed
White, S. J., Drieghe, D., Liversedge, S. P., & Staub, A. (2018). The word frequency effect during sentence reading: A linear or nonlinear effect of log frequency? Quarterly Journal of Experimental Psychology, 71(1), 46–55.CrossRefGoogle Scholar
Winke, P., Gass, S., & Sydorenko, T. (2013). Factors influencing the use of captions by foreign language learners: An eye-tracking study, The Modern Language Journal, 97(1), 254–275.CrossRefGoogle Scholar
ZDF. (2015). Untertitel-Standards von ARD, ORF, SRF, ZDF. Retrieved on 19 August, 2020, from https://www.zdf.de/barrierefreiheit-im-zdf/untertitel-standards-von-ard-orf-srf-und-zdf-100.html Google Scholar