1. Introduction
Multimodal composing, which has sometimes been referred to synonymously as multimodal composition or multimodal writing, is the use of different semiotic resources (e.g., audio, visual, gestural, and/or spatial resources) in addition to linguistic text for making meaning. Notably, multimodal composing is neither a new type of writing nor a new area of research, with studies dating back to the early 2000s. In the domain of second language (L2) research, Tardy's (2005Footnote *) study on multimodal composition in academia was one of the earliest to bring attention to the nonlinguistic features of L2 written output. Even after this pioneering study, in the few years that followed, only a handful of studies further explored aspects of L2 learners’ multimodal compositions. However, over the past decade, the fields of applied linguistics and second language acquisition (SLA) have witnessed an explosion of interest in both its study and classroom applications, with teachers’ adoption of multiple modes becoming an indispensable part of their pedagogical toolkits (e.g., Kessler, Reference Kessler2022; Li, Reference Li2021; Zhang et al., Reference Zhang, Akoto and Li2021).
Notably, the delayed attention to multimodal composing is likely attributable to dominant beliefs surrounding language as the chief semiotic resource in communication, especially within the field of SLA, as some scholars have argued that linguistic forms should be the primary focus of instruction (e.g., Manchón, Reference Manchón2017; Qu, Reference Qu2017). This view of non-linguistic modes as being secondary and of diminished importance is referred to as the weak version of multimodality. In contrast to this weak version, most of the multimodal composing studies to date have tended to advocate for a strong version of multimodality (Grapin, 2019*), positing that both linguistic and nonlinguistic resources are (and should be) of comparable importance. Many of such studies have been situated in or originated from first language (L1) composition research and bilingual education, where the learning goals are typically to succeed in content learning and to mitigate language difficulties with assistance from nonlinguistic resources (e.g., Pacheco & Smith, 2015*; Smith et al., 2017*). However, scholars within SLA have found multimodal composing (or digital multimodal composing when occuring with digital tools and software, see Kessler & Marino, Reference Kessler and Marino2022) to be of increasing relevance, primarily owing to the growing body of literature that has demonstrated its capacity to positively influence various aspects of the L2 learning process.
2. The relevance of multimodal composing to SLA
Before introducing the research timeline, we clarify why multimodal composing matters in the context of instructed SLA. In particular, recent intervention studies have identified numerous positive effects of multimodal composing on L2 learning (e.g., Vandommele et al., 2017*; Xu, Reference Xu2021), and studies have also shown comparable and/or favorable results for learners’ multimodal texts when compared with the production of more traditional, monomodal texts (e.g., Cho & Kim, 2021*; Kim & Belcher, 2020*). Additionally, studies have revealed students’ preferences for multimodal composing when compared with traditional writing in terms of motivation and increasing the effectiveness of communication (e.g., Dzeoke, 2017*). While there is still considerable room for new and innovative research in the future, this developing body of empirical research has addressed many of SLA researchers’ original concerns about the utility of multimodal composing for L2 learning (see Lim & Kessler, Reference Lim, Kessler, Manchón and Polio2022 for a related research agenda).
Another important note regarding the role of multimodal composing in language teaching and SLA pertains to the changing nature of communication. That is, the target of language instruction has changed as the view of language, or competence, is dynamic and evolving. After a long period of focusing on linguistic elements only, the current target for many instructors is to develop communicative competence, which is a multifaceted construct consisting of different competencies (Polio & Montgomery, Reference Polio, Montgomery, Kanwit and Solon2022). Similarly, literacy, which has traditionally been thought of as consisting of reading and writing, has been reconceptualized by some as multiliteracies (e.g., New London Group, 1996) – that is, the capacity to comprehend and convey information via different modes – including those that are linguistic, visual, audio, gestural, and spatial in nature. Researchers have discussed how and why these multiliteracies need to be integrated into the classroom mainly on theoretical reasoning (e.g., Belcher, Reference Belcher2017; Grapin & Llosa, Reference Grapin and Llosa2020; Jewitt, Reference Jewitt2008; Kress, Reference Kress2000), and recent research has shown that the target of instruction is being expanded to include diverse modes of communication. Such change can be found in recent research that has proposed pedagogical and assessment ideas for multimodal composing in L2 classrooms (e.g., Hafner & Ho, 2020*; Jiang et al., 2022*; Lim & Polio, 2020*). Studies have also investigated how L2 writers use their multiple semiotic resources to construct intended meaning (e.g., Cimasko & Shin, 2017*; Hafner & Miller, 2011*; Jiang, Reference Jiang2018*). Finally, researchers have implemented multimodal tasks in class and explored how learners and/or teachers perceive and respond to multimodal tasks in different learning contexts (e.g., Jiang et al., 2021*, Reference Jiang, Yu and Zhao2022).
3. Overview of the research timeline
Given the increasingly multimodal nature of communication and the influx of scholarly activity on the topic of multimodal composing, a comprehensive synthesis of studies in this area is necessary. As such, the current piece aims to provide a research timeline on the topic of multimodal composing in the field of SLA. In this research timeline, we specifically define multimodal composing in terms of writers’ use of nonlinguistic resources along with written (and/or spoken) words to achieve a goal of constructing messages. Some L1-based research may view visual arts or dance performances as multimodal texts, but we find that in L2 research, participants typically engage in text construction activities while composing a multimodal text such as digital storytelling, presentation slides, or similar tasks (see Lim & Polio, 2020*). Thus, in this research timeline, we focus on previous L2 literature in which multimodal composing is the primary focus of researchers’ inquiry.
In terms of inclusion and exclusion criteria, studies needed to: (1) focus on the intersection of multimodal composing and SLA, (2) have been published before the end of the 2022 calendar year, and (3) be published in mid-to-high impact journals. These three criteria are further explained in the paragraphs that follow.
As mentioned, we were interested in studies that discussed some aspect(s) of multimodal composing in the domain of SLA. Therefore, although there is a significant body of scholarship in the area of L1 composition studies, these were not included unless there was also an overt focus on an aspect of L2 learning. Similarly, we did not include any computer-based writing studies that have only an ancillary or incidential focus on multimodality. For example, Kessler (Reference Kessler2020) touches upon students’ uses of different modes for meaning-making when engaging in writing tasks (e.g., using different colors for facilitating reading-writing connections), yet the focus of the study is primarily on student-initiated technology use rather than multimodality or meaning-making. Additionally, other earlier digital writing studies with wikis and blogs often involve multimodal composing tasks in which writers use nonlinguistic resources (e.g., images, colors, animated objects) to enrich their texts. However, the focus is usually placed more so on analyzing students’ subsequent written texts (i.e., the linguistic features) or students’ linguistic interactions (typically involving language-related episodes). Thus, such studies were also excluded. For a review and research agenda of digital learning tasks, see Reinhardt (Reference Reinhardt2019) and Smith and González-Lloret (Reference Smith and González-Lloret2020).
Apart from having an overt focus on multimodality and SLA, in order to be included in the timeline, studies needed to have been published before the end of the 2022 calendar year (i.e., the time when this article was written). Finally, in the timeline, we included only empirical studies that were published in mid-to-high impact journals. This was operationalized as the journal having an impact factor of at least 1.0 or higher.
When searching for studies on the topic of multimodal composing in the context of SLA, we note that our search included both targeted journal searches and broader searches of scholarly databases. For example, the two authors collectively made a list of SLA-oriented journals that were known (to the authors) to have published research on multimodal composing (e.g., Journal of Second Language Writing, Language Teaching Research, System, TESOL Quarterly, and others). Apart from searching these journals for specific keywords (e.g., multimodal, mode, DMC [digital multimodal composing]), a broader search was also conducted using the research article databases Google Scholar and Linguistics and Language Behavior Abstracts. Both authors contributed to the creation of an exhaustive list of multimodal composing studies.
In what follows, we outline some of the most prominent studies on the topic of multimodal composing, beginning with Tardy's (2005)* study and continuing with the nearly two decades that follow. When doing so, we highlight the following main topic areas (or themes) and the results of the research. Notably, these themes were developed as a result of a collaborative and iterative thematic coding process (see Polio & Friedman, Reference Polio and Friedman2017), which involved both authors. Specifically, each author initially analyzed approximately 20 different studies and generated thematic codes (i.e., themes) based on the topic areas discussed in those studies that were reviewed. Afterwards, the authors compared and combined their lists of thematic codes. This resulted in seven different themes, which were later used to code the studies presented in the research timeline. Notably, in many instances, a single article might address multiple areas. Therefore, in such cases, articles in the timeline are coded with more than one thematic code. These seven themes consisted of:
-
A. Direct and/or indirect evidence of L2 learning
-
B. Identity and authorial voice in multimodal composing
-
C. Teacher perceptions and beliefs
-
D. Learner perceptions and reflections
-
E. Writers’ multimodal composing processes
-
F. Interplay of linguistic and nonlinguistic modes
-
G. Outcomes and assessment of multimodal composing
Similarly, to provide additional information about the methodological choices of the studies’ authors, each study was subsequently coded to track trends in their research designs. Using the same thematic coding procedure described above, we used the additional codes below to signal information about the studies’ methods:
• Qual = Qualitative research design
• Quant = Quantitative research design
• MMR = Mixed methods research design
• SL = Second language context
• FL = Foreign language context
• K12 = Primary and/or secondary school contexts
• Univ = University context
• Ind = Individual writing
• CW = Collaborative writing
Note. Authors’ names are shown in small capitals when the study referred to appears in this timeline.
Jungmin Lim is an Assistant Professor in the College of Liberal Arts at Dankook University, South Korea. Her research interests are in the areas of second language writing, language testing, and research methods. Her research has appeared in Journal of Second Language Writing, Assessing Writing, Language Learning, and Studies in Second Language Acquisition.
Matt Kessler is an Assistant Professor of Applied Linguistics at the University of South Florida's Department of World Languages. His research focuses on issues related to second language writing, genre-based teaching and learning, and computer assisted language learning. Matt's research has appeared in journals such as ELT Journal, English for Specific Purposes, Foreign Language Annals, Journal of Second Language Writing, and System.
