Linguistic encoding of locative relations

The linguistic encoding of locative spatial relations requires the mention of Figure and Ground objects as well as the spatial relation between them. In a spatial configuration, the Figure refers to the smaller and foregrounded object, which is located with respect to a backgrounded, and usually bigger object, known as the Ground (Talmy, Reference Talmy1985). Figure 1 depicts various locative spatial relations between the pen (Figure) and the paper (Ground). Descriptions of locative spatial relations can vary in requiring an external perspective, which may be viewer- or environment-centered (Levinson, Reference Levinson, Bloom, Peterson, Nadel and Garrett1996, Reference Levinson2003; Majid, Reference Majid2002; Pederson, Danziger, Wilkins, Levinson, Kita, & Senft, Reference Pederson, Danziger, Wilkins, Levinson, Kita and Senft1998; see also Li & Gleitman, Reference Li and Gleitman2002). In this study, we are interested in the viewer-centered spatial relations that are especially likely to manifest in cases where Ground objects do not have intrinsic features, and thus require speakers to consider a viewpoint in using spatial terms. For instance, in Figure 1a, the spatial relation between the objects is independent of the viewpoint of the observer. However, in some spatial relations, such as Left-Right (Figure 1b) or Front-Behind (Figure 1c), the spatial relation between the objects depends on the viewpoint of the observer (see Martin & Sera, Reference Martin and Sera2006 for a discussion; see also Landau, Reference Landau2017; Levinson, Reference Levinson2003). For Front-Behind, informational cues such as visibility (in the case of Front) and occlusion (in the case of Behind) provide information for the asymmetrical relationship that helps distinguish the two spatial relations from each other (Grigoroglou et al., Reference Grigoroglou, Johanson and Papafragou2019). The case of Left-Right, however, does not contain any informational cues to distinguish them from each other and remains to be two categorically distinct symmetrical spatial layouts. The current study focuses on encoding of Left-Right relations in which language users need to be explicit in their descriptions to be informative.

Figure 1. Objects in viewpoint-independent (a) and viewpoint-dependent (b & c) spatial configurations.

Linguistic encoding of space in speech, sign and co-speech gestures

Sign

In encoding locative spatial relations, sign languages incorporate linguistic forms that bear iconic links to their meanings. The most frequent iconic form for describing spatial relations, including Left-Right, is through the use of morphologically complex classifier constructions, as shown in Figure 2d (Emmorey, Reference Emmorey2002; Janke & Marshall, Reference Janke and Marshall2017; Perniss, Zwitserlood, & Özyürek, Reference Perniss, Zwitserlood and Özyürek2015a; Supalla, Reference Supalla1982; Zwitserlood, Reference Zwitserlood, Pfau, Steinbach and Woll2012). In these constructions, the location of the hands encodes the location of the objects with respect to each other, while the handshape encodes objects’ shape information (Emmorey, Reference Emmorey2002; Perniss et al., Reference Perniss, Zwitserlood and Özyürek2015a; Supalla, Reference Supalla1982; Zwitserlood, Reference Zwitserlood, Pfau, Steinbach and Woll2012). To illustrate, while describing the spatial relation between the cup and the toothbrush, signers first introduce the lexical signs for the cup (Figure 2a) and the toothbrush (Figure 2c), and later they choose classifier handshapes to indicate the size and shape of these two objects (e.g., Figure 2d). More specifically, signers choose a round handshape to represent the round nature of the cup and an elongated handshape (i.e., index finger) to represent the shape of the toothbrush. Later, they position their hands in the signing space in a way analogue to the spatial relations in the picture. Thus, the representation of spatial relations between objects on the signing space maps onto the exact spatial relation between the objects in real space from a specific viewpoint (mainly signer/viewer viewpoint). For instance, if the toothbrush was located on the right of the cup, then, the signer would have positioned her handshape with an index finger for the toothbrush to the right side of the classifier handshape used for locating the cup to her left from her viewpoint. This then allows a diagrammatically iconic expression considering the relative locations of objects (Perniss, Reference Perniss2007).

Figure 2. Informative description from a TİD signer by using a classifier construction in encoding the spatial relation between the cup and the toothbrush.

In addition to classifier constructions, signers can use other linguistic forms – albeit less frequently – to express the spatial relation between objects. These include relational lexemes (Arık, Reference Arık2003; Sümer, Reference Sümer2015), tracing the shape of the objects and locating them on the signing space (Perniss et al., Reference Perniss, Zwitserlood and Özyürek2015a), pointing to indicate the object’s location in the signing space (Karadöller, Sümer, & Özyürek, Reference Karadöller, Sümer and Özyürek2021), and lexical verb placements (Newport, Reference Newport1988) (See coding section and Figures 9, 10, for more details). Even though the handshapes in these forms are not iconic themselves, similar to classifier constructions, all of these forms give iconic information about the relative spatial locations of the objects with respect to each other from signers’ viewpoint in a diagrammatically iconic way. In this sense, they are almost always informative in conveying object locations and differ from the under-informative expressions in spoken languages (e.g., Yan ‘Side’ in Turkish or Next to in English), which fail to distinguish between the two symmetrical layouts.

Co-speech gestures

Visual modes of expressions allowing iconic and analogue encodings are not specific to sign languages. These types of expressions can be found in spoken languages in the form of co-speech gestures (Kendon, Reference Kendon2004; Kita & Özyürek, Reference Kita and Özyürek2003; McNeill, Reference McNeill1992, Reference McNeill2005; Özyürek, Reference Özyürek, Hickmann, Veneziano and Jisa2018). Co-speech gestures can be used to indicate locations of objects in gesture space in an analogue manner due to their iconic affordances. Therefore, spoken expressions accompanied by gestures might convey more spatial information than speech alone. For instance, when describing locations in space, speakers sometimes encode space in an ambiguous way (e.g., Here or There) in speech while also using gestures to indicate relative locations of entities in space (McNeill, Reference McNeill2005; Peeters & Özyürek, Reference Peeters and Özyürek2016). Figure 3 exemplifies the use of a directional pointing gesture used with speech. In this example, although speech fails to give information regarding the exact spatial relation between the objects, the directional pointing gesture to the right gestural space indicates that the fork is on the right side. In this sense, even a pointing gesture can map the location of an object in real space to gesture space in a diagrammatically iconic way in relation to speaker’s body. In such descriptions, gestures might serve as a helpful tool during communication by disambiguating information conveyed in speech (McNeill, Reference McNeill1992; see more examples in the coding section) and thus contribute to linguistic encoding of the spatial relation.

Figure 3. An example from a Turkish speaker using a pointing gesture towards the right while mentioning “Side” in speech.

Notes. The underlined word denotes the speech that gesture temporally overlaps with. The description is informative only when both speech and gesture are considered.

Development of linguistic encoding of space in speech, sign, and co-speech gestures

Present study

As the above research shows, the visual modality of expression (i.e., sign and gesture) seems to be privileged in providing more spatial information compared to speech possibly due to the affordances of iconic form-meaning mappings (Goldin-Meadow & Beilock, Reference Goldin-Meadow and Beilock2010; Sommerville, Woodward, & Needham, Reference Sommerville, Woodward and Needham2005). However, the role of visual modality as a modulating factor in spatial language development has not been fully examined. Until now, researchers have typically studied sign or speech independently. Moreover, limited work studying both sign and speech has compared sign to speech alone. However, comparing sign to speech and speech-gesture combinations could more realistically approximate the development of spatial language use as it would capture all semiotic tools available to spoken languages, including both arbitrary/categorical (i.e., in auditory-vocal speech) and iconic/analogue (i.e., in visual-spatial co-speech gestures) expressions (Goldin-Meadow & Brentari, Reference Goldin-Meadow and Brentari2017; Özyürek & Woll, Reference Özyürek, Woll and Hagoort2019). Hence, here we investigate, for the first time, how deaf child and adult signers acquiring sign language from birth and hearing child and adult speakers express Left-Right relations in sign, speech, and speech-gesture combinations to provide informative expressions.

We defined informativeness in terms of whether a participant’s description distinguishes symmetrical Left-Right relations (see Grigoroglou & Papafragou, Reference Grigoroglou and Papafragou2019a for a similar approach in the domain of events). In the present study, participants engaged in a communicative task in which they saw displays with 4 pictures presenting different spatial configurations of the same two objects (see Karadöller, Sümer, Ünal, & Özyürek, Reference Karadöller, Sümer, Ünal and Özyürek2022; Manhardt et al., Reference Manhardt, Özyürek, Sümer, Mulder, Karadöller and Brouwer2020; Manhardt, Brouwer, & Özyürek, Reference Manhardt, Brouwer and Özyürek2021 for a similar procedure). Within one display, the only distinguishing feature of the pictures was the spatial configuration between the objects (see Figure 4 for examples of displays). One of the pictures in the display was the “target picture” to be described to a confederate addressee who had to find it on her tablet among the same four pictures displayed in a different way. A detailed description of the stimuli material, procedure, and coding is provided in the methods section.

Figure 4. Non-contrast (a) and contrast (b) experimental displays.

We chose Turkish and TİD because there is a strong tendency for Turkish speakers, especially children, to use under-informative descriptions (e.g., Yan ‘Side’) while describing Left-Right relations between objects, even at the age of 8 (Sümer, Reference Sümer2015). We focused on age 8 to build on this previous work in Turkish and TİD. Moreover, although not directly studied for the domain of space, Turkish has been found to be a high gesture culture in general (Azar, Özyürek, & Backus, Reference Azar, Özyürek and Backus2020). Due to these features of Turkish and based on previous work showing that gestures can be used as a tool to convey spatial information by children (Sauter et al., Reference Sauter, Uttal, Alman, Goldin-Meadow and Levine2012; Sekine, Reference Sekine2009), we investigate whether signing-children still have an advantage in describing Left-Right relations in informative ways compared to speaking-children even when their multimodal expressions are taken into account.

Predictions

We grouped our predictions into two clusters. First, we compared sign to speech (i.e., Unimodal Descriptions). Then, we compared sign to speech by also taking into account gestures (i.e., Multimodal Descriptions). In each section, we also compared the development of spatial expressions of children to adults.

In unimodal descriptions, we expected an overall effect of modality, such that signers would produce informative descriptions in sign more frequently than speakers would do so in speech. This would be due to the affordances of visual modality that allow iconic/analogue expressions (Goldin-Meadow & Brentari, Reference Goldin-Meadow and Brentari2017; Özyürek & Woll, Reference Özyürek, Woll and Hagoort2019; Taub, Reference Taub2001; Taub & Galvan, Reference Taub and Galvan2001). Regarding developmental differences between children and adults in the two groups, there are two possibilities. One possibility is that speaking-children would produce informative expressions less frequently than adults, but signing-children would produce informative descriptions equally frequently as signing-adults. This would be in line with previously reported developmental patterns for speaking-children (Clark, Reference Clark1973) and signing-children who have been found to produce adult-like expressions of Left-Right relations starting from 4 years of age (Sümer, Reference Sümer2015). Alternatively, signing-children, similar to speaking-children, might produce informative descriptions less frequently than adults despite the advantage of the visual modality. This latter possibility would indicate a universal challenge in conceptual development of the spatial domain, specifically for Left-Right regardless of the modality of expression (Clark, Reference Clark1973).

Turning to multimodal descriptions, we first predicted that iconic affordances of gestures might facilitate expressing spatial relations in informative ways. In line with this, we expected speaking-children to use co-speech gestures that complement their under-informative speech more frequently than adults who would be mostly already informative in their speech (Alibali & Goldin-Meadow, Reference Alibali and Goldin-Meadow1993; Church & Goldin-Meadow, Reference Church and Goldin-Meadow1986; Perry, Church, & Goldin-Meadow, Reference Perry, Church and Goldin-Meadow1992; Sauter et al., Reference Sauter, Uttal, Alman, Goldin-Meadow and Levine2012).

Next, when comparing speech-gesture combinations to sign, if co-speech gestures help with the informativeness of speakers’ expressions, we expected modality differences between speakers and signers and developmental differences between speaking-children and speaking-adults to disappear (Goldin-Meadow & Brentari, Reference Goldin-Meadow and Brentari2017; Özyürek & Woll, Reference Özyürek, Woll and Hagoort2019). However, it is still possible for signers to produce informative descriptions more frequently than speakers. Such a finding could be due to the fact that iconic forms in sign are conventional linguistic tools (Brentari, Reference Brentari2010; Emmorey, Reference Emmorey2002; Klima & Bellugi, Reference Klima and Bellugi1979), unlike co-speech gestures that are learned and used flexibly as a composite system together with speech (Kendon, Reference Kendon2004; McNeill, Reference McNeill1992, Reference McNeill2005; see Özyürek & Woll, Reference Özyürek, Woll and Hagoort2019 and Perniss, Özyürek, & Morgan Reference Perniss, Özyürek and Morgan2015b for a discussion).

Participants

Speaking participants consisted of hearing monolingual Turkish speaking-children (N = 24; 14 Female; Mean Age = 8;6; SD Age = 0.93; Age Range = 6;7 – 9;5) and adults (N = 23; 14 Female; Mean Age = 35;9; SD Age = 10.29; Age Range = 19;8 – 50). Data from 1 additional speaking-child and 2 additional speaking-adults were excluded from the study since they were bilingual. Additionally, 2 speaking-children and 2 speaking-adults were excluded due to having average number of spatial encodings being 3 standard deviations below the group mean.

Signing participants consisted of TİD signing-children (N = 21; 12 Female; Mean Age = 8;5; SD Age = 1.29; Age Range = 6;8 – 11) and adults (N = 26; 21 Female; Mean Age = 29;10; SD Age = 8.34; Age Range = 18;2 – 48;7). Data from an additional 6 signing-children and 4 signing-adults were excluded from the study due to failure to follow the instructions (n = 7), problems with the testing equipment (n = 1), or disruption during the testing sessions (n = 2). All signing participants were profoundly and congenitally deaf and acquired TİD from birth from their deaf signing parents. They did not receive speech therapy and were exposed to written Turkish when they started the school for the deaf.Footnote ¹

We determined the sample size based on convenience. Working with special populations poses certain challenges in reaching participants. Here, we report data from signers who had been exposed to sign language from birth by their signing deaf parents. This group represents 10% of the deaf population in the world (Mitchell & Karchmer, Reference Mitchell and Karchmer2004) and in Turkey (İlkbaşaran, Reference İlkbaşaran2015). Hence, the number of participants in each group reported in this study (speaking-children, signing-children, speaking-adults, and signing-adults) was determined based on the total number of deaf children attending the deaf schools in İstanbul that we could collect data from. We collected data from all students who matched our criteria (e.g., age, absence of comorbid health issues). Finally, to our knowledge, the current sample incorporates the largest number of deaf signers who have been exposed to sign language from birth by their parents in comparison to previous studies conducted in the field. We could not balance, however, the gender diversity within the adults’ group as we were limited to the number of deaf adults living in İstanbul. We collected data from almost all deaf adults falling under our criteria and willing to participate. Participation was voluntary and at the end of the study all children received a gender-neutral color pencil kit and adult participants received monetary compensation for their participation.

We compared speaking and signing participants’ ages and visual spatial cognitive abilities (i.e., Corsi Block Tapping Task Score; Corsi, Reference Corsi1972) to ensure similarity, separately for children and adults. To do so, we conducted Bayesian t-tests which assessed the probability of the mean difference (M_DIFF) greater than zero and less than zero using the R package BayesianFirstAid (version 0.1; Bååth, Reference Bååth2014). Signing- and speaking-children were similar in age (Bayesian two sample t-test: M_DIFF (–5) > 0: p = 0.556, M_DIFF (5) < 0: p = 0.444) and in Corsi Block Tapping Task score (Bayesian two sample t-test: M_DIFF (–5) > 0: p = 0.972, M_DIFF (5) < 0: p = 0.280). Moreover, signing- and speaking-adults were also similar in age (Bayesian two sample t-test: M_DIFF (–5) > 0: p = 0.736, M_DIFF (5) < 0: p = 0.264) and Corsi Block Tapping Task score (Bayesian two sample t-test: M_DIFF (–5) > 0: p = 0.866, M_DIFF (5) < 0: p = 0.134).

Materials

Stimuli consisted of 84 displays. Each display had 4 pictures presented in a 2 x 2 grid. Individual pictures in each display showed the same two objects in various spatial configurations. Ground objects (e.g., a jar) were always in the center of the pictures and they did not have “intrinsic” sides determined by their shape (e.g., a picture frame has an intrinsic front, but a jar does not). Figure objects (e.g., a pencil) changed their location in relation to the Ground objects. In each display, the target picture to be described was indicated by an arrow. Experimental displays (n = 28) consisted of Left-Right spatial configurations between objects (e.g., the pencil is to the left of the cup). In half of the experimental displays, only the target picture contained Left or Right spatial configuration between objects and all non-target pictures contained spatial configurations other than Left-Right (i.e., Non-contrast displays). In the remaining half of the experimental displays, one non-target picture contained the contrastive spatial configuration (i.e., contrast picture; if the target picture contained Left spatial configuration, contrast picture contained Right spatial configuration or vice versa) and remaining pictures contained spatial configurations other than Left-Right (i.e., Contrast displays). See Figure 4 for example displays. The rationale for having Contrast displays in addition to Non-contrast displays was to increase the need for informativeness in describing the spatial relation between the objects in the target pictures. With this manipulation, we aimed to test if participants use more informative descriptions for Contrast than Non-Contrast displays to distinguish the target picture in a more distinctive way among the other pictures in the display (see Manhardt et al., Reference Manhardt, Özyürek, Sümer, Mulder, Karadöller and Brouwer2020, Reference Manhardt, Brouwer and Özyürek2021 for a similar procedure).

In addition to the experimental displays, we included 56 filler displays to avoid attention to the Left-Right spatial configurations. Filler displays consisted of target pictures in Front (n = 14), Behind (n = 14), In (n = 14), and On (n = 14) spatial relations between objects.

All visual displays were piloted to ensure that both children and adults could identify and name the objects in the display. Within all 84 displays, Figure objects (e.g., pen) were presented only once. Ground objects (e.g., cup) were presented 4 times but always with other Figure objects (e.g., cup-pencil, cup-egg, cup-fork, cup-chocolate). The same Ground objects were never presented twice in a row. Moreover, the same relation between the objects as a target picture was not presented more than twice in a row to avoid biases to one type of spatial relation. There were two sets of displays with the same Ground objects but with different Figure objects. All other configurations were similar across the two sets. The order of the displays and locations of the pictures in each display were randomized across each participant.

Procedure

The description and familiarization tasks were originally designed as part of an eye-tracking experiment – however, for the purpose of this paper, we only reported the description data.

Description task

Participants were presented with the description task after the familiarization task was completed (see details below). Trials started with a fixation cross (2000ms), followed by a display of 4 pictures (1000ms). Next, an arrow appeared (for 500ms) to indicate the target item and disappeared and 4 pictures remained on the screen (2000ms) until visual white noise appeared. Participants were instructed to describe the target picture to an addressee sitting across the table immediately after the appearance of the visual white noise. This was done to prevent children from pointing towards the screen to show the pictures or objects in a picture while describing. Participants were instructed that the addressee would choose the target picture on her tablet based on the participant’s description. They were also aware that the addressee had the same 4 pictures but in a different arrangement in the display and without the arrow. The addressee was a confederate and pretended to choose a picture on her tablet based on the participant’s description. Participants moved to the next trial by pressing the ENTER key on the keyboard. Having an addressee, albeit as a confederate, was especially important considering previous reports on children’s tendencies to be under-informative in the presence of an inattentive listener or in the absence of a listener (Bahtiyar & Küntay, Reference Bahtiyar and Küntay2009; Girbau, Reference Girbau2001; Grigoroglou & Papafragou, Reference Grigoroglou and Papafragou2019b). See figure 5 for the timeline of a trial in the description task.

Figure 5. Timeline of a trial in the description task.

At the beginning of the description task, participants engaged in practice trials (n = 3) and these trials were repeated if necessary. During practice trials, when participants failed to understand the task instructions, the experimenter repeated them. Both during the practice trials and throughout the experiment, the addressee did not give feedback on whether or not the description was correct in order to avoid biasing the responses in the upcoming trials but pretended to have found the right picture. When there was missing spatial information in participant’s description, the addressee only asked the location of the Figure object. In such cases, speaking participants were asked for the location of the figure object (e.g., Kalem nerede? ‘Where is the pencil?’) in Turkish, and signing participants were asked for the location of the Figure object using the lexical sign of where and the lexical sign of the Figure object found in the target picture in TİD. In order to provide consistent feedback, no other instructions were given to the participants. The addressee asked for such a clarification only once. Even if participants provided a description with missing spatial information in the second round, the addressee did not ask for further clarification and pretended to choose a picture in her tablet. Moreover, we did not provide explicit instructions for speaking participants to gesture or not. Thus, all of the gestures were spontaneously produced. Hearing adult speakers of Turkish were present as an addressee and as an experimenter for speaking participants, and deaf adult signers of TİD were present as an addressee and as an experimenter for signing participants.

Annotation and coding

All descriptions produced in the description task were annotated for Target Pictures. Descriptions with speech, gesture, and sign were coded using ELAN (Version 4.9.3), a free annotation tool (http://tla.mpi.nl/tools/tla-tools/elan/) for multimedia resources developed by the Max Planck Institute for Psycholinguistics, The Language Archive, Nijmegen, The Netherlands (Wittenburg, Brugman, Russel, Klassmann, & Sloetjes, Reference Wittenburg, Brugman, Russel, Klassmann and Sloetjes2006).

We coded descriptions in speech, gesture, and sign. Next, we formed informativeness categories first based on the information conveyed in speech, later by considering co-speech gestures along with speech, and in sign. We operationalized informativeness as whether participants’ descriptions provide a uniquely referring expression to distinguish the spatial relation between the objects in the target picture from other referent pictures in the display. In almost all descriptions, participants used their own perspective in encoding the spatial relations between objects.

Speech

Speech data were annotated and coded by the first author who is a hearing native speaker of Turkish. We did not have a reliability coding for speech as speech coding involved the presence of spatial terms (e.g., Left) that were unambiguously heard and identified by a hearing native speaker of Turkish. We grouped participants’ descriptions into two categories based on whether or not the linguistic form used to encode the spatial relation in the description was informative in uniquely identifying the target picture when only speech is considered.

1. (1) Informative in Speech: This category consisted of descriptions that included Sol ‘Left’- Sağ ‘Right’ spatial terms. The specific spatial terms used in these descriptions provided uniquely referring expressions that distinguished the target picture (Figure 6a).

2. (2) Under-informative in Speech: This category consisted of all the remaining descriptions as they failed to provide enough information to uniquely identify the target picture. These descriptions included the following sub-categories:

   1. (2a) Descriptions with a general relational term (Yan ‘Side’; Figure 6b) that failed to provide uniquely referring information that distinguished the actual spatial relation (e.g., which object is on the left side and which object is on the right side).

   2. (2b) Specific spatial terms other than Left-Right (e.g., Ön ‘Front’; Figure 6c). These descriptions were especially frequent in children’s descriptions as they tended to encode Left-Right with other spatial relations especially with Front (Abarbanell & Li, Reference Abarbanell and Li2021; Sümer et al., Reference Sümer, Perniss, Zwitserlood, Özyürek, Bello, Guarini, McShane and Scassellati2014). We did not want to render these descriptions as incorrect since we could not be sure whether children used these spatial terms due to having a difficulty in mapping Left-Right spatial terms onto Left-Right relations. A few cases also included spatial terms other than Front to describe target pictures in Left-Right spatial relations. Based on our definition of informativeness, all of these descriptions that encoded Left-Right with other spatial terms were not informative enough for the addressee to pick up the correct picture from her tablet and thus was considered under-informative.

   3. (2c) Descriptions with missing spatial relation where participants only labeled the objects but not the spatial relation between them (Figure 6d).

Figure 6. Examples from Turkish speakers describing the spatial relation between the pencil and the cup using (a) Left-Right spatial terms, (b) general relational term Side, (c) spatial terms other than Left-Right, (d) missing encoding of spatial relation between the objects.

Speech and gesture

We further coded spontaneous co-speech gestures as identified by strokes (see Kita, van der Hulst, & van Gijn, Reference Kita, van der Hulst, van Gijn, Wachsmuth and Fröhlich1998) produced by participants that conveyed information regarding the location of the two objects or the spatial relation between the objects. We did not take into account other types of gestures such as beat gestures. We did this coding per description and regardless of the type of speech used in the description.

In order to ensure reliability, 25% of the gesture data (5 Children and 5 Adults) were coded by another hearing native speaker of Turkish. There was substantial agreement between the coders for the type of spatial gestures used to localize Figure (88% Agreement, kappa = 0. 77) and Ground (92% Agreement, kappa = 0.79) objects. All disagreements were discussed to reach a 100% agreement.

For each description, we coded gestures separately for Figure and Ground objects. These gestures included either directional pointing gestures indicating the location of the Figure or Ground Object in an analogue way (Figure 3 and 7) or iconic hand placement gestures indicating the location of the Figure and/or Ground object on the gesture space (Figure 8). Both of the spatial gesture types, like linguistic structures found in sign to represent space, give spatial information about the Left-Right relations between objects from the viewpoint of the speaker, and help identify the target picture uniquely from other referents in the display. As a next step, we considered these spatial gestures on top of what has been conveyed in speech and redefined the informativeness categories for speakers creating multimodally informative categories:

1. (1) Informative in speech: This category only involved Left-Right spatial terms as described above. Some of the descriptions with Left-Right spatial terms also included accompanying spatial gestures. However, for these descriptions, spatial gestures did not add to the informativeness of the description and were considered redundant. That is, speech was already informative even without considering the gestures. Thus, they did not form a new category (see Figure 7).

2. (2) Informative in speech-plus-gesture: This category consisted of descriptions that include general spatial term Yan ‘Side’ in speech together with spatial gestures. In these descriptions, spatial information missing from descriptions with Yan ‘Side’ in speech was conveyed via spatial gestures (see Figures 3 and 8 for examples). Thus, these descriptions were informative only when the spatial gestures were considered.

3. (3) Under-informative even when gestures are considered: This category consisted of descriptions with specific spatial terms other than Left-Right (e.g., Front; Figure 6c) as well as descriptions with missing spatial relation where participants only label the objects but not the spatial relation between them (Figure 6d). These descriptions were still under-informative even when gestures were considered together with speech. That is, gestures did not contribute to the informativeness of the description above speech.

Figure 7. Informative in Speech description from a Turkish speaker using a specific spatial term (Left) together with a directional pointing gesture to the left.

Note. Underlined words denote the speech that the gesture overlapped with. The description is informative even when only speech is considered.

Figure 8. Informative in speech-plus-gesture description from a Turkish speaker using a general spatial term (Side) together with iconic hand placement gestures.

Notes. Participant introduced gestures sequentially. Gesture indicating the basket (RH) was performed when the participant mentioned the basket in her speech. Gesture indicating the newspaper (LH) was performed when the participant mentioned the newspaper. Both gestures remained in the gesture space until the end of the sentence. The description is informative only when information in both speech and gestures combined is considered.

Sign

Sign data were annotated by a hearing L2 signer of TİD. The data were coded by another hearing L2 signer of TİD. Annotations and coding were checked by a trained native deaf signer of TİD. We did not have a reliability coding for sign as we only included the linguistic forms that were unambiguously approved by this signer in the final dataset.

We coded descriptions for the presence of spatial relation between the objects and the type of linguistic form used to localize the Figure object in relation to the Ground object. Signers used 5 different Linguistic forms. These forms were classifier constructions (Figure 2d), which are one of the most common forms to localize the Figure object in relation to the Ground object in sign languages in general (Emmorey, Reference Emmorey2002) and also in TİD (Arık, Reference Arık2003; Karadöller et al., Reference Karadöller, Sümer and Özyürek2021; Sümer, Reference Sümer2015; Sümer, et al., Reference Sümer, Perniss, Zwitserlood, Özyürek, Bello, Guarini, McShane and Scassellati2014). They allow signers to encode information about the entities through the handshape classifications of objects (e.g., Emmorey, Reference Emmorey2002; Janke & Marshall, Reference Janke and Marshall2017; Manhardt et al., Reference Manhardt, Özyürek, Sümer, Mulder, Karadöller and Brouwer2020; Perniss et al., Reference Perniss, Zwitserlood and Özyürek2015a; Zwitserlood, Reference Zwitserlood, Pfau, Steinbach and Woll2012). Alternatively, signers also used other forms such as relational lexemes, which are the lexical signs for spatial terms used in sign languages (Arık, Reference Arık2003; Manhardt et al., Reference Manhardt, Özyürek, Sümer, Mulder, Karadöller and Brouwer2020; Figure 9); tracing the shape of the Figure object on the signing space (Karadöller et al., Reference Karadöller, Sümer and Özyürek2021; Figure 10); pointing to the location of the Figure object on the signing space (Karadöller et al., Reference Karadöller, Sümer and Özyürek2021); placing a lexical verb to locate an object on the signing space (see Karadöller, Reference Karadöller2021).

Figure 9. Informative in sign description from a TİD signer by using a relational lexeme for Left in encoding the spatial relation between the cup and the ruler.

Figure 10. Informative in sign description from a TİD signer by tracing the shape of the Figure object on the signing space in encoding the spatial relation between the cup and the ruler.

We grouped participants’ descriptions into two categories in terms of whether or not the description was uniquely informative (i.e., which object is where relative to the other based on diagrammatical iconicity) in identifying the target picture depending on the linguistic form that is used to encode the spatial relation in sign.

1. (1) Informative in Sign: This category included all of the linguistic forms mentioned above as they were describing the exact spatial relation between the objects and distinguishing the target picture uniquely from the other pictures in the display.

2. (2) Under-informative in Sign: This category included descriptions with incorrect (e.g., describing that the pen is in front of the paper, despite the target picture showing that the pen is to the left of the paper) and missing (e.g., only labeling the Figure and Ground object but not the spatial relation of the Figure object in relation to the Ground object, Figure 11) spatial relation.

Figure 11. Under-informative description in sign with missing spatial relation between objects.

Unimodal descriptions

First, we investigated whether the frequency of informative descriptions differs across the modalities and age groups (see dark green bars compared to yellow bars in Figure 12). We used a glmer model to test the fixed effects of Modality (Informative in speech versus Informative in sign) and Age Group (Children versus Adults), and an interaction between them on binary values for the presence of informative descriptions (Present = 1, Absent = 0) at the item level. The fixed effects of Modality and Age Group were analyzed with centered contrasts (–0.5, 0.5). The model revealed a fixed effect of Modality (β = –4.32, SE = 0.68, p < 0.001): signers produced informative descriptions (Mean = 0.94; SD = 0.24) more frequently than speakers (Mean = 0.58; SD = 0.49). The model also revealed a fixed effect of Age Group (β = 4.57, SE = 0.69, p < 0.001): adults produced informative descriptions (Mean = 0.93; SD = 0.26) more frequently than children (Mean = 0.58; SD = 0.49). There was no interaction between Modality and Age Group (β = 1.61, SE = 1.26, p = 0.203).

Figure 12. Proportion of Informative descriptions across Age Groups and Modality.

Multimodal descriptions

First of all, we tested if children’s spatial gestures help convey more information that is missing in Under-Informative speech than adults. To test this, we compared the frequency of descriptions that were Informative in speech-plus-gesture across children and adults (see light green bars in Figure 12). We used a glmer model to test the fixed effect of Age Group (Children, Adults) on whether the descriptions were Informative in speech-plus-gesture (1) or not (0) at the item level. The fixed effect of Age Group was analyzed with centered contrasts (–0.5, 0.5). The model revealed a fixed effect of Age Group (β = 2.94, SE = 0.70, p < 0.001): children (Mean = 0.45; SD = 0.46) produced descriptions that were Informative in speech-plus-gesture more frequently than adults (Mean = 0.08; SD = 0.26).

Finally, we investigated whether frequency of informative descriptions changes across the modalities and age groups (see light and dark green bars compared to yellow bars in Figure 12). We used a glmer model to test the fixed effects of Modality (Informative in speech and Informative in speech-plus-gesture combined versus Informative in sign) and Age Group (Children versus Adults), and an interaction between them on binary values for the presence of informative description (Present = 1, Absent = 0) at the item level. The fixed effects of Modality and Age Group were analyzed with centered contrasts (–0.5, 0.5). The model revealed a fixed effect of Modality (β = –1.30, SE = 0.37, p < 0.001): signers (Mean = 0.94; SD = 0.27) produced informative descriptions (in sign) more frequently than speakers (Mean = 0.85; SD = 0.36) (in speech and speech-plus-gesture combined). The model also revealed a fixed effect of Age Group (β = 2.25, SE = 0.37, p < 0.001): adults (Mean = 0.97; SD = 0.18) produced informative descriptions more frequently than children (Mean = 0.82; SD = 0.38) There was no interaction between Modality and Age Group (β = –1.12, SE = 0.74, p = 0.127).

Sign has an advantage over speech both for children and adults

Unimodal comparisons across modalities revealed that signers produced informative descriptions more frequently than speakers regardless of the age group. This can be attributed to the facilitating effect of iconicity of sign language expressions providing more information compared to expressions in speech. Nevertheless, expressions in sign being more informative than those in speech should not be taken to indicate that signers have a more developed conception of Left-Right relations between objects than speakers do. Rather, it implies that iconic affordances of sign language expressions allow for more direct encoding and thus increase informativeness of the expression (see Sümer, Reference Sümer2015; Slonimska, Özyürek, & Capirci, Reference Slonimska, Özyürek and Capirci2020; Taub, Reference Taub2001 for adults). These findings provide important contributions for the development of spatial language use in speech and sign. That is, spatial language development may depend on several linguistic factors varying across languages (see Johnston & Slobin, Reference Johnston and Slobin1979). Despite these linguistic differences between TİD and Turkish, signing- and speaking-children encode In-On-Under at similar ages (Sümer & Özyürek, Reference Sümer and Özyürek2020). By contrast, they differ in encoding Left-Right. This difference points to a complex interplay between modality of expression, cognitive and linguistic development of space.

Gesture enhances informativeness of spoken expressions and more for children than for adults

When we considered multimodal descriptions of speakers, we found that both children and adults used spatial gestures that disambiguated the descriptions with Side. This trend was more prominent for children than adults. This is reminiscent of previous findings from other domains showing that gestures can help clarify the meaning of Side. For instance, Cook, Duffy, and Fenn (Reference Cook, Duffy and Fenn2013) showed that using gestures to teach math operations when referring to two sides of an equation helps children solve math equations more accurately. It seems that the use of gestures is an important tool for clarifying under-informative speech (see also Kelly, Reference Kelly2001). Moreover, we extend this finding to situations where participants were not explicitly instructed to gesture (Sauter et al., Reference Sauter, Uttal, Alman, Goldin-Meadow and Levine2012), since all of the gestures elicited in our study were spontaneous. Here, the frequent use of spontaneous gestures by Turkish speakers could be a reflection of Turkish being a high gesture culture (Azar et al., Reference Azar, Özyürek and Backus2020), hence raises possibilities for further investigations in other high (e.g., Italian) and low (e.g., Dutch) gesture cultures.

Moreover, we showed that children could communicate Left-Right spatial relations between two objects informatively through co-speech gestures before communicating them informatively in speech. This corroborates previous literature on gestures preceding speech, which was already established for several other domains (Alibali & Goldin-Meadow, Reference Alibali and Goldin-Meadow1993; Church & Goldin-Meadow, Reference Church and Goldin-Meadow1986; Perry et al., Reference Perry, Church and Goldin-Meadow1992; Sauter et al., Reference Sauter, Uttal, Alman, Goldin-Meadow and Levine2012). It seems that by age 8, children have some conceptual understanding of Left-Right spatial relations, yet they fail to map arbitrary/categorical linguistic forms in speech onto these conceptual representations. In these instances, gestures could act as a medium for representing already established spatial concepts that fail to surface in speech. These instances were very rare in adult language as adults could already map Left-Right spatial terms to these concepts. Together, our findings highlight the importance of considering children’s multimodal encodings in assessing their pragmatic (i.e., informativeness; Grigoroglou et al., Reference Grigoroglou, Johanson and Papafragou2019) and cognitive development (Hermer-Vazquez, Moffet, & Munkholm, Reference Hermer-Vazquez, Moffet and Munkholm2001).

Signed descriptions are more informative even when gestures are considered both for children and adults

When spatial gestures were considered together with speech and compared to expressions in sign, signers continued to be more informative than speakers. This can be attributed to having iconic expressions as obligatory and conventional linguistic forms for sign languages (Brentari, Reference Brentari2010; Emmorey, Reference Emmorey2002; Klima & Bellugi, Reference Klima and Bellugi1979). Conversely, co-speech gestures are used flexibly and only as a composite system together with speech (Kendon, Reference Kendon2004; McNeill, Reference McNeill1992, Reference McNeill2005; see Perniss et al., Reference Perniss, Özyürek and Morgan2015b for a discussion). Differences in the way co-speech gestures and signs are used during acquisition might have been the underlying factor of sign advantage.

Visual modality conveys more information than speech alone

Findings obtained for enhanced informativeness in both sign and gesture compared to speech add further evidence for the importance of body’s interaction with the world in shaping language and cognition (embodied cognition; Chu & Kita, Reference Chu and Kita2008; Goldin-Meadow, Reference Goldin-Meadow2016). It seems that children map their bodies’ interaction with the spatial relations between objects more easily onto iconic gestures/signs than speech (see Sümer, Reference Sümer2015 for a discussion). Both signers and speakers described the spatial relation between objects dominantly from their perspective. In those descriptions, participants might be mentally aligning the location of the objects that they saw on the computer screen with the left and right sides of their body. This alignment might have eased encoding the object locations on the sign/gesture space by placing hands rather than by mapping them by abstract spatial terms in speech. Moreover, signers used specific linguistic forms that are body-anchored (i.e., relational lexemes) – although few in frequency compared to classifiers. These signs have been found to facilitate learning to encode spatial relations earlier in sign than in speech and especially for Left-Right (Sümer, Reference Sümer2015; Sümer et al., Reference Sümer, Perniss, Zwitserlood, Özyürek, Bello, Guarini, McShane and Scassellati2014). Thus, having body-anchored lexical signs already in sign language lexicon might have allowed signers to encode more information with respect to Left-Right spatial relations between objects using their own body as a reference. Overall, visual modality allows more direct mapping of visual/bodily experience onto linguistic labels, which, therefore, results in more informative descriptions of Left-Right relations between objects.

Left-Right remains to be a challenging spatial domain for children even when visual modality is considered

Contrary to our initial expectation, developmental differences for the informativeness of Left-Right expressions did not disappear even when we considered visual modality of expression in sign or co-speech gestures. Children were less informative than adults for both unimodal and multimodal descriptions. This suggests an intricate interplay between language development, cognitive development, and the visual modality of expression. On the one hand, spatial gestures contribute to the informativeness of the descriptions for children more than adults. On the other hand, this contribution is not sufficient for speaking-children to reach adult levels of informativeness. Together, these results speak for the general claim that Left-Right is challenging for children regardless of the modality of communication (Abarbanell & Li, Reference Abarbanell and Li2021; Clark, Reference Clark1973; Rigal, Reference Rigal1994, Reference Rigal1996; Sümer, Reference Sümer2015; Sümer et al., Reference Sümer, Perniss, Zwitserlood, Özyürek, Bello, Guarini, McShane and Scassellati2014).

It is possible to attribute the differences between children and adults to the development of pragmatic knowledge required to provide informative descriptions. One way to investigate whether children and adults differ due to differences in pragmatic knowledge (as opposed to ease of encoding with iconic expressions) is to observe differences in contrast versus non-contrast trials. However, participants did not change the way they describe when there is a contrast or a non-contrast displays. Another way would be to have a non-confederate addressee and investigate various description/selection instances. For example, investigating the possible changes in the description patterns of the participants upon incorrect picture selection by addressee could reveal important insights for the development of pragmatic knowledge of children when compared to adults. This can be investigated in future research.

Future directions

First of all, the current study focused on manual articulators in sign and co-speech gestures in order to maintain similarity to previous research on the development of encoding space. In addition to the manual articulators, head/torso movements and eye-gaze direction may provide important contributions to our understanding of the role of multimodal communication. We call for further research to establish systematical and conventional ways of integrating those aspects in investigating the development of multimodal communication.

Secondly, the current study did not directly assess participants’ actual knowledge of Left-Right terms which could be used to strengten our claims for modulating effect of visual modality on the acquisition of Left-Right language.

Finally, even though our findings in spatial language use suggest an advantage when using sign compared to using speech or speech-gesture combinations, this advantage seems to be limited to tasks where language is explicitly used within the task. We did not, for instance, find differences between visual-spatial working memory across signers and speakers. Similarly, we did not find differences in participants’ spatial memory accuracy when we asked them to remember the picture that had been described in different modalities (Karadöller et al., Reference Karadöller, Sümer and Özyürek2021, see also Karadöller, Reference Karadöller, Sümer and Özyürek2021 for a discussion). Together, these findings call for investigations in other indices of cognition, such as visual attention during a planning of a description where some research demonstrated variations based on modality (see Manhardt et al., Reference Manhardt, Brouwer and Özyürek2021).