Japanese pitch accent is phonemic, making it crucial for second-language learners to acquire. Building on theories of multimodal learning, the present study explored how auditory, visual and gestural training of Japanese pitch accent affected behavioral, neural and meta-cognitive aspects of pitch perception across two experiments. Experiment 1 used a between-subjects pre/posttest design to train native English speakers to perceive Japanese pitch accents in one of the following three conditions: (1) baseline (audio + flat notation), (2) pitch height notation (audio + notation mimicking pitch height) and (3) pitch height notation + a left-hand gesture (L-gesture) (to engage the contralateral right hemisphere specialized for suprasegmental pitch processing). Our results indicated that (2) pitch height notation training was most robust in its benefits, as participants in this condition improved on trained and novel words alike. Experiment 2 used a within-subjects design to extend Experiment 1 in three ways: adding a right-hand gesture (R-gesture) condition (to engage more segmental language areas in the left hemisphere), introducing a neural correlate of cognitive load (measured by EEG alpha and theta power) and performing a metacognitive subjective assessment of learning (e.g., ‘Which training did you find the most helpful?’). The results showed that although there were no differences among our four training conditions on learning outcomes or EEG power, participants made the most positive subjective evaluations about pitch height notation and R-gesture training. Together, the results suggest that there may be a ‘just right’ amount of multimodal instruction to boost learning and increase engagement during foreign language pitch instruction.

This study aims to explore the target concepts of metonymical and metaphorical uses of ‘head’ in Jordanian Arabic (JA) compared to those used in Tunisian Arabic (TA). Extended conceptual metaphor theory (ECMT) as envisaged by Kövecses (2020, Review of Cognitive Linguistics, 18, 112–-130) is adopted as the theoretical framework. Data analysis reveals that through metonymic metaphors, the head in JA is used to profile character traits, mental faculty, cultural values and emotions. The head in JA is also capitalized upon to provide explanations of several daily life experiences. The primacy of head in JA was clear in the informants’ comprehension of the means by which embodiment provides the grounding for cognition, perception and language, which supports Gibbs’ (2014, The Bloomsbury companion to cognitive linguistics, pp. 167–184) ‘embodied metaphorical imagination’. Similarities in the cultural model of head between the two dialects were found, yet differences were also detected. In contrast to TA, the head is more productive in JA in profiling character traits and emotions. These differences were attributed to the existence of a cultural filter that has the ability to function between two cultures that belong to one matrix Arab culture and differences in experiential focus between the two examined speech communities.

This study addresses the importance of orofacial gestures and acoustic cues to execute prosodic patterns under different communicative settings in Farsi. Given that Farsi lacks morpho-syntactic markers for polar questions, we aim to determine whether specific facial movements accompany the prosodic correlates of questionhood in Farsi under conditions of degraded information, that is, whispering and wearing face masks. We hypothesise speakers will employ the most pronounced facial expressions when whispering questions with a face mask to compensate for the absence of F0, reduced intensity and lower face invisibility. To this end, we conducted an experiment with 10 Persian speakers producing 10 pairs of statements and questions in normal and whispered speech modes with and without face masks. Our results provide support to our hypotheses that speakers will intensify their orofacial expressions when confronted with marked conditions. We interpreted our results in terms of the ‘hand in hand’ and ‘trade-off’ hypothesis. In whispered speech, the parallel realisation of longer word duration and orofacial expressions may be a compensatory mechanism for the limited options to convey intonation. Also, the lower face coverage is mutually compensated for by word duration and intensified upper facial expressions, all of which in turn support the trade-off hypothesis.

Prosody includes the pitch, timing and loudness in speech, which can convey meaning and emotion. This study examines whether prosodic categories affect novel noun learning and whether the referent characteristic influence learning. Previous research showed that emotional prosody interfered with adults’ noun learning (West et al., 2017), but it had no effect on children (West et al., 2022). However, these researchers varied their method across ages, including animacy and complexity of the referent, and it is unclear if the results extend beyond the three emotional prosodies tested. Participants in the current set of studies heard novel words presented in five prosodic categories (within-subject) in order to learn the label for either animate or inanimate objects (between-subject). Study 1 compared inanimate objects and aliens, with better noun learning performance for inanimate objects. Study 2 compared inanimate objects with the same objects with faces added, but there was no difference in noun learning by object type. Both studies showed differences in noun learning by the prosodic category, with warning less accurate than naming. These results demonstrate how extralinguistic factors like prosody, attention and referent complexity influence noun learning.

An important quality to assess in others is their cooperativeness. We hypothesized that people use linguistic markers in their partners’ speech as a proxy of their cooperativeness in other tasks: specifically, we predicted that participants would prefer syntactically similar conversation partners as cooperation partners in a monetary game. We found that, indeed, participants preferably selected syntactically similar conversation partners as cooperation partners, but only when the participants could communicate using their naturally preferred constructions. In contrast, when participants were forced to communicate using dispreferred constructions, they rather cooperated with those partners that matched their natural preference than with those that matched their overt linguistic use. This pattern of results was likely driven by participants valuing representational alignment (i.e., being aligned on both linguistic features and their mental representations) more than incidental behavioral alignment (i.e., superficial convergence on similar linguistic features during interaction). This is because representational alignment is a potential indicator of group membership and may be associated with in-group benefits such as reputation, reciprocity and normative behavior. Those benefits may outweigh the benefits of simple behavioral alignment, which could be a potential indicator of others’ willingness to cooperate. This has important implications for communication in intercultural settings where members of diverse linguistic groups negotiate cooperative actions.

Movement scientists have proposed to ground the relation between prosody and gesture in ‘vocal-entangled gestures’, defined as biomechanical linkages between upper limb movement and the respiratory–vocal system. Focusing on spoken language negation, this article identifies an acoustic profile with which gesture is plausibly entangled, specifically linking the articulatory behaviour of onset consonant lengthening with forelimb gesture preparation and facial deformation. This phenomenon was discovered in a video corpus of accented negative utterances from English-language televised dialogues. Eight target examples were selected and examined using visualization software to analyse the correspondence of gesture phase structures (preparation, stroke, holds) with the negation word’s acoustic signal (duration, pitch and intensity). The results show that as syllable–onset consonant lengthens (voiced alveolar /n/ = 300 ms on average) with pitch and intensity increasing (e.g. ‘NNNNNNEVER’), the speaker’s humerus is rotating with palm pronating/adducing while his or her face is distorting. Different facial distortions, furthermore, were found to be entangled with different post-onset phonetic profiles (e.g. vowel rounding). These findings illustrate whole-bodily dynamics and multiscalarity as key theoretical proposals within ecological and enactive approaches to language. Bringing multimodal and entangled treatments of utterances into conversation has important implications for gesture studies.

Age of first exposure (AoFE) is an important factor that influences the quality of L2 acquisition. This study aims to investigate the AoFE effect on the contextual learning of L2 novel words at the neural level, as measured by the N400 component from event-related potentials (ERPs). Eighty-eight participants were recruited for the experiment of L2 pseudoword learning, which includes a learning session and a testing session. The participants’ EEG data were recorded from the testing session, and the N400 effect was derived from target words that were either congruous or incongruous with the context. The linear mixed model and multiple regression model revealed a positive AoFE effect on the N400 effect in discourses that were designed for testing retrieval of episodic and semantic memory even after accounting for the variance contributed by several confounding factors. In addition to AoFE, the effects of total L2 exposure, L2 proficiency and personality on the L2 novel word learning performance indicated by the N400 effect were also confirmed in the statistical results.

With a focus on politicians’ and medical experts’ gratitude expressions in UK government COVID-19 briefings, this research describes how perspective and intensity were modulated in expressing gratitude to realise different pragmatic intentions. This corpus-assisted analysis finds that retrospective or prospective gratitude expression was adopted by the two British elite groups to build solidarity (encouraging) and/or make requests (directing) for protecting public health. Gratitude of varying intensities was expressed (e.g. by highlighting metaphorical dimensions such as WIDTH and DEPTH) to correspond to the importance of a benefit (judged by how much the given benefit matches the receiver’s needs and preferences) and/or to implicitly display the evaluation of the benefactor’s responsibility and efforts. We tentatively formulate a dynamic model of gratitude expression in public discourse and shed light on the metaphorical conceptualisation of English gratitude expression and the power of gratitude expression in boosting social cohesion and directing social actions in a discourse of crisis.

This essay is the introduction to the Special Issue ‘Events in language and mind: Theoretical and empirical advances in the event integration theory’. We first review Leonard Talmy’s event integration theory in addition to some critiques of this framework. Following this, we point to some empirical research inspired by this framework, which explores the interaction between language and cognition. We then briefly introduce the papers in this volume and discuss their contributions to the event integration framework. We conclude with some limitations, questions and future directions.

This study investigated whether speakers use multimodal information (speech and gesture) to differentiate the physical and emotional meanings of the polysemous verb touch. We analyzed 302 hand gestures that co-occurred with this perception verb. For each case, we annotated (1) the meaning of touch (physical vs. emotional), (2) the gesture referent speakers physically touched (other-touch vs. self-touch), (3) the personal pronoun following the verb and (4) if they used intensifiers and negation. There were three main findings. First, we have seen that when speakers express the physical meaning, they are likely to reach an external referent (other-touch), but when they imply the emotional meaning, they tend to touch their own body (self-touch). Second, the most frequent co-speech gesture (chest-touching gesture) was associated with the emotional meaning, uncovering the metaphor the heart is container for emotions. Third, this study showed that the physical meaning of touch usually coexists with a wide variety of personal pronouns and negation words; in contrast, the emotional meaning of touch occurs primarily with the pronoun me and it is usually modified by intensifiers. Thus, speakers use both speech and gesture to differentiate the meanings of the polysemous verb touch.

A well-known method of studying iconic words is through the collection of subjective ratings. We collected such ratings regarding familiarity, iconicity, imagery/imageability, concreteness, sensory experience rating (SER), valence and arousal for Mandarin ABB words. This is a type of phrasal compound consisting of a prosaic syllable A and a reduplicated BB part, resulting in a vivid phrasal compound, for example, wù-mángmáng 雾茫茫 ‘completely foggy’. The correlations between the newly collected ABB ratings are contrasted with two other sets of prosaic word ratings, demonstrating that variables that characterize ABB words in an absolute sense may not play a distinctive role when contrasted with other types of words. Next, we provide another angle for looking at ABB words, by investigating to what degree rating data converges with corpus data. By far, the variable that characterizes ABB items consistently throughout these case studies is their high score for imageability, showing that they are indeed rightfully characterized as vivid. Methodologically, we show that it pays off to not take rating data at face value but to contrast it with other comparable datasets of a different phenomenon or data about the same phenomenon compiled in an ontologically different manner.

Research using masked priming and parafoveal preview techniques has shown that visual letter similarity has an impact on word processing during the initial stages in Latin-derived scripts. However, these effects appear to be absent in Arabic. One reason for this discrepancy could be attributed to the distinctive features of the Arabic script, which includes numerous letters sharing a basic form while varying in the location or number of diacritics. To shed light on this issue, the present study employed Arabic letters rather than words in two masked priming experiments: an alphabetic decision task and a letter-matching task. Both experiments showed that visually similar letters were more effective as primes than visually dissimilar letters. These findings suggest that the processes of letter identification in Arabic and Latin scripts may be roughly alike, implying that differences in visual letter similarity across scripts may arise at later stages of processing.

Distributional semantic representations were used to investigate crossmodal correspondences within language, offering a comprehensive analysis of how sensory experiences interconnect in linguistic constructs. By computing semantic proximity between words from different sensory modalities, a crossmodal semantic network was constructed, providing a general view of crossmodal correspondences in the English language. Community detection techniques were applied to unveil domains of experience where crossmodal correspondences were likely to manifest, while also considering the role of affective dimensions in shaping these domains. The study revealed the existence of an architecture of structured domains of experience in language, whereby crossmodal correspondences are deeply embedded. The present research highlights the roles of emotion and statistical associations in the organization of sensory concepts across modalities in language. The domains identified, including food, the body, the physical world and emotions/values, underscored the intricate interplay between the senses, emotion and semantic patterns. These findings align with the embodied lexicon hypothesis and the semantic coding hypothesis, emphasizing the capacity of language to capture and reflect crossmodal correspondences’ emotional and perceptual subtleties in the form of networks, while also revealing opportunities for further perceptual research on crossmodal correspondences and multisensory integration.

Reading official letters and being able to react to them appropriately is part of the daily life of every adult in many countries. Although the history of the plain language movement dates to the past century, it is only now that efforts are being made in Poland to adapt official documents to their audience. In this paper, we describe the results of a study (N = 685) in which we examined how particular linguistic forms and text structure affect the reception of an official-looking text – comprehension as well as emotions experienced, trust in the office, feelings of helplessness in the situation of reading the writing, and many other variables. Based on Imbir’s (2016, New Ideas in Psychology, 43, 39–49), two-system theory of emotions, we aim to examine how emotions – triggered by these complex stimuli – affect their processing. We hope that the results of our interdisciplinary research will answer the questions and hypotheses posed by researchers and find application in the reforms currently underway. We believe that by linking findings from cognitive psychology, the psychology of emotions and linguistics, we will also expand knowledge in the broad discussion on overcoming barriers between administrative offices and target audiences.

In this work, we use a mathematical model of the property listing task dynamics and test its ability to predict processing time in semantic and lexical decision tasks. The study aims at exploring the temporal dynamics of semantic access in these tasks and showing that the mathematical model captures essential aspects of semantic access, beyond the original task for which it was developed. In two studies using the semantic and lexical decision tasks, we used the mathematical model’s coefficients to predict reaction times. Results showed that the model was able to predict processing time in both tasks, accounting for an independent portion of the total variance, relative to variance predicted by traditional psycholinguistic variables (i.e., frequency, familiarity, concreteness imageability). Overall, this study provides evidence of the mathematical model’s validity and generality, and offers insights regarding the characterization of concrete and abstract words.

In this study, our objective was to explore the impact of hearing loss on the conceptual system underlying word meaning. We collected perceptual strength norms for 200 Italian words from early deaf individuals with limited or no access to auditory information and compared them to existing norms from hearing individuals. For each word, participants provided perceptual strength ratings for each perceptual modality. Our results revealed a significant reduction of the auditory modality in the norms provided by deaf individuals compared to the hearing population. However, we did not observe an overall decrease in reported perceptual strength. Interestingly, we found a heightened involvement of other sensory modalities accompanied by reduced modality exclusivity in the conceptualization of words, indicating that deaf individuals heavily rely on information coming from the other perceptual modalities to form concepts. These findings suggest that hearing loss leads to a reorganisation of word conceptualization, characterised by increased multisensoriality. Importantly, although diminished, the auditory modality remains present, suggesting that deaf individuals can still infer auditory-associated knowledge about words to some extent.

This paper presents an audiovisual perceptual analysis of the wh-question and wh-exclamation intonation in Brazilian Portuguese using auditory–visual congruent and incongruent stimuli, to investigate the relative importance of each modality in signaling pragmatic meanings. Ten Brazilian Portuguese speakers (five female) were filmed while producing both speech acts 10 times. Next, artificial stimuli were created: audio and visual cues were either matched (audio and video from the same speech act) or mismatched (audio and video from the different speech acts), resulting in 10 congruent and 10 incongruent stimuli of the wh-questions and the wh-exclamations. The perceptual experiment was taken by 36 Brazilians who identified the stimulus as a question or an exclamation. Results from the logistic regression showed that the factor ‘congruence’ was significant and had a significant interaction with ‘speakers’, which means that the congruent stimuli increased the comprehension of the Brazilian Portuguese wh-questions and wh-exclamations. In contrast, the incongruent stimuli tended to lower listeners’ identification, but to a degree depending on individual speakers’ strategies. Although variation in the accuracy of expressing both speech acts was also found across speakers, this study corroborates that the visual channel impacts the perceptual identification of the pragmatic intonation function of distinguishing sentence mode.

Embodied imagery hypothesis proposes the activation of perceptual-motor systems during language processing. Previous studies primarily used concrete visual stimuli to investigate mental imagery in language processing by native speakers (NSs) and second language (L2) learners, but few studies employed schematic diagrams. The study aims to investigate mental imagery in processing prepositional phrases by English NSs and L2 learners. Using image-schematic diagrams as primes, we examine whether any mental imagery effect is modulated by target preposition (over, in), the abstractness of meaning (spatial, extended), and stimulus onset asynchrony (SOA; 1,040 ms, 2,040 ms). A total of 79 adult L2 learners and 100 NSs of English completed diagram–picture matching and semantic priming phrasal decision tasks. Results revealed interference effects on L2 processing of over phrases and under 2,040 ms SOA, but no such effects were observed in the NS group. The selective interference effects in L2 suggest different mental imagery patterns between L1 and L2 processing, and processing schematic diagram primes requires high cognitive demands, potentially leading to difficulties in integrating visual and linguistic information and making grammaticality judgments. The findings partially validate schematic diagrams as visual representations of concepts and suggest the need for further examination of schematic diagrams with varying degrees of complexity.

A commonly held assumption is that demonstration and pantomime differ from ordinary action in that the movements are slowed down and exaggerated to be better understood by intended receivers. This claim has, however, been based on meagre empirical support. This article provides direct evidence that the different functional demands of demonstration and pantomime result in motion characteristics that differ from those for praxic action. In the experiment, participants were dressed in motion capture suits and asked to (1) perform an action, (2) demonstrate this action so that somebody else could learn how to perform it, (3) pantomime this action without using the object so that somebody else could learn how to perform it, and (4) pantomime this action without using the object so that somebody else could distinguish it from another action. The results confirm that actors slow down and exaggerate their movements in demonstrations and pantomimes when compared to ordinary actions.