Ana-Maria Pleşca, University of Gent

Bilingual Emotion and Deception through a Multimodal Lens

How do bilinguals differ in their linguistic and gestural expressions of emotion across their first (L1) and second (L2) languages? Extant literature on bilingual emotionality remains inconclusive. Some studies support the emotional distance theory (Harris, 2004; Dewaele & Pavlenko, 2002), suggesting that individuals experience emotional detachment when using  their second language. In contrast, other research finds no significant behavioural differences in response to emotional words across languages (Ferré et al., 2010), while a third line of evidence highlights how cognitive load associated with operating in a second or foreign language impacts behavioural cues of emotion (Özder, 2023). Existing research, while foundational to our understanding of bilingual emotionality, has often relied on experimental designs that overlook the multimodal nature of communication - an essential aspect of how emotions are naturally expressed. 

Building on this context, this study examines the (dis)similarities in linguistic and gestural cues of emotion in bilinguals across L1 and L2. In dyads, participants will engage in face-to-face, partially scripted conversations prompted by emotional (sad) and neutral video stimuli. We examine the effects of language (L1, L2) and emotion (non/emotional speech) through two research questions: (1) Are there differences in gestural behaviour and linguistic production across emotion conditions? (2) Does the interaction between language and emotion affect linguistic and gestural cues? 

To address these questions, we will analyze audio-video recordings, focusing on gestural behaviour and linguistic parameters such as F0, F0 range, and word use. Pilot results revealed significant interactions between bilinguals’ languages and emotions, which may suggest that emotions are embodied to a seemingly different extent across bilinguals' languages. 

Shifting focus to another related phenomenon, while data collection for the full experiment is pending, the remainder of this talk will concern deception production in bilinguals. Specifically, do bilinguals differ in their production of deception across their languages? Current approaches to deception processing will be discussed, as well as a proposed multimodal methodology for studying face-to-face, low-stakes deception.