Dans cette contribution, nous présentons la base CLAPI développée au laboratoire ICAR dans le contexte de l’évolution des bases de données de langues parlées en France au cours des trente dernières années. Nous détaillons les deux composantes de CLAPI, l’archive de corpus de langue parlée en interaction audio et vidéo enregistrés dans des situations sociales naturelles variées, et la plateforme d’outils.L’usage et l’apport de CLAPI sont illustrés par deux études. L’une décrit comment la base peut être utilisée pour des travaux (...) de linguistique interactionnelle intégrant la multimodalité (« oh là là ») ; l’autre concerne une recherche combinant données et métadonnées (« trop »). L’article est aussi l’occasion d’un bilan plus général. La mise en perspective montre en effet qu’après la période des questions est venue celle des dilemmes. La période des questions, choix et décisions à toutes sortes de niveaux a accompagné la mise en place des bases de données. L’expérience permet maintenant de mesurer leurs indéniables apports en termes non seulement de quantité de données disponibles (et traitables grâce aux outils), mais aussi de qualité (comme conséquence des exigences de standardisation liées au partage des données). La période des dilemmes nous conduit à nous interroger sur les meilleurs choix à opérer aujourd’hui dans les relations entre la poursuite des recherches sur des corpus variés (et parfois sensibles) et les exigences des bases de données ouvertes. (shrink)

The paper considers some intellectual roots of the contemporary multimodal analysis. The prehistory of microanalysis of social interaction includes semiotics of nonverbal communication and the anthropological study of patterns of expressive and communicative behavior as it was performed by Gregory Bateson and Margaret Mead in the pioneering research on Balinese character. The Batesonian approach to interaction was influenced by cybernetic ideas—particularly the notion of feedback— which led to theoretical advances on communication in general and, particularly, in the study of family (...) interaction in a psychiatric perspective. The interdisciplinary research project that became known as “The Natural History of an Interview” was the first example of a broad interdisciplinary collaboration involving anthropologists, linguists and psychiatrists. It has brought about a detailed analysis of a filmed interaction that correlated data about talk, paralanguage, and body language (kinesics). The results of this project, as well as of the other attempts at kinesic microanalysis, allowed one to question the goals of research and the adequacy of research methods employed. However, the input from kinesics and, broader, semiotics of non-verbal communication was instrumental for including data about participants’ attention traced as gaze-direction into analysis of talk-in-interaction, as it was put in practice by Charles Goodwin. (shrink)

The article investigates a multimodal practice for self-selecting observed in a video-taped corpus of work meetings: the use of pointing gestures predicting possible turn completions and projecting the emergence of possible next speakers. This practice is analyzed in various sequential positions, namely at turn beginnings and at pre-beginnings. It displays recipients' practical online turn parsing, and their orientation to transition spaces, and to TCU, completions in a visible, recognizable, public way. It shows the emergent and progressive establishment of speakership, exploiting (...) both systematic features of turn-taking and specific features of the interactional space. The article explores the emic, locally situated and contingent definition of speakership by considering not only where pointing gestures begin but also where they end: pointing does not only end in pre-completion positions, projecting turn end, but can also persist through sequences, showing speaker's orientations to her turn's sequential implicativeness. Thus, by observing a particular multimodal practice within a specific interactional setting, the article explores participants' orientations to the rights and obligations associated with talk-related categories such as `non current speaker', `possible next speaker', `incipient speaker' and `current speaker'. (shrink)

The study of so-called ‘disfluency’ phenomena (uh and um, filled and unfilled pauses, self-repairs and the like) has gained a lot of attention in various fields in linguistics in the past few decades, but a majority of studies tend to be production-oriented and often disregard fundamental aspects of face-to-face communication such as interactional dynamics and gesture. This paper presents a multimodal and multilevel model of “inter-fluency”, considering different levels of analysis, mainly, talk, gesture, and interaction, by combining different theoretical (...) frameworks and methodologies in gesture studies and interactional linguistics in order to bridge this gap and go beyond previous cognitive-oriented models. (shrink)

Studies of talk-and-bodily-conduct-in-interaction have inspired new insights into the way in which language, interaction and cognition might be articulated. More particularly, they have shown that participants mutually orient to the finely tuned multimodal details by which talk and action in interaction are sequentially organized. This article deals with this form of ‘participants’ multimodal online analysis’ by focusing on a particular phenomenon - the methodical practices and resources by which the end of a turn and of an activity phase (...) is projected and collectively achieved - in a specific videorecorded setting - a meeting in an architect’s office. It aims at questioning both how these local orientations are systematically displayed and exploited by the participants for the sequential organization of their activity and how they can be demonstrably observed by the analyst. (shrink)

Interviewing offers the biographer unique opportunities for gathering data. I offer three examples. The emphatic bacterial geneticist Norton Zinder confronted me with an interpretation of Barbara McClintock's science that was as surprising as it proved to be robust. The relaxed setting of the human geneticist Walter Nance's rural summer home contributed to an unusually improvisational oral history that produced insights into his experimental and thinking style. And "embedding" myself with the biochemical geneticist Charles Scriver in his home, workplace, and city (...) enabled me to experience the social networks that drive the practical events of his career, which in turn helped me explain the theoretical basis of his science. Face-to-face interaction and multisensory experience will shape each biographer's experience uniquely. Recent developments in sensory physiology suggest that the experience of integrating sense data encourages different patterns of observation and reflection. It is reasonable, then, to think that biography based on face-to-face interviews will, for a given author, have a different character than one based entirely on documents. I reflect on how interviewing shapes my own writing and I encourage the reader to do the same. (shrink)

This article starts with the environmental changes in human cognition, analyzes the virtual as the main feature of visual perception under digital technology, and explores the transition from passive to active human cognitive activities. With the diversified understanding of visual information, human contradiction of memory also began to become prominent. Aiming at the problem that the existing multimodal TV media recognition methods have low recognition rate of unknown application layer protocols, an adaptive clustering method for identifying unknown application layer protocols (...) is proposed. This method clusters application layer protocols based on similarity of the load characteristics of network stream application layer protocol data. The method divides the similarity calculation in the clustering algorithm to improve the clustering efficiency of the algorithm. Experimental results show that the proposed method can efficiently and accurately recognize unknown visual communication. This article proposes that, in the interactive multimodal visual information transmission, human visual perception experience has changed, the diversity of visual information content expression makes the aesthetic subject more personalized and stylized. (shrink)

The article analyzes the implementation of an online educational module and its impact on the organization of the classroom’s interaction order. The latter is institutionally constrained by the presence of a goal and the distribution of roles between teacher and students. The introduction of a digital learning platform adds a technological context to the institutional setting. The article considers technologies as possessing communicative affordances — opportunities for action made possible or delimited through their use. Technologies bring new interactive resources to (...) the process of ed­ucation and can affect the organization of the classroom’s interaction order. Using multimodal conversation analysis, we analyzed video recordings of the telemediated interaction of Russia-based students and teachers within a gamified online educational module. We investigate a case in which a student’s correct answer is nevertheless corrected by the teacher. We dem­onstrate that the teacher initiates the correction because they are guided by the ordering of the game elements within the interface. Based on a detailed analysis of the teacher’s mouse movement in relation to ongoing turns-at-talk, we show that this orientation is sustained by all participants. The work contributes to classroom interaction studies and affordance theory and develops the methodology of multimodal transcription for mediated contexts. The primary result of the study is an empirical demonstration that the relevance of technological affordances for interactants is situation­ally produced, and that this process is associated with the interweaving of the institutional and technical context of interaction. The conclusion discusses the relationship between affordances and institutional norms. (shrink)

This article analyses the sequential ordering of multi-modal discussions in real-time online classes in postgraduate education contexts. The article explores the ways that text and verbal talk are organized by the participants as inter-connecting modes of interaction. Focusing on Initiation, Response and Feedback sequences as an example of a form of exchange, the article shows that the interaction was comparatively disorderly where conducted across talk and text modes. For instance, written responses to questions or to encouragement turns often (...) overlapped with verbal answers from other students or with encouragement turns from the tutor. However, through the withholding of speech turns, the close latching of speech to written talk, and the ceasing of written turns at points where the topics or speakers changed, the participants showed a concern with re-establishing sequential order across the modalities. (shrink)

This article examines the interaction between grammar and multimodal resources by analyzing reported speech in Korean multiparty face-to-face interaction. The operation of two relevancy rules — minimization and recognition in interaction — is examined together with how the absence or presence of grammar is complemented by multimodal resources of various sorts. For the analysis, three categories are posited depending on who the quoted character is in the talk. In quoting oneself or co-participants in the talk, syntactic resources, prosody, (...) sequential organization and embodied participant frameworks collaborate in the interactional management of the quote without overt realization of grammatical resources traditionally proposed for building quotations in Korean. The multimodal character of quoted speech is most weakly realized when quoting a non-present character, not only because of the limitation in body and spatial resources available in the local context to recover various referential terms, but more importantly, because the interactional project that the parties are engaged in makes a distancing device such as the quotative particle -ko relevant. This study provides evidence for the intricate interaction between multimodal resources and grammar through investigation of an important realm of human interaction, that of reported speech. (shrink)

This article aims to describe the construction and annotation of a multimodal and multilingual (French, English, and Mandarin Chinese) corpus for the study of second language acquisition and professional development in language teaching. The corpus was built within a research project whose main objective is to determine which techno-semio-pedagogical competencies are developed informally and which ones require formal training. In this paper, we explain the theoretical framework adopted, characterized by an ecological approach to the interactive environment. We subsequently illustrate the (...) procedures for the collection of audio, video and eye-tracking data. We also provide details of the annotation of raw data to produce a corpus of analysis drawing on different automatic and semi-automatic annotation tools. Finally, we explain how such a corpus enables us to study the development of learners’ multimodal competence and tutors’ techno-pedagogical competence and point to some other possible research questions. (shrink)

The analysis of video-recorded interaction consists of various professionalized ways of seeing participant behavior through multimodal, co-operative, or intercorporeal lenses. While these perspectives are often adopted simultaneously, each creates a different view of the human body and interaction. Moreover, microanalysis is often produced through local practices of sense-making that involve the researchers’ bodies. It has not been fully elaborated by previous research how adopting these different ways of seeing human behavior influences both what is seen from a video and how (...) it is seen, as well as the way the interpretation of the data ultimately unfolds in the interaction between researchers. In this article, we provide a theoretical-methodological discussion of the microanalytic research process. We explore how it differs from “seeing” affect in interaction either as a co-operative and multimodal action or as an intercorporeal experience. First, we introduce the multimodal conversation analytic, co-operative, and intercorporeal approaches to microanalysis. Second, we apply and compare these practices to a video-recorded interaction of a romantic couple. Furthermore, we examine a video-recorded episode of us, the researchers, reflecting on our analytic observations about this interaction. We suggest that adopting a multimodal and co-operative perspective constructs affect as co-produced and displayed through observable action, while an intercorporeal perspective produces affect as an embodied and experienced phenomenon. While the former enables locating affect in a specific moment and identifiable body parts, the latter facilitates recognizing the experienced side of affect. These different modes of professional vision complement one another in capturing affect in interaction while being fundamentally used in local interactions between the researchers. (shrink)

This article investigates a multiparty interaction in an accounting office by applying a multimodal approach to discourse. This approach allows the incorporation of all relevant communicative modes and is based on the following three notions: 1) the notion of mediated action; 2) the notion of modal density; and 3) the notion of a foreground– background continuum of attention/awareness. The article illustrates that a social actor in a multiparty interaction simultaneously co-constructs several higher-level actions with the various participants on different levels (...) of their attention/awareness. On a theoretical level, the article argues that traditional approaches to discourse analysis, with their unconditional focus on language as the primary mode, misconstrue the multiple higher-level actions that a social actor is engaged in simultaneously as dyadic or triadic interactions in quick succession. (shrink)

Adult-infant early dyadic interactions have been extensively explored by developmental psychologists. Around the age of 2 months, infants already demonstrate complex, delicate and very sensitive behaviors that seem to express their ability to interact and share emotions with their caregivers. This paper presents 3 pilot studies of parent-infant dyadic interaction in various set-ups. The first two present longitudinal data collected on two infants aged between 1 and 6 months and their mothers. We analyzed the development of coordination between them, at (...) the motor and at the vocal level. The 3rd pilot study aims to explore interpersonal coordination in both vocal behavior and motor activity for one infant and his mother at 2, 4 and 6 months. These pilot studies however leave a number of questions open concerning developmental changes and infants’ progressive mastery of interaction. We identify areas worth examining and try to tease out specific issues that may help develop new methodological pathways for the study of early naturalistic social interaction. We assume that a continuous, rather than discrete, approach would better capture the changes taking place in the various communicative modalities, while also displaying each dyad’s specificity and the narrative dimension of social engagement between infants and caregivers. (shrink)

Recent studies of naturalistic face‐to‐face communication have demonstrated coordination patterns such as the temporal matching of verbal and non‐verbal behavior, which provides evidence for the proposal that verbal and non‐verbal communicative control derives from one system. In this study, we argue that the observed relationship between verbal and non‐verbal behaviors depends on the level of analysis. In a reanalysis of a corpus of naturalistic multimodal communication (Louwerse, Dale, Bard, & Jeuniaux, ), we focus on measuring the temporal patterns of specific (...) communicative behaviors in terms of their burstiness. We examined burstiness estimates across different roles of the speaker and different communicative modalities. We observed more burstiness for verbal versus non‐verbal channels, and for more versus less informative language subchannels. Using this new method for analyzing temporal patterns in communicative behaviors, we show that there is a complex relationship between verbal and non‐verbal channels. We propose a “temporal heterogeneity” hypothesis to explain how the language system adapts to the demands of dialog. (shrink)

Everyday media consumption leaves no doubt that the perception of moving images from various media is characterized by experience and understanding. Corresponding research in this field has shown that the stimulus patterns flooding in on us are not only processed mentally, but also bodily. Building on this, the following study argues that incoming stimuli are processed not only visually, but multimodally, with all senses, and moreover affectively. The classical binding of a sensory organ to a medium, on whose delimitation the (...) common understanding of multimodality is based, is thus abolished. Can arguments for this assumption be found? Building on the research of cultural anthropologist Ernst Cassirer and his successors, the two American philosophers Susanne K. Langer and John M. Krois, and supported by research from the life sciences in which, in parallel, the arts also play an important role in understanding image formation, as in the work of Heinz Werner and, following on from this, in the more recent research of the child psychologist Daniel N. Stern in collaboration with mirror neuron research by the group of neuroscientists around Giacomo Rizzolatti in Parma, justifications for this will be provided. What the multimodal processing of film images is concretely based on will finally be made clear by means of an exemplary analysis of a video film by the Swiss artist Pipilotti Rist. It aims to prove, that the prerequisites for the thesis can only lie in the fact, that both the modes of perception (visual, tactile, auditory, olfactory, gustatory) as well as the medially conditioned modes of image formation are based on non-discursive, formal-abstract structures that are also processed affectively. At the same time, the analysis thus shows how the selection and composition of the vital experienced forms can provide information about the socio-culturally shaped self-image of the producer and the viewer. (shrink)

Speech is a perceptuo-motor system. A natural computational modeling framework is provided by cognitive robotics, or more precisely speech robotics, which is also based on embodiment, multimodality, development, and interaction. This paper describes the bases of a virtual baby robot which consists in an articulatory model that integrates the non-uniform growth of the vocal tract, a set of sensors, and a learning model. The articulatory model delivers sagittal contour, lip shape and acoustic formants from seven input parameters that characterize the (...) configurations of the jaw, the tongue, the lips and the larynx. To simulate the growth of the vocal tract from birth to adulthood, a process modifies the longitudinal dimension of the vocal tract shape as a function of age. The auditory system of the robot comprises a “phasic” system for event detection over time, and a “tonic” system to track formants. The model of visual perception specifies the basic lips characteristics: height, width, area and protrusion. The orosensorial channel, which provides the tactile sensation on the lips, the tongue and the palate, is elaborated as a model for the prediction of tongue-palatal contacts from articulatory commands. Learning involves Bayesian programming, in which there are two phases: specification of the variables, decomposition of the joint distribution and identification of the free parameters through exploration of a learning set, and utilization which relies on questions about the joint distribution. Two studies were performed with this system. Each of them focused on one of the two basic mechanisms, which ought to be at work in the initial periods of speech acquisition, namely vocal exploration and vocal imitation. The first study attempted to assess infants’ motor skills before and at the beginning of canonical babbling. It used the model to infer the acoustic regions, the articulatory degrees of freedom and the vocal tract shapes that are the likeliest explored by actual infants according to their vocalizations. Subsequently, the aim was to simulate data reported in the literature on early vocal imitation, in order to test whether and how the robot was able to reproduce them and to gain some insights into the actual cognitive representations that might be involved in this behavior. Speech modeling in a robotics framework should contribute to a computational approach of sensori-motor interactions in speech communication, which seems crucial for future progress in the study of speech and language ontogeny and phylogeny. (shrink)

In this article we study how members of an interdisciplinary research team use an online forum for communicating about their research project. We use the concepts of "community of practice" and "connectivity" to consider the online interaction within a wider question of how people from different academic traditions "do" interdisciplinarity. The online forum for this Grey and Pleasant Land project did not take off as hoped, even after a series of interventions and amendments, and we consider what the barriers were (...) and how they might be overcome. Barriers to involvement included participants' expectations of interaction and collaboration--expectation that real interaction happens elsewhere, tensions between academic discourse and forum talk norms, unfamiliarity with the technology, and different conceptions of appropriate academic discussion. We suggest that common academic practice does not prepare us well for creating interdisciplinary research communities through online collaboration tools, whereas such tools are our best bet currently for including geographically dispersed members in collaborative projects. Therefore, careful planning and competence building would be necessary if such tools are to be used in collaborative research. Suggested interventions, based on our experience, include providing a more focused forum, making technical support easily available, and setting up particular tasks or items to debate, within a preset, synchronous timeframe, focusing on issues relevant to the project at that time. (shrink)

A growing number of children in the United States are exposed to multiple languages at home from birth. However, relatively little is known about the early process of word learning—how words are mapped to the referent in their child-centered learning experiences. The present study defined parental input operationally as the integrated and multimodal learning experiences as an infant engages with his/her parent in an interactive play session with objects. By using a head-mounted eye tracking device, we recorded visual scenes from (...) the infant’s point of view, along with the parent’s social input with respect to gaze, labeling, and actions of object handling. Fifty-one infants and toddlers from an English monolingual or a diverse bilingual household were recruited to observe the early multimodal learning experiences in an object play session. Despite that monolingual parents spoke more and labeled more frequently relative to bilingual parents, infants from both language groups benefit from a comparable amount of socially coordinated experiences where parents name the object while the object is looked at by the infant. Also, a sequential path analysis reveals multiple social coordinated pathways that facilitate infant object looking. Specifically, young children’s attention to the referent objects is directly influenced by parent’s object handling. These findings point to the new approach to early language input and how multimodal learning experiences are coordinated socially for young children growing up with monolingual and bilingual learning contexts. (shrink)

In this paper we compare grammatical inference in the context of simple and of mixed Lambek systems. Simple Lambek systems are obtained by taking the logic of residuation for a family of multiplicative connectives /, *, \, together with a package of structural postulates characterizing the resource management properties of the * connective. Different choices for Associativity and Commutativity yield the familiar logics NL, L, NLP, LP. Semantically, a simple Lambek system is a unimodal logic: the connectives get a Kripke (...) style interpretation in terms of a single ternary accessibility relation modeling the notion of linguistic composition for each individual system.The simple systems each have their virtues in linguistic analysis. But none of them in isolation provides a basis for a full theory of grammar. In the second part of the paper, we consider two types of mixed Lambek systems.The first type is obtained by combining a number of unimodal systems into one multimodal logic. The combined multimodal logic is set up in such a way that the individual resource management properties of the constituting logics are preserved. But the inferential capacity of the mixed logic is greater than the sum of its component parts through the addition of interaction postulates, together with the corresponding interpretive constraints on frames, regulating the communication between the component logics.The second type of mixed system is obtained by generalizing the residuation scheme for binary connectives to families of n-ary connectives, and by putting together families, of different arities in one logic. We focus on residuation for unary connectives, hence on mixed frames, as these already represent the complexities in full. We prove a number of elementary logical results for unary families of residuated connectives and their combination with binary families. The existing proposals for unary ‘structural modalities’ are situated within this general framework, and a number of new linguistic applications is given. (shrink)

This paper presents an analytical framework for analyzing how multimodal resources of emotion expression are semiotically materialized in discursive interactions specific to political discourse. Interested in how political personae are emotionally constructed through multimodal meaning-making practices, our analysis model assumes an interdisciplinary perspective, which integrates facial expression analysis – using FaceReader™ software –, the theory of emotional arcs and bodily actions (hand gestures) analysis that express emotions, in the analytical framework of multimodality. The results show how the multimodal choices that (...) political actors make during discursive interactions allow them to build their political brand and make connections with the audience on an emotional level. (shrink)

A second-person perspective in neuroscience is particularly appropriate for the study of communication. We describe how the investigation of joint language tasks can contribute to our understanding of the mechanisms underlying interaction.

This paper addresses the question how to analyze multimodal public discourse in such a way that the resulting reconstruction of the rhetor’s accountability either obliges the rhetor to acknowledge the argumentative reconstruction as valid or to refute its validity in a meta-discussion. This is a challenge for discourse theory as well as for argument theory because multimodal discourse seems far removed from the ‘standard’ propositional format of an argument. We argue that multimodal discourse should be analyzed as a coherent and (...) relevant discourse, assuming the possibility of instant interactions between all modes. We introduce a method that allows us to account for an argumentative reconstruction in a systematic way. We illustrate our method by analyzing the ABC news item titled Hu Jintao Visit: Economics and Panda Bears of January 20, 2011, holding ABC news as a rhetor accountable for several far reaching standpoints that are implied in the multimodal format. (shrink)

Our brains make up a series of signs and are engaged in making or manifesting or reacting to a series of signs: through this semiotic activity they are at the same time engaged in “being minds” and so in thinking intelligently. An important effect of this semiotic activity of brains is a continuous process of “externalization of the mind” that exhibits a new cognitive perspective on the mechanisms underling the semiotic emergence of abductive processes of meaning formation. To illustrate this (...) process I will take advantage of the analysis of some aspects of the cognitive interplay between internal and external representations. I consider this interplay critical in analyzing the relation between meaningful semiotic internal resources and devices and their dynamical interactions with the externalized semiotic materiality suitably stocked in the environment. Hence, minds are material, “extended” and artificial in themselves. A considerable part of human abductive thinking is occurring through an activity consisting in a kind of reification in the external environment and a subsequent re-projection and reinterpretation through new configurations of neural networks and chemical processes. I also illustrate how this activity takes advantage of hybrid representations and how it can nicely account for various processes of creative and selective abduction, bringing up the question of how multimodal aspects involving a full range of sensory modalities are important in hypothetical reasoning. (shrink)

The learning style myth is a commonly held myth that matching instruction to a student's “learning style” will result in improved learning, while providing mismatched instruction will result in suboptimal learning. The present study used a short online reasoning exercise about the efficacy of multimodal instruction to investigate the nature of learning styles beliefs. We aimed to: understand how learning style beliefs interact with beliefs about multimodal learning; characterize the potential complexity of learning style beliefs and understand how this short (...) exercise might influence endorsements of learning styles. Many participants who believed in the learning style myth supported the efficacy of multimodal learning, and many were willing to revise their belief in the myth after the exercise. Personal experiences and worldviews were commonly cited as reasons for maintaining beliefs in learning styles. Findings reveal the complexity of learning style beliefs, and how they interact with evidence in previously undocumented ways. (shrink)

The quality of the animal-human relationship and, consequently, the welfare of animals can be improved by gentle interactions such as stroking and talking. The perception of different stimuli during these interactions likely plays a key role in their emotional experience, but studies are scarce. During experiments, the standardization of verbal stimuli could be increased by using a recording. However, the use of a playback might influence the perception differently than ‘live’ talking, which is closer to on-farm practice. Thus, we compared (...) heifers’ (n = 28) reactions to stroking while an experimenter was talking soothingly (‘live’) or while a recording of the experimenter talking soothingly was played (‘playback’). Each animal was tested three times per condition and each trial comprised three phases: pre-stimulus, stimulus (stroking and talking) and post-stimulus. In both conditions, similar phrases with positive content were spoken calmly, using long low-pitched vowels. All tests were video recorded and analyzed for behaviors associated with different affective states. Effects on the heifers’ cardiac parameters were assessed using analysis of heart rate variability. Independently of the auditory stimuli, longer durations of neck stretching occurred during stroking, supporting our hypothesis of a positive perception of stroking. Observation of ear positions revealed longer durations of the back up position and less ear flicking and changes of ear positions during stroking. The predicted decrease in HR during stroking was not confirmed; instead we found a slightly increased mean HR during stroking with a subsequent decrease in HR, which was stronger after stroking with live talking. In combination with differences in HRV parameters, our findings suggest that live talking might have been more pleasurable to the animals and had a stronger relaxing effect than ‘playback’. The results regarding the effects of the degree of standardization of the stimulus on the variability of the data were inconclusive. We thus conclude that the use of recorded auditory stimuli to promote positive affective states during human-animal interactions in experimental settings is possible, but not necessarily preferable. (shrink)

A persistent controversy in language evolution research has been whether language emerged in the gestural-visual or in the vocal-auditory modality. A “dialectic” solution to this age-old debate has now been gaining ground: language was fully multimodal from the start and remains so to this day. In this paper, we show this solution to be too simplistic and outline a more specific theoretical proposal, which we designate as pantomime-first. To decide between the multimodal-first and pantomime-first alternatives, we review several lines of (...) interdisciplinary evidence and complement it with a cognitive-semiotic experiment. In the study, the participants saw – and then matched to hand-drawn images – recordings of short transitive events enacted by 4 actors in two conditions: visual and multimodal. Significantly, the matching accuracy was greater in the visual than the multimodal condition, though a follow-up experiment revealed that the emotional profiles of the events enacted in the multimodal condition could be reliably detected from the sound alone. We see these results as supporting the proposed pantomime-first scenario. (shrink)

Irregular self-selection is a demonstration of active involvement in interaction. English as a foreign language learners’ talk-in-interaction is one of such cases. Yet, little research has explored when, how, and why learners implement this action. The aim of this article is to address these issues in Chinese postgraduate EFL learners’ conversations from the perspective of multimodal interaction. To this end, we provide descriptive statistics and use multimodal conversation analysis to investigate the detailed process of irregular self-selection. The results show (...) the interactional sensitivity of learners, and all the successful irregular self-selection can be divided into the following three types: turn interruption, turn competition TC), and turn holding abortion. Learners implement this action by using multimodal resources, including lexis, syntax, pitch reset, intensity enhancement, gaze, and so forth. However, their body movements lack diversity, causing behaviors to be constrained and inactive. The main purpose of irregular self-selection is to provide knowledge that contributes to topical development. This study reveals that Chinese postgraduate EFL learners are interactionally competent members. They are able to achieve communicative goals but in a low diversity of body movements. The findings help to understand the detailed process of speakership claiming in EFL learners’ conversations. (shrink)

Investigations of iconicity in language, whereby interactants coordinate meaningful bodily actions to create resemblances, are prevalent across the human communication sciences. However, when it comes to analysing and comparing iconicity across different interactions and modes of communication, it is not always clear we are looking at the same thing. For example, tokens of spoken ideophones and manual depicting actions may both be analysed as iconic forms. Yet spoken ideophones may signal depictive and descriptive qualities via speech, while manual actions may (...) signal depictive, descriptive, and indexical qualities via the shape, movement, and placement of the hands in space. Furthermore, each may co-occur with other semiotics articulated with the face, hands, and body within composite utterances. The paradigm of iconicity as a single property is too broad and coarse for comparative semiotics, as important details necessary for understanding the range of human communicative potentialities may be masked. Here, we draw on semiotic approaches to language and communication, including the model of language as signalled via describing, indicating and/or depicting and the notion of non-referential indexicality, to illustrate the multidimensionality of iconicity in co-present interactions. This builds on our earlier proposal for analysing how different methods of semiotic signalling are combined in multimodal language use. We discuss some implications for the language and communication sciences and explain how this approach may inform a theory of biosemiotics. (shrink)

This paper focuses on corporate social responsibilities to employees, one key stakeholder for each firm. In particular, the views and attitudes of managers and entrepreneurs with respect to various social aspects related to their employees are investigated. The context of this research, Romania, a postcommunist country in Eastern Europe, allows us to look for dissimilarities between the talk of local firms and MNCs or foreign-based companies. The analysis is based on qualitative research and adopts an interpretative approach.The articles of (...) the weekly Romanian Capital magazine are investigated from an eight-year longitudinal perspective. (shrink)

The conceptual metaphor “THOUGHT IS FOOD” is exemplified in many verbal expressions. Nevertheless, how food metaphors are realized through the actual dining experience remains unexplored. Based on a food design event called EATAIPEI that took place in the London Design Festival in 2015, one aimed at promoting Taipei as World Design Capital 2016, this article analyzes how the multimodal metaphors of food were creatively represented and elaborated within it. This study proposes an analytical framework that combines insights from cognitive linguistics (...) with an intercultural performative view of food presentation and food metaphor. More specifically, it treats complex metaphors as resulting from the interactions of primary metaphors, proposition schemas and intercultural performance. The multimodal metaphors of food are complex metaphors composed of combined sets of primary metaphors, enriched by schema propositions and embedded within food performance. The experience of eating in the event reactivates the sensorimotor experiences that underpin the complex metaphors. (shrink)

Since Montague’s work it is well known that treating a single modality as a predicate may lead to paradox. In their paper “No Future”, Horsten and Leitgeb show that if the two temporal modalities are treated as predicates paradox might arise as well. In our paper we investigate whether paradoxes of multiple modalities, such as the No Future paradox, are genuinely new paradoxes or whether they “reduce” to the paradoxes of single modalities. In order to address this question we develop (...) a notion of reducibility based on a version of Smoryński Diagonalized Operator Logic. We show that there are reducible multimodal paradoxes as well as irreducible paradoxes of interaction. In particular, we show the No Future paradox to be an irreducible paradox according to our notion of reducibility. (shrink)

One of the recent areas of discussion in aesthetics and the visual arts is the tension between the so-called “ocularcentric” tradition, on the one hand, and the tendency to move in a multisensory, multimodal direction, on the other. My aim in this paper is to bring out this tension by tracing it in a number of moments; firstly, in the late 19th-early 20th century discussion, concerning the “total art work” and the contribution of synaesthesia; secondly, the reaction to what Clement (...) Greenberg called the modernist “reduction to the visual”, expressed in a variety of multimodal art forms in the latter half of the 20th century. The weight of my paper is given to two contemporary discussions with significant repercussions in aesthetics: a. the opposition between “disembodiment” and “embodied immersion” in the area of new media; and b. the debate between modularity and plasticity of perception in neuroscience. The conclusion to which I am led is that the possibilities of sensory integration offered by new media and of sensory plasticity - offered by synaesthesia - suggest new creative ways in which our senses may interact and enhance our experience of the world, as well as elucidate artistic creativity and aesthetic appreciation. (shrink)

The framework of depicting put forward byClark (2016)offers a schematic vantage point from which to examine iconic language use. Confronting the framework with empirical data, we consider some of its key theoretical notions. Crucially, by reconceptualizing the typology of depictions, we identify an overlooked domain in the literature: “speech-embedded nonverbal depictions,” namely cases where meaning is communicated iconically, nonverbally, and without simultaneously co-occurring speech. In addition to contextualizing the phenomenon in relation to existing research, we demonstrate, with examples from American (...) TV talk shows, how such depictions function in real-life language use, offering a brief sketch of their complexities and arguing also for their theoretical significance. (shrink)

Literary narratives are well recognised for their power to foster engagement with complex social themes. Transmedia narratives, which present the same story in different media, can help advance both critical multimodal discourse studies and multiliteracies pedagogies. To harness this potential, we need to develop methods for systematically relating media affordances to discourse-semantic patterns and the broad social themes these patterns construct in narratives, and ensure these methods build on the knowledge learners bring to the classroom. This article introduces a social (...) semiotic method for analysing transmedia narratives and illustrates its ability to reflect the persuasive power of narrative and build on young children's critical multimodal awareness. This is achieved through a case study comprising data from three adaptations of the same award-winning narrative - animated film (Joyce, W. (Producer), Joyce, W., & Oldenburg, B. (Directors) (2011). The fantastic flying books of Mr Morris Lessmore [Motion Picture]. Moonbot Studios), interactive app (Moonbot Studios. (2011). The fantastic flying books of Mr Morris Lessmore (Version 1.4). [Mobile Application Software]), and traditional-format picture book (Joyce, W. (2012). The fantastic flying books of Mr Morris Lessmore. Illustrations by William Joyce and Joe Bluhm. Moonbot Books) - and mother-child interactions with the book and the app. (shrink)

This article focuses on how to provide environmental descriptions of the context with the intent of creating access to information and dialogical participation for deafblind persons. Multimodal interaction is needed to communicate with deafblind persons whose combined sensory loss impedes their access to the environment and ongoing interaction. Empirical data of interpreting for deafblind persons are analyzed to give insight into how this task may be performed. All communicative activities vary due to their context, participants, and aim. In this study, (...) our data are part of a cross-linguistic study of tactile sign language and were gathered during a guided tour for a deafblind group. The guided tour was tailored to a specific group (adult deafblind tactile signers and their interpreters) visiting one of the oldest cathedrals and pilgrim sites in Scandinavia, with interpreters following up the guide’s presentation and providing descriptions based on the given situation. The tour and the interpreters’ work were videotaped, and the ongoing interaction and communication have been studied through video-ethnographic methods and conversational analysis. The data have been investigated for the research question: What elements are involved in descriptions to provide deafblind individuals access to their environments? Theories from multimodality communicative studies are relevant for the ways tactile descriptions are presented and analyzed. Some of this is an investigation at a microlevel of interaction. An overall inspiration for this study is interaction studies with data from authentic formal and informal conversations and ways of analyzing embodied action and situated gestures in studies of human interaction. Also, concepts of “frontstage,” “backstage,” and “main conversation” are brought into our interpreter-mediated data to follow the role of building meaning in complex conversations. Theories on interaction are used in the analyses to illustrate the participating framework between the guide, the interpreter, the deafblind person, and the situated frame of their interaction. The study opens for a broader understanding of the repertoire of multimodal interaction and how such interaction may be handled as inputs in communication processes. This is of relevance for communication with deafblind persons, for professionals meeting blind and deafblind clients, and for knowledge of multimodal interaction in general. (shrink)

This paper contributes to the Business Ethics literature by unpacking the multimodal construction of moral narratives in popular culture and its portrayals of organizations and organizational roles. Understanding such portrayals and their construction is crucial to Business Ethics scholarship because they shape organizational imaginaries, influencing understandings and expectations of the ethical/moral responsibilities of organizations and the actors within them. In particular, we study the construction of moral narratives within a reality TV show that focuses on immigration and border control at (...) an airport. We find that the immigration officers are depicted as rational and heroic figures whilst the travellers are presented as emotional and potentially dangerous characters. Our analysis highlights how this is achieved via five multimodal editing dimensions—the structure of interactional scenes, the ability to address the camera, the narrator’s comments, the visual and music effects—that are key in constructing clearly defined personae. We show how, through the intersubjective construction of clear-cut characters, the show downplays the moral complexity of its content. Portraying immigration officers as heroic, while presenting travellers as potentially dangerous, allows for a silencing of any ethical questioning of immigration officers’ organizational practices. (shrink)

Even with obvious deficiencies, large prompt-commanded multimodal models are proving to be flexible cognitive tools representing an unprecedented generality. But the directness, diversity, and degree of user interaction create a distinctive “human-centred generality” (HCG), rather than a fully autonomous one. HCG implies that —for a specific user— a system is only as general as it is effective for the user’s relevant tasks and their prevalent ways of prompting. A human-centred evaluation of general-purpose AI systems therefore needs to reflect the personal (...) nature of interaction, tasks and cognition. We argue that the best way to understand these systems is as highly-coupled cognitive extenders, and to analyse the bidirectional cognitive adaptations between them and humans. In this paper, we give a formulation of HCG, as well as a high-level overview of the elements and trade-offs involved in the prompting process. We end the paper by outlining some essential research questions and suggestions for improving evaluation practices, which we envision as characteristic for the evaluation of general artificial intelligence in the future. (shrink)

With the spread of new media, traditional media, such as TV faced a problem: how to attract and retain the audience and how to offer something new, that competitors do not have. And for a long time now even well-known and influential mass media have been using interactive content. The statement is that interactive content just for fun is no longer perceived. Interactive content includes quizzes, puzzles, crosswords, various polls, games, tests, quests, memories, interactive graphics, flash games, etc. Interactive elements (...) in the media are used both separately and as a part of journalistic material. For the research of using interactive content in media, we examined the official sites of 30 channels in Ukraine, which belong to DTV and are translated for each region. In general, we have revealed 593 elements of interactive content on the sites of TV media. In the article, there were examined the kinds, types, themes, and amount of interactive content on each site of Ukrainian TV channels. It is investigated that such content can help to stay in touch with the audience and to popularize content that is broadcasted on the TV channel. Interactive media content has several obvious benefits, including the influx of new readers and, consequently, the growth of unique visitors, the growing number of people who talk about your media on social media and in real life, and the opportunity to draw attention to your television projects. (shrink)

This volume brings together the advanced research results obtained by the European COST Action 2102: “Cross Modal Analysis of Verbal and Nonverbal Communication”. The research published in this book was discussed at the 3rd joint EUCOGII-COST 2102 International Training School entitled “Toward Autonomous, Adaptive, and Context-Aware Multimodal Interfaces: Theoretical and Practical Issues ” and held in Caserta, Italy, on March 15-19, 2010.

Early number skills represent critical milestones in children's cognitive development and are shaped over years of interacting with quantities and numerals in various contexts. Several connectionist computational models have attempted to emulate how certain number concepts may be learned, represented, and processed in the brain. However, these models mainly used highly simplified inputs and focused on limited tasks. We expand on previous work in two directions: First, we train a model end-to-end on video demonstrations in a synthetic environment with multimodal (...) visual and language inputs. Second, we use a more holistic dataset of 35 tasks, covering enumeration, set comparisons, symbolic digits, and seriation. The order in which the model acquires tasks reflects input length and variability, and the resulting trajectories mostly fit with findings from educational psychology. The trained model also displays symbolic and non-symbolic size and distance effects. Using techniques from interpretability research, we investigate how our attention-based model integrates cross-modal representations and binds them into context-specific associative networks to solve different tasks. We compare models trained with and without symbolic inputs and find that the purely non-symbolic model employs more processing-intensive strategies to determine set size. (shrink)

This book presents one of the first attempts at developing a precise, grammatically rooted, theory of conversation motivated by data from real conversations. The theory has descriptive reach from the micro-conversational - e.g. self-repair at the word level - to macro-level phenomena such as multi-party conversation and the characterization of distinct conversational genres. It draws on extensive corpus studies of the British National Corpus, on evidence from language acquisition, and on computer simulations of language evolution. The theory provides accounts of (...) the opening, middle game, and closing stages of conversation. it also offers a new perspective on traditional semantic concerns such as quanitifcation and anaphora. The Interactive Stance challenges orthodox views of grammar by aruging that, unless we wish to excluse from analysis a large body of frequently occurrring words and constructions, the right way to construe grammar is as a system that characterizes types of talk in interaction. (shrink)

Emotion recognition is an important part of building an intelligent human-computer interaction system and plays an important role in human-computer interaction. Often, people express their feelings through a variety of symbols, such as words and facial expressions. A business ecosystem is an economic community based on interacting organizations and individuals. Over time, they develop their capabilities and roles together and tend to develop themselves in the direction of one or more central enterprises. This paper aims to study a multimodal ER (...) method based on attention mechanism. It analyzes the current emotional state of consumers and the development direction of enterprises through multi-modal ER of human emotions and analysis of market trends, so as to provide the most appropriate response or plan. This paper firstly describes the related methods of multimodal ER and deep learning in detail, and briefly outlines the meaning of enterprise strategy in the business ecosystem. Then, two datasets, CMU-MOSI and CMU-MOSEI, are selected to design the scheme for multimodal ER based on self-attention mechanism. Through the comparative analysis of the accuracy of single-modal and multi-modal ER, the self-attention mechanism is applied in the experiment. The experimental results show that the average recognition accuracy of happy under multimodal ER reaches 91.5%. (shrink)

As humans interact in the world, they often orient one another's attention to objects through the use of spoken demonstrative expressions and head and/or hand movements to point to the objects. Although indicating behaviors have frequently been studied in lab settings, we know surprisingly little about how demonstratives and pointing are used to coordinate attention in large-scale space and in natural contexts. This study investigates how speakers of Quiahije Chatino, an indigenous language of Mexico, use demonstratives and pointing to give (...) directions to named places in large-scale space across multiple scales. The results show that the use and coordination of demonstratives and pointing change as the scale of search space for the target grows. At larger scales, demonstratives and pointing are more likely to occur together, and the two signals appear to manage different aspects of the search for the target: demonstratives orient attention primarily to the gesturing body, while pointing provides cues for narrowing the search space. These findings underscore the distinct contributions of speech and gesture to the linguistic composite, while illustrating the dynamic nature of their interplay.s in Spanish and Quiahije Chatino are provided as appendices.Se incluyen como apéndices resúmenes en español y en el chatino de San Juan Quiahije. SonGktyiCreCinH, ngyaqCskaEktyiCnoEndaHsonBnaFngaJnoIngyaqCloEktyiCreC, ngyaqCranFchaqExlyaKqoEchaqFjnyaJnoAndywiqArenqAKchinAKyqyaC. (shrink)

Emotion recognition plays an important role in intelligent human–computer interaction, but the related research still faces the problems of low accuracy and subject dependence. In this paper, an open-source software toolbox called MindLink-Eumpy is developed to recognize emotions by integrating electroencephalogram and facial expression information. MindLink-Eumpy first applies a series of tools to automatically obtain physiological data from subjects and then analyzes the obtained facial expression data and EEG data, respectively, and finally fuses the two different signals at a decision (...) level. In the detection of facial expressions, the algorithm used by MindLink-Eumpy is a multitask convolutional neural network based on transfer learning technique. In the detection of EEG, MindLink-Eumpy provides two algorithms, including a subject-dependent model based on support vector machine and a subject-independent model based on long short-term memory network. In the decision-level fusion, weight enumerator and AdaBoost technique are applied to combine the predictions of SVM and CNN. We conducted two offline experiments on the Database for Emotion Analysis Using Physiological Signals dataset and the Multimodal Database for Affect Recognition and Implicit Tagging dataset, respectively, and conducted an online experiment on 15 healthy subjects. The results show that multimodal methods outperform single-modal methods in both offline and online experiments. In the subject-dependent condition, the multimodal method achieved an accuracy of 71.00% in the valence dimension and an accuracy of 72.14% in the arousal dimension. In the subject-independent condition, the LSTM-based method achieved an accuracy of 78.56% in the valence dimension and an accuracy of 77.22% in the arousal dimension. The feasibility and efficiency of MindLink-Eumpy for emotion recognition is thus demonstrated. (shrink)