Using a cross-modal semantic priming paradigm, both experiments of the present study investigated the link between the mental representations of iconic gestures and words. Two groups of the participants performed a primed lexical decision task where they had to discriminate between visually presented words and nonwords (e.g., flirp). Word targets (e.g., bird) were preceded by video clips depicting either semantically related (e.g., pair of hands flapping) or semantically unrelated (e.g., drawing a square with both hands) gestures. The duration of (...) gestures was on average 3,500 ms in Experiment 1 but only 1,000 ms in Experiment 2. Significant priming effects were observed in both experiments, with faster response latencies for related gesture–word pairs than unrelated pairs. These results are consistent with the idea of interactions between the gestural and lexical representational systems, such that mere exposure to iconic gestures facilitates the recognition of semantically related words. (shrink)

Iconic gesture-speech integration is a relatively recent field of investigation with numerous researchers studying its various aspects. The results obtained are just as diverse. The definition of iconic gestures is often overlooked in the interpretations of results. Furthermore, while most behavioral studies have demonstrated an advantage of bimodal presentation, brain activity studies show a diversity of results regarding the brain regions involved in the processing of this integration. Clinical studies also yield mixed results, some suggesting parallel processing (...) channels, others a unique and integrated channel. This review aims to draw attention to the methodological variations in research on iconic gesture-speech integration and how they impact conclusions regarding the underlying phenomena. It will also attempt to draw together the findings from other relevant research and suggest potential areas for further investigation in order to better understand processes at play during speech integration process. (shrink)

The economic principle of communication, according to which successful communication can be reached by least effort, has been studied for verbal communication. With respect to nonverbal behavior, it implies that forms of iconic gestures change over the course of communication and become reduced in the sense of less pronounced. These changes and their effects on learning are currently unexplored in relevant literature. Addressing this research gap, we conducted a word learning study to test the effects of changing gestures on (...) children’s slow mapping. We applied a within-subject design and tested 51 children, aged 6.7 years, who learned unknown words from a story. The storyteller acted on the basis of two conditions: In one condition, in which half of the target words were presented, the story presentation was enhanced with progressively reduced iconic gestures ; in the other condition, half of the target words were accompanied by fully executed iconic gestures. To ensure a reliable gesture presentation, children were exposed to a recorded person telling a story in both conditions. We tested the slow mapping effects on children’s productive and receptive word knowledge three minutes as well as two to three days after being presented the story. The results suggest that children’s production of the target words, but not their understanding thereof, was enhanced by PRG. (shrink)

We present a formal analysis of iconic coverbal gesture. Our model describes the incomplete meaning of gesture that’s derivable from its form, and the pragmatic reasoning that yields a more specific interpretation. Our formalism builds reported.

Visual information conveyed by iconic hand gestures and visible speech can enhance speech comprehension under adverse listening conditions for both native and non‐native listeners. However, how a listener allocates visual attention to these articulators during speech comprehension is unknown. We used eye‐tracking to investigate whether and how native and highly proficient non‐native listeners of Dutch allocated overt eye gaze to visible speech and gestures during clear and degraded speech comprehension. Participants watched video clips of an actress uttering a clear (...) or degraded (6‐band noise‐vocoded) action verb while performing a gesture or not, and were asked to indicate the word they heard in a cued‐recall task. Gestural enhancement was the largest (i.e., a relative reduction in reaction time cost) when speech was degraded for all listeners, but it was stronger for native listeners. Both native and non‐native listeners mostly gazed at the face during comprehension, but non‐native listeners gazed more often at gestures than native listeners. However, only native but not non‐native listeners' gaze allocation to gestures predicted gestural benefit during degraded speech comprehension. We conclude that non‐native listeners might gaze at gesture more as it might be more challenging for non‐native listeners to resolve the degraded auditory cues and couple those cues to phonological information that is conveyed by visible speech. This diminished phonological knowledge might hinder the use of semantic information that is conveyed by gestures for non‐native compared to native listeners. Our results demonstrate that the degree of language experience impacts overt visual attention to visual articulators, resulting in different visual benefits for native versus non‐native listeners. (shrink)

We examined whether children's ability to integrate speech and gesture follows the pattern of a broader developmental shift between 3- and 5-year-old children regarding the ability to process two pieces of information simultaneously. In Experiment 1, 3-year-olds, 5-year-olds, and adults were presented with either an iconic gesture or a spoken sentence or a combination of the two on a computer screen, and they were instructed to select a photograph that best matched the message. The 3-year-olds did not (...) integrate information in speech and gesture, but 5-year-olds and adults did. In Experiment 2, 3-year-old children were presented with the same speech and gesture as in Experiment 1 that were produced live by an experimenter. When presented live, 3-year-olds could integrate speech and gesture. We concluded that development of the integration ability is a part of the broader developmental shift; however, live-presentation facilitates the nascent integration ability in 3-year-olds. (shrink)

Speakers frequently perform representational gestures to depict concepts in an iconic fashion. For example, a speaker may hold her index finger and thumb apart to indicate the size of a matchstick. However, the process by which a physical handshape is mentally transformed into abstract spatial information is not well understood. We present a series of experiments that investigate how people decode the physical form of an articulator to derive imaginary geometrical constructs, which we call “gesture form.” We provide (...) quantitative evidence for several key properties that play a role in this process. First, “profiling,” the ability to focus on a structural subunit within the complex form of the physical hand. Second, “perspective,” for which we show that one and the same handshape seen from different perspectives can lead to different spatial interpretations. Third, “selectivity,” the fact that gestures focus on specific spatial features at the expense of others. Our results provide a first step toward mapping out the process of how representational gestures make the communication of spatial information possible. (shrink)

Studies of the evolution of language rely heavily on comparisons to nonhuman primates, particularly the gestural communication of nonhuman apes. Differences between human and ape gestures are largely ones of degree rather than kind. For example, while human gestures are more flexible, ape gestures are not inflexible. In this piece, I closely consider two features of the gestural communication of apes and humans that might display differences in kind: iconicity and temporal alignment. Iconicity has long played a privileged role in (...) theories of gestural language origins, the proposal being that it provided a steppingstone into language through a stage of pantomime. However, iconicity is not as easy as it seems. Evidence from co-speech gestures of hearing children and from homesign suggests that iconic reference is both cognitively complex and slow to develop in humans. There is no conclusive evidence that nonhuman apes understand or produce iconic gestures; some gestures may appear iconic to human observers but the apes themselves may not understand the similarity between form and meaning. Far from providing an easy pathway into the emergence of symbolic communication, iconicity relies on sophisticated capacities for analogy and abstraction, ones often lacking in apes and young children. Temporal alignment between gesture and vocalization is another area that may show sharp contrasts between adult humans and apes, though data here is sparser. I discuss the tight synchronization of gesture and speech that emerges over the first year of life in human children and contrast it with gesture and vocalization in apes, which typically are described as overlapping but not simultaneous. Human ancestors probably communicated in ways similar to other apes, but the dual emergence of the ability to use iconic reference and the alignment of hand and mouth may have set our ancestors down a unique evolutionary road toward language. (shrink)

We argue that the pictorial nature of certain constructions in signs and in gestures explains surprising properties of their syntax. In several sign languages, the standard word order (e.g. SVO) gets turned into SOV (with preverbal arguments) when the predicate is a classifier, a distinguished construction with highly iconic properties (e.g. Pavlič, 2016). In silent gestures, participants also prefer an SOV order in extensional constructions, irrespective of the word order of the language they speak (Goldin-Meadow et al., 2008). But (...) in silent gestures and in Brazilian Sign Language (Libras), intensional constructions can override these SOV preferences, yielding SVO instead (Schouwstra & de Swart, 2014; Napoli et al., 2017). This distinction was argued to be due to iconicity: arguments are expressed before the verb if they correspond to entities that are present before the action, otherwise they follow the verb. While agreeing with this intuition, we propose that the extensional/intensional distinction is neither empirically nor theoretically appropriate. In new data from American Sign Language, we replicate the distinction among extensional classifier predicates: for _x ate up the ball_, the ball is typically seen before the eating and a preposed object is preferred; but for _x spit out the ball_, the ball is typically seen after the spitting and a postposed object is preferred, although both _eat up_ and _spit out_ are used extensionally. We extend this finding to data involving pro-speech (= speech-replacing) gestures embedded in French sentences. We argue for a Visibility Generalization according to which arguments appear before the verb if their denotations are typically visible before the action, and we develop a new formal account within a pictorial semantics for visual animations (inspired by Greenberg and Abusch). It derives the observed word order preferences, it explains how the semantics of classifier predicates combines iconic and conventional properties, and it makes a more general point: sign language semantics combines logical semantics with pictorial semantics. (shrink)

Most language learners have difficulties acquiring the phonemes of a second language (L2). Unfortunately, they are often judged on their L2 pronunciation, and segmental inaccuracies contribute to miscommunication. Therefore, we aim to determine how to facilitate phoneme acquisition. Given the close relationship between speech and co-speech gesture, previous work unsurprisingly reports that gestures can benefit language acquisition, e.g., in (L2) word learning. However, gesture studies on L2 phoneme acquisition present contradictory results, implying that both specific properties of gestures (...) and phonemes used in training, and their combination, may be relevant. We investigated the effect of phoneme and gesture complexity on L2 phoneme acquisition. In a production study, Dutch natives received instruction on the pronunciation of two Spanish phonemes, /u/ and /θ/. Both are typically difficult to produce for Dutch natives because their orthographic representation differs between both languages. Moreover, /θ/ is considered more complex than /u/, since the Dutch phoneme inventory contains /u/ but not /θ/. The instruction participants received contained Spanish examples presented either via audio-only, audio-visually without gesture, audio-visually with a simple, pointing gesture, or audio-visually with a more complex, iconic gesture representing the relevant speech articulator(s). Preceding and following training, participants read aloud Spanish sentences containing the target phonemes. In a perception study, Spanish natives rated the target words from the production study on accentedness and comprehensibility. Our results show that combining gesture and speech in L2 phoneme training can lead to significant improvement in L2 phoneme production, but both gesture and phoneme complexity affect successful learning: Significant learning only occurred for the less complex phoneme /u/ after seeing the more complex iconic gesture, whereas for the more complex phoneme /θ/, seeing the more complex gesture actually hindered acquisition. The perception results confirm the production findings and show that items containing /θ/ produced after receiving training with a less complex pointing gesture are considered less foreign-accented and more easily comprehensible as compared to the same items after audio-only training. This shows that gesture can facilitate task performance in L2 phonology acquisition, yet complexity affects whether certain gestures work better for certain phonemes than others. (shrink)

The aim of this article is to mount a challenge to gesture-first hypotheses about the evolution of language by identifying constraints on the emergence of symbol use. Current debates focus on a range of pre-conditions for the emergence of language, including co-operation and related mentalising capacities, imitation and tool use, episodic memory, and vocal physiology, but little specifically on the ability to learn and understand symbols. It is argued here that such a focus raises new questions about the plausibility (...) of gesture-first hypotheses, and so about the evolution of language in general. After a brief review of the methodology used in the article, it is argued that existing uses of gesture in hominid communities may have prohibited the emergence of symbol use, rather than ‘bootstrapped’ symbolic capacities as is usually assumed, and that the vocal channel offers other advantages in both learning and using language. In this case, the vocal channel offers a more promising platform for the evolution of language than is often assumed. (shrink)

People with aphasia use gestures not only to communicate relevant content but also to compensate for their verbal limitations. The Sketch Model (De Ruiter, 2000) assumes a flexible relationship between gesture and speech with the possibility of a compensatory use of the two modalities. In the successor of the Sketch Model, the AR‐Sketch Model (De Ruiter, 2017), the relationship between iconic gestures and speech is no longer assumed to be flexible and compensatory, but instead iconic gestures are (...) assumed to express information that is redundant to speech. In this study, we evaluated the contradictory predictions of the Sketch Model and the AR‐Sketch Model using data collected from people with aphasia as well as a group of people without language impairment. We only found compensatory use of gesture in the people with aphasia, whereas the people without language impairments made very little compensatory use of gestures. Hence, the people with aphasia gestured according to the prediction of the Sketch Model, whereas the people without language impairment did not. We conclude that aphasia fundamentally changes the relationship of gesture and speech. (shrink)

People with aphasia use gestures not only to communicate relevant content but also to compensate for their verbal limitations. The Sketch Model (De Ruiter, 2000) assumes a flexible relationship between gesture and speech with the possibility of a compensatory use of the two modalities. In the successor of the Sketch Model, the AR‐Sketch Model (De Ruiter, 2017), the relationship between iconic gestures and speech is no longer assumed to be flexible and compensatory, but instead iconic gestures are (...) assumed to express information that is redundant to speech. In this study, we evaluated the contradictory predictions of the Sketch Model and the AR‐Sketch Model using data collected from people with aphasia as well as a group of people without language impairment. We only found compensatory use of gesture in the people with aphasia, whereas the people without language impairments made very little compensatory use of gestures. Hence, the people with aphasia gestured according to the prediction of the Sketch Model, whereas the people without language impairment did not. We conclude that aphasia fundamentally changes the relationship of gesture and speech. (shrink)

Recent experiments have shown that people iconically modulate their prosody corresponding with the meaning of their utterance. This article reports findings from a story reading task that expands the investigation of iconic prosody to abstract meanings in addition to concrete ones. Participants read stories that contrasted along concrete and abstract semantic dimensions of speed and size. Participants read fast stories at a faster rate than slow stories, and big stories with a lower pitch than small stories. The effect of (...) speed was distributed across the stories, including portions that were identical across stories, whereas the size effect was localized to size-related words. Overall, these findings enrich the documentation of iconicity in spoken language and bear on our understanding of the relationship between gesture and speech. (shrink)

The framework of depicting put forward byClark (2016)offers a schematic vantage point from which to examine iconic language use. Confronting the framework with empirical data, we consider some of its key theoretical notions. Crucially, by reconceptualizing the typology of depictions, we identify an overlooked domain in the literature: “speech-embedded nonverbal depictions,” namely cases where meaning is communicated iconically, nonverbally, and without simultaneously co-occurring speech. In addition to contextualizing the phenomenon in relation to existing research, we demonstrate, with examples from (...) American TV talk shows, how such depictions function in real-life language use, offering a brief sketch of their complexities and arguing also for their theoretical significance. (shrink)

When naming certain hand-held, man-made tools, American Sign Language signers exhibit either of two iconic strategies: a handling strategy, where the hands show holding or grasping an imagined object in action, or an instrument strategy, where the hands represent the shape or a dimension of the object in a typical action. The same strategies are also observed in the gestures of hearing nonsigners identifying pictures of the same set of tools. In this paper, we compare spontaneously created gestures from (...) hearing nonsigning participants to commonly used lexical signs in ASL. Signers and gesturers were asked to respond to pictures of tools and to video vignettes of actions involving the same tools. Nonsigning gesturers overwhelmingly prefer the handling strategy for both the Picture and Video conditions. Nevertheless, they use more instrument forms when identifying tools in pictures, and more handling forms when identifying actions with tools. We found that ASL signers generally favor the instrument strategy when naming tools, but when describing tools being used by an actor, they are significantly more likely to use more handling forms. The finding that both gesturers and signers are more likely to alternate strategies when the stimuli are pictures or video suggests a common cognitive basis for differentiating objects from actions. Furthermore, the presence of a systematic handling/instrument iconic pattern in a sign language demonstrates that a conventionalized sign language exploits the distinction for grammatical purpose, to distinguish nouns and verbs related to tool use. (shrink)

Analogy researchers do not often examine gesture, and gesture researchers do not often borrow ideas from the study of analogy. One borrowable idea from the world of analogy is the importance of distinguishing between attributes and relations. Gentner observed that some metaphors highlight attributes and others highlight relations, and called the latter analogies. Mirroring this logic, we observe that some metaphoric gestures represent attributes and others represent relations, and propose to call the latter analogical gestures. We provide examples (...) of such analogical gestures and show how they relate to the categories of iconic and metaphoric gestures described previously. Analogical gestures represent different types of relations and different degrees of relational complexity, and sometimes cohere into larger analogical models. Treating analogical gestures as a distinct phenomenon prompts new questions and predictions, and illustrates one way that the study of gesture and the study of analogy can be mutually informative. (shrink)

In this paper I propose to read and understand gestures as logical tools within a synthetic paradigm of knowledge. This interpretation of gesture is drawn from a new pragmatist reading of reasoning in general, and synthetic reasoning in particular. Complete gestures are actions with a beginning and an end that bear a meaning. It is our regular way to embody vague ideas into singular actions with general meaning. The tool is forged by a dense blending of icons, indices, and (...) symbols and by a complexity of phenomenological characteristics as feelings, actual actions, general concepts and habits. The paper illustrates this new way to look at gestures and different kinds of complete and incomplete gestures. (shrink)

Ever since Charles Goodwin’s seminal works on gaze, there has been a long-standing interest in Conversation Analysis in the interrelationship between talk and bodily conduct in the accomplishment of social action. Recently, a small but emerging body of research has explored the ways in which embodied conduct figures in the organization and operations of repair. In this article, I take up a similar theme and investigate the interaction between talk and iconic gestures in same-turn self-initiated repair in Mandarin conversation. (...) The phenomenon I examine concerns the use of what I call “gestural repair.” The analysis focuses on how such repair can intertwine with talk in multi-stage operations in the progressivity and resolution of repair. The data are drawn from 50 hours of naturally-occurring conversations collected in China. Some unique features of such gestural repair observed in the Mandarin data are also discussed. (shrink)

It is common to distinguish two great families of representation. Symbolic representations include logical and mathematical symbols, words, and complex linguistic expressions. Iconic representations include dials, diagrams, maps, pictures, 3-dimensional models, and depictive gestures. This essay describes and motivates a new way of distinguishing iconic from symbolic representation. It locates the difference not in the signs themselves, nor in the contents they express, but in the semantic rules by which signs are associated with contents. The two kinds of (...) rule have divergent forms, occupying opposite poles on a spectrum of naturalness. Symbolic rules are composed entirely of primitive juxtapositions of sign types with contents, while iconic rules determine contents entirely by uniform natural relations with sign types. This distinction is marked explicitly in the formal semantics of familiar sign systems, both for atomic first-order representations, like words, pixel colors, and dials, and for complex second-order representations, like sentences, diagrams, and pictures. (shrink)

When young children learn to use language, they start to use their hands in co‐verbal gesturing. There are, however, considerable differences between children, and it is not completely understood what these individual differences are due to. We studied how children at 4 years of age employ speech and iconic gestures to convey meaning in different kinds of spatial event descriptions, and how this relates to their cognitive abilities. Focusing on spontaneous illustrations of actions, we applied a semantic feature (SF) (...) analysis to characterize combinations of speech and gesture meaning and related them to the child's visual‐spatial abilities or abstract/concrete reasoning abilities (measured using the standardized SON‐R test). Results show that children with higher cognitive abilities convey significantly more meaning via gesture and less solely via speech. These findings suggest that young children's use of cospeech representational gesturing is positively related to their mental representation and reasoning abilities. (shrink)

This essay review explores Steven Mithen’s interdisciplinary approach to the origins and evolution of language in _The Language Puzzle_ ( 2024 ). It focuses mainly on what I call his _iconic vocal origins hypothesis_. Mithen challenges the prevalent gestural origins hypothesis, suggesting instead that early prehistoric languages were predominantly vocal and iconic, with conventionalization – as characteristic of symbol use – emerging later. _The Language Puzzle_ draws on research from archaeology, philosophy, computer science, developmental psychology, and many other fields, (...) thus assembling a wealth of insights from various disciplines. While intriguing, Mithen’s suggestion that prehistoric languages may have relied on iconicity instead of conventionalization faces substantial problems, which are discussed in this review essay. In the final section, I also briefly review the important conclusion chapter of Mithen’s book, which contains an imaginative outline of how language evolved from the last common ancestor up until modern _H. sapiens._ My criticisms of the iconic vocal origins hypothesis notwithstanding, _The Language Puzzle_ is a valuable resource for anyone interested in language evolution, and once again showcases Mithen’s wide-ranging expertise and masterful writing. (shrink)

This paper explores the role of iconicity in spoken language and other human communication systems. First, we concentrate on graphical and gestural communication and show how semantically motivated iconic signs play an important role in creating such communication systems from scratch. We then consider how iconic signs tend to become simplified and symbolic as the communication system matures and argue that this process is driven by repeated interactive use of the signs. We then consider evidence for iconicity at (...) the level of the system in graphical communication and finally draw comparisons between iconicity in graphical and gestural communication systems and in spoken language. (shrink)

The tradeoff hypothesis in the speech–gesture relationship claims that (a) when gesturing gets harder, speakers will rely relatively more on speech, and (b) when speaking gets harder, speakers will rely relatively more on gestures. We tested the second part of this hypothesis in an experimental collaborative referring paradigm where pairs of participants (directors and matchers) identified targets to each other from an array visible to both of them. We manipulated two factors known to affect the difficulty of speaking to (...) assess their effects on the gesture rate per 100 words. The first factor, codability, is the ease with which targets can be described. The second factor, repetition, is whether the targets are old or new (having been already described once or twice). We also manipulated a third factor, mutual visibility, because it is known to affect the rate and type of gesture produced. None of the manipulations systematically affected the gesture rate. Our data are thus mostly inconsistent with the tradeoff hypothesis. However, the gesture rate was sensitive to concurrent features of referring expressions, suggesting that gesture parallels aspects of speech. We argue that the redundancy between speech and gesture is communicatively motivated. (shrink)

In lieu of an abstract, here is a brief excerpt of the content:Iconicity, Romance and History in the Crónica SarracinaMarina S. Brownlee (bio)Though seemingly alien discourses, romance and historiography are perennially linked. Far from offering an atemporal imaginary universe that bears no resemblance to historical specificity, romance is constructed as a response to it. Rather than simply projecting for the reader the naïve appeal of a prelapsarian escapism from the harsh realities of history, romance involves a continuous and sophisticated reinvention (...) of itself as a response to an ever-changing historico-political configuration. It is, in fact, a reciprocal relationship. We frequently see history appropriating romance paradigms explicitly, as, for example, when Bernal Díaz chooses to present Spain’s New World conquest and colonization as a continuation of the exploits of Amadís [see Gilman].Whatever form it takes, romance is committed to the celebration of a coherent system of socio-political values. This extra-textual frame of reference can take a variety of forms—from political propaganda that offers a self-aggrandizing depiction of the nobility or patron for whom the text is produced, to escapist fantasy—futuristic or archaizing. It is nostalgia for the lost world of chivalric romance which Cervantes embodies in the figure of Don Quijote, a foolish old man driven mad by his obsession with this perennial literary form. Cervantes is rightly credited with a brilliant and programmatic use of romance constructs to communicate the ongoing historical crisis of the Spanish empire. A byproduct of this romance scrutiny of history was his invention of the modern novel.Testimony to the long-standing interaction between romance and history in Spain, as elsewhere in Europe, is evident in the fact that its first romances were based on history, the so-called “romances of antiquity.” In those texts historical figures and events are evoked for the purpose of legitimating the present, vernacular poets as they celebrate the empowering myths projected by their society, as well as their own literary endeavors by invoking venerable Latin models. The well known translatio studii et imperii topos offers such allegory of empire.At the same time, the confrontation of romance and historical representation can also have a destabilizing effect, and this is programmatically explored in Pedro del Corral’s Crónica sarracina (c. 1430). His text ponders the possibilities of romance and historiography in an obsessive manner and he accomplishes his task by examining representation of the most iconic figures associated with the fall of Spain in 711, namely, La Cava and her father, Count Julián, King Rodrigo, and Pelayo, the messianic hope of the Reconquest effort which would last nearly 800 years.Of related and perhaps even broader importance, Corral’s text merits careful reading because its exploration of iconicity related to Spain’s fall offers meditations on the workings of ideology, which inevitably proposes “imaginary or formal solutions to unresolvable contradictions” [Jameson 79]. Ideology—its unmasking—is intriguingly posited by Corral in powerful ways.If we turn to Corral’s text, we find a work offering an extraordinary consideration of history and romance—their profound interaction—whose full title reads, La crónica [End Page 119] del rey don Rodrigo con la destrucción de España (The Chronicle of King Rodrigo with the Destruction of Spain). My interest in the Crónica here is primarily to consider the mechanisms by which it gestures boldly toward what Žižek has described as the project of the postmodern critique of ideology—though it is clearly a recurring cultural constant—namely to “designate the elements which point towards the system’s antagonistic character, and thus ‘estrange’ us to the self-evidence of its established identity” [Žižek 7]. A reading of the Crónica provides an extended paradigm of such estrangement.Iberia is the area of the Mediterranean which underwent the most protracted encounter between Christianity and Islam, and the events of the Moorish invasion of 711 constitute Spain’s unique foundational subject-matter—on a par with the Troy legend and with King Arthur [see Deyermond 355]. Like them, the retelling of 711 is all about forging mythic connections in order to legitimize empire in a temporally remote time-frame. What interests me in this essay is... (shrink)

In this paper, we intend to discuss how, according to Peirce’s semiotics, gestures can be conceptualized and described as diagrams. We are not concerned with answering when, why or how gestures emerged as semiotic motor activities. As a prerequisite for the very formulation of these problems, we are rather interested in discussing the theoretical conditions which should be fulflled for gestures to be characterized as iconic processes - and more specifcally as diagrammatic processes. The frst step is to summarize (...) a systematic analysis of the variety of iconic signs, elaborated by Peirce, followed by his typology of hypoicons in order to describe the hypoiconic class of diagrams. After that, we present some tentative answers to the question of how to describe gestures as diagrammatic signs. (shrink)

The role of nonverbal communication in patients with post-stroke language impairment (aphasia) is not yet fully understood. This study investigated how aphasic patients perceive and produce co-speech gestures during face-to-face interaction, and whether distinct brain lesions would predict the frequency of spontaneous co-speech gesturing. For this purpose, we recorded samples of conversations in patients with aphasia and healthy participants. Gesture perception was assessed by means of a head-mounted eye-tracking system, and the produced co-speech gestures were coded according to a (...) linguistic classification system. The main results are that meaning-laden gestures (e.g., iconic gestures representing object shapes) are more likely to attract visual attention than meaningless hand movements, and that patients with aphasia are more likely to fixate co-speech gestures overall than healthy participants. This implies that patients with aphasia may benefit from the multimodal information provided by co-speech gestures. On the level of co-speech gesture production, we found that patients with damage to the anterior part of the arcuate fasciculus showed a higher frequency of meaning-laden gestures. This area lies in close vicinity to the premotor cortex and is considered to be important for speech production. This may suggest that the use of meaning-laden gestures depends on the integrity of patients’ speech production abilities. (shrink)

The gesture-as-simulated-action framework explains the occurrence of iconic gestures. Accordingly, simulated visual imagery gives rise to observer-viewpoint, whereas simulated motor imagery gives rise to character-viewpoint gestures. Because little is known about whether this relationship is either the product of becoming a competent speaker in different communicative tasks or exists from an early age, we investigated 4-year-olds. In the first session, 55 children performed three different communicative tasks. In the second session, we administered a SON-R non-verbal intelligence test to (...) assess children’s cognitive abilities. Results revealed a strong association between the frequency with which children used observer-viewpoints gestures and their scores in visual processing abilities. Because the association between gestures and visuospatial cognitive abilities was found across three communicative tasks, we can propose that it is a general ability rather than a phenomenon of task-specific discourse competence. We discuss these findings in terms of gestural threshold and cognitive offloading. (shrink)

ASL can express plurals by repeating a noun, in an unpunctuated fashion, in different parts of signing space. We argue that this construction may come with a rich iconic component: the geometric arrangement of the repetitions provides information about the arrangement of the denoted plurality; in addition, the number and speed of the repetitions provide information about the size of the denoted plurality. Interestingly, the shape of the repetitions may introduce a new singular discourse referent when a vertex can (...) be inferred to denote a singular object. Thus one may point towards the first or last iteration of a horizontal repetition of BOOK to denote the left- or right-edge of the corresponding row. This yields a remarkable interaction between iconic semantics and standard logical semantics. We show that our analysis extends to ‘punctuated’ repetitions, which involve clearly individuated iterations of a singular noun. While these may initially look like coordinated indefinites, they are better handled by the same iconic framework as plural, unpunctuated repetitions. Some repetition-based mass terms also give rise to iconic effects, and to different readings depending on whether the repetition is continuous, unpunctuated, or punctuated. Our analysis highlights the need for a formal semantics with iconicity to study the integration of such iconic and logical conditions. It also raises a question: can similar facts be found in spoken language when gestures are taken into account? We suggest that several effects can be replicated, especially when one considers examples involving ‘pro-speech gestures’. (shrink)

TThis paper aims to evidence the inherently metonymic nature of co-speech gestures. Arguing that motivation in gesture involves iconicity (similarity), indexicality (contiguity), and habit (conventionality) to varying degrees, it demonstrates how a set of metonymic principles may lend a certain systematicity to experientially grounded processes of gestural abstraction and enaction. Introducing visuo-kinetic signs as an umbrella term for co-speech gestures and signed languages, the paper shows how a frame-based approach to gesture may integrate different cognitive/functional linguistic and semiotic (...) accounts of metonymy (e.g., experiential domains, frame metonymy, contiguity, and pragmatic inferencing). The guiding assumption is that gestures metonymically profile deeply embodied, routinized aspects of familiar scenes, that is, the motivating context of frames. The discussion shows how gestures may evoke frame structures exhibiting varying degrees of groundedness, complexity, and schematicity: basic physical action and object frames; more complex frames; and highly abstract, complex frame structures. It thereby provides gestural evidence for the idea that metonymy is more basic and more directly experientially grounded than metaphor and thus often feeds into correlated metaphoric processes. Furthermore, the paper offers some initial insights into how metonymy also seems to induce the emergence of schematic patterns in gesture which may result from action-based and discourse-driven processes of habituation and conventionalization. It exemplifies how these forces may engender grammaticalization of a basic physical action into a gestural marker that shows strong metonymic form reduction, decreased transitivity, and interacting pragmatic functions. Finally, addressing basic metonymic operations in signed lexemes elucidates certain similarities regarding sign constitution in gesture and sign. English and German multimodal discourse data as well as German Sign Language (DGS) are drawn upon to illustrate the theoretical points of the paper. Overall, this paper presents a unified account of metonymy’s role in underpinning forms, functions, and patterns in visuo-kinetic signs. (shrink)

In everyday reasoning - just as in science and art - knowledge is acquired more by "doing" than with long analyses. What do we "do" when we discover something new? How can we define and explore the pattern of this reasoning, traditionally called "synthetic"? Following in the steps of classic pragmatists, especially C.S. Pierce, Giovanni Maddalena's Philosophy of Gesture revolutionizes the pattern of synthesis through the ideas of change and continuity and proposes "gesture" as a new tool for (...) synthesis. Defining gesture as an action with a beginning and an end that carries on a meaning, Maddalena explains that it is a dense blending of all kinds of phenomena - feelings and vague ideas, actual actions, habits of actions - and of signs - icons, indexes, and symbols. When the blending of phenomena and signs is densest, the gesture is "complete," and its power of introducing something new in knowledge is at its highest level. Examples of complete gestures are religious liturgies, public and private rites, public and private actions that establish an identity, artistic performances, and hypothesizing experiments. A departure from a traditional Kantian framework for understanding the nature and function of reason, The Philosophy of Gesture proposes an approach that is more attuned with our ordinary way of reasoning and of apprehending new knowledge. (shrink)

On the young bride’s shoulder is a mauve bite mark: the outline of a mouth, a double arch,teeth marks, open jaws, lips raised up over hard enamel. Not the barely open lips of a kisson the skin; open, rather, as for a kiss on the mouth, but this time penetrating the skin: abristling kiss with the teeth bared, extreme – at the limit of the kiss, or beyond. A cruelkiss: a kiss of flesh . A young couple kisses in a (...) plane: the beginning ofthe film. Later we will see this icon, not knowing when it was imprinted, like a tattoo or abranding with the red hot iron of some ancient justice.What is a kiss? This is the question posed by Claire Denis’s film. Or rather: what isfucking?2It has long been accepted and repeated that kissing is a kind of devouring. Itbelongs to a core of imagery and metaphor that includes fairy tales , the fascination with cannibalism, the symbolism of Christian communion andthat of the lacerations of Dionysos, Osiris or Acteon, together with ghouls, striges andvampires, werewolves, incubi and succubi. This entire carnivorous breed is concealedwithin the film. It is recalled in its entirety, evoked by the gesture of Coré, the sick woman, standing on a bank, filmed from a low angle raising her coat aboveher shoulders to bring to mind for a moment the silhouette of Murnau’s Nosferatu.The vampire’s true formula is only revealed when one says ‘the kiss of the vampire’.That is what is at stake here, allowing for the fact that we are no longer in the era ofvampire stories: the kiss as vampire.It is not a question of any particular kiss, but rather that the kiss, in itself, opens on to the bite, and the taste of blood. And consequently it is a question of another wellknown coupling, that of Eros and Thanatos: not in a dialectic of opposites, but in a mutualexcitation and exasperation, each asking the other to go further, to go all the way to theend, to get completely lost. (shrink)

Many sign languages display crosslinguistic consistencies in the use of two iconic aspects of handshape, handshape type and finger group complexity. Handshape type is used systematically in form-meaning pairings (morphology): Handling handshapes (Handling-HSs), representing how objects are handled, tend to be used to express events with an agent (“hand-as-hand” iconicity), and Object handshapes (Object-HSs), representing an object's size/shape, are used more often to express events without an agent (“hand-as-object” iconicity). Second, in the distribution of meaningless properties of form (morphophonology), (...) Object-HSs display higher finger group complexity than Handling-HSs. Some adult homesigners, who have not acquired a signed or spoken language and instead use a self-generated gesture system, exhibit these two properties as well. This study illuminates the development over time of both phenomena for one child homesigner, “Julio,” age 7;4 (years; months) to 12;8. We elicited descriptions of events with and without agents to determine whether morphophonology and morphosyntax can develop without linguistic input during childhood, and whether these structures develop together or independently. Within the time period studied: (1) Julio used handshape type differently in his responses to vignettes with and without an agent; however, he did not exhibit the same pattern that was found previously in signers, adult homesigners, or gesturers: while he was highly likely to use a Handling-HS for events with an agent (82%), he was less likely to use an Object-HS for non-agentive events (49%); i.e., his productions were heavily biased toward Handling-HSs; (2) Julio exhibited higher finger group complexity in Object- than in Handling-HSs, as in the sign language and adult homesigner groups previously studied; and (3) these two dimensions of language developed independently, with phonological structure showing a sign language-like pattern at an earlier age than morphosyntactic structure. We conclude that iconicity alone is not sufficient to explain the development of linguistic structure in homesign systems. Linguistic input is not required for some aspects of phonological structure to emerge in childhood, and while linguistic input is not required for morphology either, it takes time to emerge in homesign. (shrink)

Why is the urge to lose the iconic image relevant to reformation and modernism? A question so central in a society built more than ever on visual media dependency. Is that relevant to sceptical questioning of the essence of reality, and if the image is a reflection of reality in the era of new technology of image creating and manipulating? As iconoclasts began deliberately destroying images at the alter as a sign of reformation, modern art was no longer bound (...) by truthful representations of iconic reality; from shift in colour as a medium to a scientific light reflection, to pushing colour boundaries in reality, to losing the object for the absolute representation of abstract and supermatist visual, to deliberate losing of intended visual representation to materials, then to the empty frame, and to the empty gallery … on goes the iconoclastic gesture of modern art. Followed by a transitional shift from iconic representation all the way to resonance in visual communications, exploring with the possibilities of different referential functions of signs, and thru to minimalism and replacing visual by verbal referential codes. This article studies actions and motives of deliberate loss of iconic image, examining the factors associated with the development and process, criss-crossing the boundaries of two image dependant yet very different domains; art and visual communication. (shrink)

Human communication is by defaultpolysemiotic: it involves the spontaneous combination of two or moresemiotic systems, the most important ones beinglanguage,gesture, anddepiction. We formulate an original cognitive-semiotic framework for the analysis of polysemiosis, contrasting this with more familiar systems based on the ambiguous term “multimodality.” To be fully explicit, we developed a coding system for the analysis of polysemiotic utterances containing speech, gesture, and drawing, and implemented this in the ELAN video annotation software. We used this to analyze 23 (...) video-recordings of sand drawing performances on Paama, Vanuatu and 20 sand stories of the Pitjantjatjara culture in Central Australia. Methodologically we used theconceptual-empirical loopof cognitive semiotics: our theoretical framework guided general considerations, such as distinguishing between the “tiers” of gesture and depiction, and the three kinds of semiotic grounds (iconic, indexical, symbolic), but the precise decisions on how to operationalize these were made only after extensive work with the material. We describe the coding system in detail and provide illustrative examples from the Paamese and Pitjantjatjara data, remarking on both similarities and differences in the polysemiosis of the two cultural practices. We conclude by summarizing the contributions of the study and point to some directions for future research. (shrink)

Cognitive Science, Volume 46, Issue 10, October 2022.