This paper provides a categorization of cross-modal experiences. There are myriad forms. Doing so allows us to think clearly about the nature of different cross-modal experiences and allows us to clearly formulate competing hypotheses about the kind of experiences involved in different cross-modal phenomena.

Cross-modal perceptual illusions occur when a stimulus to one modality impacts perceptual experience associated with another modality. Unlike synaesthesia, cross-modal illusions are intelligible as results of perceptual strategies for dealing with sensory stimulation to multiple modalities, rather than as mere quirks. I argue that understanding cross-modal illusions reveals an important flaw in a widespread conception of the senses, and of their role in perceptual experience, according to which understanding perception and perceptual experience is a (...) matter of assembling independently viable stories about vision, audition, olfaction, and the rest. (shrink)

The article explores cross-modal iconic relations in nine diverse Western-music songs ranging from 1600 to 2015, all of them thematizing dysphoric weeping. Initial input comes from five recurrent features observed in ancient Greek texts associated with performative events, including the prominence of sound, interjections and strong self-referentiality, repetitions and refrains, the motif of endlessness, and tears associated with streams of water, dew, and libation liquids. The analysis adopts Peirce’s conceptual distinction between image, diagram, and metaphor iconicity, although the (...) continuum reading proposed by several scholars is ultimately favored. The sample turns out to offer plenty of evidence of iconic relations. Interjections, falsetto, verbal repetitions, musical repetitions, musical rendering of sighs and endlessness, ostinato patterns, downward notes, and water images (rain and rivers), all of this is shown to substantiate cross-modal iconicity encompassing weeping, lyrics, music (including musical notation), and images, in different combinations. Quantitative investigations confirming/disconfirming and adding iconicity patterns, and comparative analyses linking Western and non-Western lament traditions constitute desiderata for future work. (shrink)

In the present study, cross-modal (i.e., across sensory modalities such as smell and sound) iconicity (i.e., resemblance) and indexicality (i.e., contiguity) in lexical sensory and emotional signs in Finnish Sign Language will be considered from an articulatory perspective (i.e., the production of signs). Such cross-modal iconicity has not been extensively studied previously, so here, with the help of cognitive semiotics, I aim to carefully describe the cross-modal patterns observed across 118 signs, including 60 sensory (...) signs and 58 emotional signs. The analysis is framed within the theoretical model of Semiotic Hierarchy, which entails a non-reductionist view of meaning. In addition, a pheno-methodological triangulation will be applied: phenomenology (first-person method), literature of phenomenological and semiotic descriptions (second-person perspective) and experimental findings (third-person perspective). The results of this analysis show that (a) 71 of the 118 sensory and emotional signs are cross-modally indexical, (b) only 10 of the 71 signs can be regarded as cross-modally iconic, (c) cross-modal iconicity is highly diagrammatic, (d) iconicity and indexicality are highly integrated, and (e) articulatory feedback matters in the formation of semiotic patterns. This study contributes to our understanding of cross-modal iconicity in signed languages, as well as studies in semiotic systems more generally. (shrink)

The thesis of this paper is that the capacity to think of one’s perceptions as cross-modally integrated is incompatible with a reductionist account of the self. In §2 I distinguish three versions of the argument from cross-modality. According to the ‘unification’ version of the argument, what needs to be explained is one’s capacity to identify an object touched as the same as an object simultaneously seen. According to the ‘recognition’ version, what needs to be explained is one’s capacity, (...) having once seen an object, to reidentify that same object by touch alone. According to the ‘objectivity’ version, what needs to be explained is one’s capacity to think of one’s perceptions in different modalities as perceptions of one and the same object. The third version seems to establish that one must conceive of oneself substantially, as the numerically identical owner of one’s experiences, a conclusion in agreement with recent work in developmental psychology claiming to show that an infant’s cross-modal capacities are essentially implicated in their development of a sense of self. There is further work to be done if this is to be tumed into an argument against reductionism: there is no swift route from the epistemology of self-consciousness to the metaphysics of the self. In the §3, I will claim that there is, nevertheless, an argument linking the two. What I propose is an argument derived, not from the token-reflexive rule for the first-person, but resting on its anaphoric behaviour spanning intensional operators. (shrink)

Early modern humans developed mental capabilities that were immeasurably greater than those of non-human primates. We see this in the rapid innovation in tool making, the development of complex language, and the creation of sophisticated art forms, none of which we find in our closest relatives. While we can readily observe the results of this high-order cognitive capacity, it is difficult to see how it could have developed. We take up the topic of cave art and archeoacoustics, particularly the discovery (...) that cave art is often closely connected to the acoustic properties of the cave chambers in which it is found. Apparently, early modern humans were able to detect the way sound reverberated in these chambers, and they painted artwork on surfaces that were acoustic “hot spots,” i.e., suitable for generating echoes. We argue that cave art is a form of cross-modality information transfer, in which acoustic signals are transformed into symbolic visual representations. This form of information transfer across modalities is an instance of how the symbolic mind of early modern humans was taking shape into concrete, externalized language. We also suggest that the earliest rock art found in Africa may constitute one of the first fossilized proxies for the expression of full-fledged human linguistic behavior. (shrink)

Three priming experiments were conducted to determine how information about the self from different sensory modalities/cognitive domains affects self-face recognition. Being exposed to your body odor, seeing your name, and hearing your name all facilitated self-face recognition in a reaction time task. No similar cross-modal facilitation was found among stimuli from familiar or novel individuals. The finding of a left-hand advantage for self-face recognition was replicated when no primes were presented. These data, along with other recent results suggest (...) the brain processes/represents information about the self in highly integrated ways. (shrink)

In this chapter we suggest that the origin of language, specifically the protolinguistic stage, was iconic rather than arbitrary, and fundamentally based on shared cross-modal associations. We provide evidence from natural language in the form of sound symbolism, distinguishing conventional sound symbolism from sensory sound symbolism. Sensory sound symbolism, or the presence of iconicity in natural language, is considered alongside psychological experiments in naming, and other investigations of cross-modal associations specifically involving linguistic sound. This evidence supports (...) the idea that we can directly express a variety of sensory experiences through linguistic sound, and thus that language systems have the capacity to be iconic to a large extent. We outline a theory of sensory protolanguage emergence, considering how and why arbitrariness eventually became the dominant relationship between form and meaning. Finally, we consider the possibility of continuity between cross-modal transfer in other primates and more abstract cross-modal associations among humans, suggesting this may have scaffolded the emergence of symbolic communication. (shrink)

Experiencing a stimulus in one sensory modality is often associated with an experience in another sensory modality. For instance, seeing a lemon might produce a sensation of sourness. This might indicate some kind of cross-modal correspondence between vision and gustation. The aim of the current study was to provide explore whether such cross-modal correspondences influence cross-modal integration during perceptual learning. To that end, we conducted 2 experiments. Using a speeded classification task, Experiment 1 established (...) a cross-modal correspondence between visual lightness and the frequency of an auditory tone. Using a short-term priming procedure, Experiment 2 showed that manipulation of such cross-modal correspondences led to the creation of a crossmodal unit regardless of the nature of the correspondence (i.e., congruent, Experiment 2a or incongruent, Experiment 2b). However, a comparison of priming-effects sizes suggested that cross-modal correspondences modulate cross-modal integration during learning and thus leading to new learned units that have different stability over time. We discuss the implications of our results for the relation between cross-modal correspondence and perceptual learning in the context of a Bayesian explanation of cross-modal correspondences. (shrink)

Thepuzzle of cross‐modal shape experienceis the puzzle of reconciling the apparent differences between our visual and haptic experiences of shape with their apparent similarities. This paper proposes that we can resolve the cross‐modal puzzle by reflecting on another puzzle. Thepuzzle of perspectival characterchallenges us to reconcile the variability of shape experience through shifts in perspective with its constancy. An attractive approach to the latter puzzle holds that shape experience is complex, involving bothperspectivalaspects andconstantaspects. I argue here (...) that parallel distinctions between perspectival and constant aspects of shape experience arise in sight and touch, and that perspectival aspects are modality‐specific while at least some constant aspects are constitutively multisensory. I then address a powerful challenge to the idea that aspects of spatial phenomenology are shared cross‐modally. (shrink)

Objective: Evidence suggests people experience an oral size illusion and commonly perceive oral size inaccurately; however, the nature of the illusion remains unclear. The objectives of the present study were to confirm the presence of an oral size illusion, determine the magnitude (amount) and direction (underestimation or overestimation) of the illusion, and determine whether immediately prior crossmodal perceptual experiences affected the magnitude and direction. Design: Participants (N = 27) orally assessed 9 sizes of stainless steel spheres (1/16 in to 1/2 (...) in) categorized as small, medium, or big, and matched them with digital and visual reference sets. Each participant completed 20 matching tasks in 3 assessments. For control assessments, 6 oral spheres were matched with reference sets of same-sized spheres. For primer-control assessments, similar to control, 6 matching tasks were preceded by cross-modal experiences of the same-sized sphere. For experimental assessments, 8 matching tasks were preceded by a cross-modal experience of a differently sized sphere. Results: For control assessments, small and medium spheres were consistently underestimated, and big spheres were consistently overestimated. For experimental assessments, magnitude and direction of the oral size illusion varied according to the size of the sphere used in the cross-modal experience. Conclusion: Results seemed to confirm an oral size illusion, but direction of the illusion depended on the size of the object. Immediately prior cross-modal experiences influenced magnitude and direction of the illusion, suggesting that aspects of oral perceptual experience are dependent upon factors outside of oral perceptual anatomy and the properties of the oral stimulus. (shrink)

We investigate the dynamics of sensory integration for perceiving musical performance, a complex natural behavior. Thirty musically trained participants saw, heard, or both saw and heard, performances by two clarinetists. All participants used a sliding potentiometer to make continuous judgments of tension (a measure correlated with emotional response) and continuous judgments of phrasing (a measure correlated with perceived musical structure) as performances were presented. The data analysis sought to reveal relations between the sensory modalities (vision and audition) and to quantify (...) the effect of seeing the performances on participants' overall subjective experience of the music. In addition to traditional statistics, functional data analysis techniques were employed to analyze time-varying aspects of the data. The auditory and visual channels were found to convey similar experiences of phrasing but different experiences of tension through much of the performances. We found that visual information served both to augment and to reduce the experience of tension at different points in the musical piece (as revealed by functional linear modeling and functional significance testing). In addition, the musicians' movements served to extend the sense of phrasing, to cue the beginning of new phrases, to indicate musical interpretation, and to anticipate changes in emotional content. Evidence for an interaction effect suggests that there may exist an emergent quality when musical performances are both seen and heard. The investigation augments knowledge of human communicative processes spanning language and music, and involving multiple modalities of emotion and information transfer. (shrink)

The importance of learning and categorizing social objects and events has become widely acknowledged over a couple of decades. Although findings from field studies have suggested that non-human animals have sophisticated abilities to recognize social objects, there is relatively little experimental evidence on this issue. Some studies have revealed animals’ excellent skills for discriminating visual and auditory social stimuli. However, because of perceptual resemblances among stimuli, it is still not clear that they recognize these objects with conceptual mechanisms that are (...) independent of the perceptual characteristics of the stimuli. At the same time, whether their concepts have an aspect of transferring information from one modality to another has not received much attention. This paper advocates approaches to a cross-modal aspect of concepts as a new framework to solve these problems, and introduces the latest studies on cross-modal representations of social objects in non-humans. (shrink)

I argue that a class of recently-discovered cross-modal illusions gives reason to posit a dimension of content shared across perceptual modalities and to abandon the traditional view according to which perceptual content is exclusively constituted by discrete modality-specific contents.

Maintaining and coordinating multiple task-sets is difficult and leads to costs, however task-switching training can reduce these deficits. A recent study in young adults demonstrated that this training effect occurs at an amodal processing level. Old age is associated with reduced cognitive plasticity and further increases the performance costs when mixing multiple tasks. Thus, cognitive aging might be a limiting factor for inducing cross-modal training effects in a task-switching environment. We trained participants, aged 62–83 years, with an auditory (...) task-switching paradigm over four sessions, to investigate whether training-related reductions in task-switching costs would also manifest in an untrained visual modality version of the task. Two control groups trained with single tasks or not trained allowed us to identify improvements specific to task-switching training. To make statistical evaluations of any age differences in training and cross-modal transfer, the data from the Kattner cohort were incorporated into the present analysis. Despite the tendency for older adults to respond more cautiously, task-switching training specifically led to a mixing cost reduction in both trained and untrained modalities, the magnitude of which was statistically similar regardless of age. In line with a growing body of research, we failed to observe any far transfer effects in measures of inhibition, working memory or fluid intelligence. Overall, we conclude that any apparent cognitive limitations associated with aging do not prevent cognitive control processes which support set-shifting from improving at an amodal level. (shrink)

It is being increasingly recognized that the Saussurean dictum of “the arbitrariness of the linguistic sign” is in conflict with the pervasiveness of the phenomenon commonly known as “sound symbolism”. After first presenting a historical overview of the debate, however, we conclude that both positions have been exaggerated, and that an adequate explanation of sound symbolism is still lacking. How can there, for example, be (perceived) similarity between expressionsand contents across different sensory modalities? We offer an answer, based on the (...) Peircian notion of iconic ground, and G. Sonesson’s distinction betweenprimary and secondary iconicity. Furthermore, we describe an experimental study, in a paradigm first pioneered by W. Kohler, and recently popularized by V.Ramachandran, in which we varied vowels and consonants in fictive word-forms, and conclude that both types of sounds play a role in perceiving an iconic ground between the word-forms and visual figures. The combination of historical conceptual analysis, semiotic explication and psychological experimentationpresented in this article is characteristic of the emerging paradigm of cognitive semiotics. (shrink)

For more than a century now, researchers have acknowledged the existence of seemingly arbitrary crossmodal congruency effects between dimensions of sensory stimuli in the general population. Such phenomena, known by a variety of terms including 'crossmodal correspondences', involve individual stimulus properties, rely on a crossmodal mapping of unisensory features, and appear to be shared by the majority of individuals. In other words, members of the general population share underlying preferences for specific pairings across the senses. Crossmodal correspondences between complementary sensory (...) cues have often been referred to as synesthetic correspondences but, we would argue, differ from full-blown synesthetic experiences in a number of important ways, including the fact that there are no idiosyncratic concurrent sensations. Recent psychophysical evidence suggests that such crossmodal correspondences can modulate multisensory integration by helping to resolve the crossmodal binding problem. Here, we propose a model to account for the effects of crossmodal correspondences between complementary auditory and visual cues and critically review their relation to full-blown synesthesia. (shrink)

The representational theory of phenomenal experience is often stated in terms of a supervenience thesis: Byrne recently characterises it as the thesis that “there can be no difference in phenomenal character without a difference in content”, while according to Tye, “[a]t a minimum, the thesis is one of supervenience: necessarily, experiences that are alike in their representational contents are alike in their phenomenal character.” Consequently, much of the debate over whether representationalism is true centres on purported counter-examples – that is (...) to say, purported failures of supervenience. The refutation of putative counter-examples has been, it seems to me, by and large successful. But there is a certain class of these for which the representationalist response has been something less than completely convincing. These are the cross-modality cases. I will explain what I mean, and then argue that the response in question is not only unconvincing but actually undermines the representationalist position. (shrink)

Cognitive Science, Volume 46, Issue 6, June 2022.

The brain-computer interface based on functional near-infrared spectroscopy has received more and more attention due to its vast application potential in emotion recognition. However, the relatively insufficient investigation of the feature extraction algorithms limits its use in practice. In this article, to improve the performance of fNIRS-based BCI, we proposed a method named R-CSP-E, which introduces EEG signals when computing fNIRS signals’ features based on transfer learning and ensemble learning theory. In detail, we used the Independent Component Analysis algorithm for (...) the correspondence between the sources of the two signals. We then introduced the EEG signals when computing the spatial filter based on a modified Common Spatial Pattern algorithm. Experimental results on public datasets show that the proposed method in this paper outperforms traditional methods without transfer. In general, the mean classification accuracy can be increased by up to 5%. To our knowledge, it is an innovation that we tried to apply transfer learning between EEG and fNIRS. Our study’s findings not only prove the potential of the transfer learning algorithm in cross-model brain-computer interface, but also offer a new and innovative perspective to research the hybrid brain-computer interface. (shrink)

In a new integration, we show that the visual-spatial structuring of time converges with auditory-spatial left–right judgments for time-related words. In Experiment 1, participants placed past and future-related words respectively to the left and right of the midpoint on a horizontal line, reproducing earlier findings. In Experiment 2, neutral and time-related words were presented over headphones. Participants were asked to indicate whether words were louder on the left or right channel. On critical experimental trials, words were presented equally loud binaurally. (...) As predicted, participants judged future words to be louder on the right channel more often than past-related words. Furthermore, there was a significant cross-modal overlap between the visual-spatial ordering and the auditory judgments , which were continuously related. These findings provide support for the assumption that space and time have certain invariant properties that share a common structure across modalities. (shrink)

Discrepancies between an agent's goals and beliefs play an important, if implicit, role in determining what a rational agent is motivated to do. This is most obvious in cases where an agent achieves a complex goal incrementally and must deliberate anew as each milestone is reached. In such cases the concept of goal/belief discrepancy defines an appropriate space to which a degree-of-achievement yardstick can be applied. This paper presents soundness and completeness results concerning a logic for reasoning about goal/belief discrepancy, (...) and it is suggested that a certain species of goal/belief discrepancy captures the concept of desire. (shrink)