We examined whether facial expressions of performers influence the emotional connotations of sung materials, and whether attention is implicated in audio-visual integration of affective cues. In Experiment 1, participants judged the emotional valence of audio-visual presentations of sung intervals. Performances were edited such that auditory and visual information conveyed congruent or incongruent affective connotations. In the single-task condition, participants judged the emotional connotation of sung intervals. In the dual-task condition, participants judged the emotional connotation of intervals (...) while performing a secondary task. Judgements were influenced by melodic cues and facial expressions and the effects were undiminished by the secondary task. Experiment 2 involved identical conditions but participants were instructed to base judgements on auditory information alone. Again, facial expressions influenced judgements and the effect was undiminished by the secondary task. The results suggest that visual aspects of music performance are automatically and preattentively registered and integrated with auditory cues. (shrink)

Can we find different means of communicative visual solutions to aid societies in the development of multicultural global communities? This means not revolving only around a particular language or even human language, but rather the concept and structure of communication. This transforms any kind of sign, body gesture or visual element into a method of communication, not simple phenomena. The aim of the research is to use media art and new technology to establish a better (wide) visual (...) communicative language in multicultural societies and provide a solid bridge between east and west, through in-depth visual/technological experimentation, seeking a transcultural visual solution to the problem of visual integration. Can any perceivable phenomenon become a conveyor of meaning? While language is based very much on common agreements, the way each of us responds to a message can be entirely different. Communication therefore creates diversity from uniformity: the idea of mixing setups of humans and machines (robots, computers, etc.) that are somehow able to communicate. The communication would give rise to some sort of choreography of movements that is synchronized because of the communication but diverse because of different responses to the communication. (shrink)

Motion-sound synesthesia is characterized by illusory auditory sensations linked to the pattern and rhythms of motion of visually experienced but soundless object, like an optical flow array, a ball bouncing or a horse galloping. In an MRI study with a group of three synesthetes and a group of eighteen control participants, we found structural changes in the brains of synesthetes in the subcortical multisensory areas of the superior and inferior colliculi. In addition, functional magnetic resonance imaging data showed activity in (...) motion-sensitive regions, as well as temporal and occipital areas, and the cerebellum. However, the synesthetes had a higher activation within the left and right cuneus, with stronger activations when viewing optical flow stimuli. There was also a general difference in connectivity of the colliculi with the above mentioned regions between the two groups. These findings implicate low-level mechanisms within the human neuroaxis as a substrate for local connectivity and cross activity between perceptual processes that are “distant” in terms of cortical topography. The present findings underline the importance of considering the role of subcortical systems and their connectivity to multimodal regions of the cortex and they strengthen a parsimonious account of synesthesia, at the least of the visual-auditory type. (shrink)

We propose here that a visual-motion integration deficit constitutes a crucial neuropsychological marker for at least a subgroup of ASD.

Corballis suggests that apes lack voluntary control over their vocal production. However, recent evidence implicates voluntary control of vocalizations in apes, which suggests that intentional control of vocal communication predates the hominid-pongid split. Furthermore, the ease with which apes in captivity manipulate the visual attention of observers implies a common cognitive basis for joint attention in humans and apes.

Previous research has successfully used feature integration theory to operationalise the predictions of Perceptual Load Theory, while simultaneously testing the predictions of both models. Building on this work, we test the extent to which these models hold up in a 3D world. In two experiments, participants responded to a target stimulus within an array of shapes whose apparent depth was manipulated using a combination of monoscopic and stereoscopic cues. The search task was designed to test the predictions of feature (...) integration theory, as the target was identified by a single feature or a conjunction of features and embedded in search arrays of varying size, and perceptual load theory, as the task included congruent and incongruent distractors presented alongside search tasks imposing high or low perceptual load. Findings from both experiments upheld the predictions of feature integration theory, regardless of 2D/3D condition. Longer search times in conditions with a combination of monoscopic and stereoscopic depth cues suggests that binding features into three-dimensional objects requires greater attentional effort. This additional effort should have implications for perceptual load theory, yet our findings did not uphold its predictions; the effect of incongruent distractors did not differ between conjunction search trials and feature search trials. Individual differences in susceptibility to the effects of perceptual load were evident and likely explain the absence of load effects. Overall, our findings suggest that feature integration theory may be useful for predicting attentional performance in a 3D world. (shrink)

It is argued that the alleged cases of cognitive penetration of visual modules actually arise from the integration of information among different modules. This would reflect a general computational strategy according to which constraints to a particular module would be provided by information coming from different modules. Examples are provided from the integration of stereopsis and occlusion and from computation of motion direction.

Music can act as a mnemonic device that can elicit multiple memories. How musical and non-musical information integrate into complex cross-modal memory representations has however rarely been investigated. Here, we studied the ability of human subjects to associate visual objects with melodies. Musical laypersons and professional musicians performed an associative inference task that tested the ability to form and memorize paired associations between objects and melodies and to integrate these pairs into more complex representations where melodies are linked with (...) two objects across trials. We further investigated whether and how musical expertise modulates these two processes. We analyzed accuracy and reaction times of direct and indirect trials in both groups. We reasoned that the musical and cross-modal memory demands of musicianship might modulate performance in the task and might thus reveal mechanisms that underlie the association and integration of visual information with musical information. Although musicians showed a higher overall memory accuracy, non-musicians’ accuracy was well above chance level in both trial types, thus indicating a significant ability to associate and integrate musical with visual information even in musically untrained subjects. However, non-musicians showed shorter RTs in indirect compared to direct trials, whereas the reverse pattern was found in musicians. Moreover, accuracy of direct and indirect trials correlated significantly in musicians but not in non-musicians. Consistent with previous accounts of visual associative memory, we interpret these findings as suggestive of at least two complimentary mechanisms that contribute to visual-melodic memory integration. A default mechanism that mainly operates at encoding of complex visual-melodic associations and that works with surprising efficacy even in musically untrained subjects. A retrieval-based mechanism that critically depends on an expert ability to maintain and discriminate visual-melodic associations across extended memory delays. Future studies may investigate how these mechanisms contribute to the everyday experience of music-evoked memories. (shrink)

The sense of agency is a subjective feeling that one's own actions drive action outcomes. Previous studies have focused primarily on the temporal contingency between actions and sensory inputs as a possible mechanism for the sense of agency. However, the contribution of the integrity of visual inputs has not been systematically addressed. In the current study, we developed a psychophysical task to examine the role of visual inputs as well as temporal contingencies toward the sense of agency. Specifically, (...) participants were required to track a target on a sinusoidal curve on a computer screen. Visual integrity of sensory inputs was manipulated by gradually occluding a computer cursor, and participants were asked to report the sense of agency on a nine-point Likert scale. Temporal contingency was manipulated by varying the delay between finger movements on a touchpad and cursor movements. The results showed that the sense of agency was influenced by both visual integrity and temporal contingency. These results are discussed in the context of current models that have proposed that the sense of agency emerges from the comparison of visual inputs with motor commands. (shrink)

An important insight derived from Kant about the workings of the mind is that conscious activity involves both the selection of relevant information, and the integration of that information, so as to form mental coherency. The conscious mind can then utilize this coherent information to solve problems, invent tools, synthesize concepts, produce works of art, and the like. In this paper, it will be suggested that just as biological processes, in general, exhibit selective and integrative functions, and just as (...) visual integration performs the function of selecting and integrating visual module areas, so too, consciousness emerged as a property of the brain to act as a kind of meta-cognitive process that selects and integrates relevant information from psychological modules. The upshot is to establish a psycho-neuro-biological continuum by suggesting that the conscious psychological properties of selectivity and integration are possible because of similar properties that other neurobiological and biological processes exhibit. (shrink)

Recent advances in the field of statistical learning have established that learners are able to track regularities of multimodal stimuli, yet it is unknown whether the statistical computations are performed on integrated representations or on separate, unimodal representations. In the present study, we investigated the ability of adults to integrate audio and visual input during statistical learning. We presented learners with a speech stream synchronized with a video of a speaker’s face. In the critical condition, the visual (e.g. (...) /gi/) and auditory (e.g. /mi/) signals were occasionally incongruent, which we predicted would produce the McGurk illusion, resulting in the perception of an audiovisual syllable (e.g. /ni/). In this way, we used the McGurk illusion to manipulate the underlying statistical structure of the speech streams, such that perception of these illusory syllables facilitated participants’ ability to segment the speech stream. Our results therefore demonstrate that participants can integrate audio and visual input to perceive the McGurk illusion during statistical learning. We interpret our findings as support for modality-interactive accounts of statistical learning. (shrink)

Integration information theories posit that the integration of information is necessary and/or sufficient for consciousness. In this paper, we focus on three of the most prominent information integration theories: Information Integration Theory, Global Workspace Theory, and Attended Intermediate-Level Theory. We begin by explicating each theory and key concepts they utilize. We then argue that the current evidence indicates that the integration of information is neither necessary nor sufficient for consciousness. Unlike GWT and AIR, IIT maintains (...) that conscious experience is both necessary and sufficient for consciousness. We present empirical evidence indicating that simple features are experienced in the absence of feature integration and argue that it challenges IIT’s necessity claim. In addition, we challenge IIT’s sufficiency claim by presenting evidence from hemineglect cases and amodal completion indicating that contents may be integrated and yet fail to give rise to subjective experience. Moreover, we present empirical evidence from subjects with frontal lesions who are unable to carry out simple instructions and argue that they are irreconcilable with GWT. Lastly, we argue that empirical evidence indicating that patients with visual agnosia fail to identify objects they report being conscious of present a challenge to AIR’s necessity claim. (shrink)

was to test the role of both early and higher visual areas in the integration of local features into global shapes. To this end, we conducted functional magnetic resonance imaging studies. Although fMRI lacks the high spatial resolution of intracortical recordings, it allows simultaneous collection of responses to the same stimulus set from multiple visual areas that is not possible with standard recording techniques. We performed these studies in monkeys, where much is known about the properties of (...) neurons in different visual areas, and in humans, where recent fMRI.. (shrink)

In the visual system, the represented features of individual objects (shape, colour, movement, and so on) are distributed both in space and time within the brain. Representations of inner and outer event sequences arrive through different sense organs at different times, and are likewise distributed. Objects are nevertheless perceived as integrated wholes - and event sequences are experienced to form a coherent "consciousness stream." In their thoughtful article, Dennett & Kinsbourne ask how this is achieved.

This article introduces an experimental paradigm to selectively probe the multiple levels of visual processing that influence the formation of object contours, perceptual boundaries, and illusory contours. The experiments test the assumption that, to integrate contour information across space and contrast sign, a spatially short-range filtering process that is sensitive to contrast polarity inputs to a spatially long-range grouping process that pools signals from opposite contrast polarities. The stimuli consisted of thin subthreshold lines, flashed upon gaps between collinear inducers (...) which potentially enable the formation of illusory contours. The subthreshold lines were composed of one or more segments with opposite contrast polarities. The polarity nearest to the inducers was varied to differentially excite the short-range filtering process. The experimental results are consistent with neurophysiological evidence for cortical mechanisms of contour processing and with the Boundary Contour System model, which identifies the short-range filtering process with cortical simple cells, and the long-range grouping process with cortical bipole cells. (shrink)

In the social world, multiple sensory channels are used concurrently to facilitate communication. Among human and nonhuman pri- mates, faces and voices are the primary means of transmitting social signals (Adolphs, 2003; Ghazanfar and Santos, 2004). Primates recognize the correspondence between species-specific facial and vocal expressions (Massaro, 1998; Ghazanfar and Logothetis, 2003; Izumi and Kojima, 2004), and these visual and auditory channels can be integrated into unified percepts to enhance detection and discrimination. Where and how such communication signals are (...) integrated at the neural level are poorly understood. In particular, it is unclear what role “unimodal” sensory areas, such as the auditory cortex, may play. We recorded local field potential activity, the signal that best correlates with human imaging and event-related potential signals, in both the core and lateral belt regions of the auditory cortex in awake behaving rhesus monkeys while they viewed vocalizing conspecifics. We demonstrate unequivocally that the primate auditory cortex integrates facial and vocal signals through enhancement and suppression of field potentials in both the core and lateral belt regions. The majority of these multisensory responses were specific to face/voice integration, and the lateral belt region shows a greater frequency of multisensory integration than the core region. These multisensory processes in the auditory cortex likely occur via recip- rocal interactions with the superior temporal sulcus. (shrink)

Recent studies have shown the importance of integrating multisensory information in the body representation for constituting self-consciousness. However, one idea that has received only scant attention is that our body representation is also constituted by knowledge of bodily visual characteristics . Here in two experiments we used a full body crossmodal congruency task in which visual distractors were presented on a photograph of the participant, another person, who was either familiar or unfamiliar, or an object. Results revealed that (...) during the ‘self-condition’ CCEs were enhanced compared to the ‘other condition’. The CCE was similar for unfamiliar and familiar others. CCEs for the object condition were significantly smaller. The results show that presentation of an irrelevant image of a body affects multimodal processing and that the effect is enhanced when that image is of the self. The results hold intriguing implications for body representation in social situations. (shrink)

In 2012, and 2014 Libraries and Learning Services from the University of Auckland created two online courses to introduce students to the concept of academic integrity and its associated values and expectations. The challenge was to introduce the somewhat dry subject matter to a diverse group of students in an engaging way and to avoid large tracts of text that were difficult to comprehend. Initial research undertaken by the development team suggested that visually representing bodies of text was an effective (...) way of communicating with the target audience. Dual coding theory, multimodal instructional design and cognitive load theories provided a framework for the learning design approach taken by the design team who ultimately selected Richard Mayer’s (2001) Cognitive Theory of Multimedia Learning as a basis for designing the two courses. This paper outlines how the two courses were designed in accordance principles outlined in Mayer’s theory and how the design team integrated seven of Mayer’s twelve principles to create dynamic, visual and aural elements such as animations, narrations and text to engage learners in a more meaningful learning process. (shrink)

We propose a new relational direction in higher education that acknowledges external and internal images as integrated in thinking and learning. We expand educational theory and practice that commonly rely on discrete conceptual developments that exclude images. Our argument epistemologically relies on certain semiotic views that consider the role of iconic signs and iconicity (meaning making by the virtue of similarity) as significant in relation to knowledge and learning. The analogical and imaginative work required to discover similarity between external pictures (...) and any educational concept is a form of iconic mind work that opens a space for transformational thinking and creative solutions. The novelty of our argument lies in the schematic philosophy of mind by C.S. Peirce, and specifically, an adaptation of Peirce’s triadic sign diagram for learning, exemplified via an ‘inquiry graphic’ sign and reflective method. This outlook seeks connections in diversity, arguing for an education that acknowledges and teaches intrinsic human relatedness to other humans, beings and things in the environment, thus challenging any notions of anthropocentrism and superiority. It calls for a reaction in international higher education to acknowledge the iconicity of concepts and thinking and embrace integrated image-concept signs for deep and critical teaching-learning, towards a modest educational contribution to counteracting social and environmental crises. (shrink)