Recent advances in the field of statistical learning have established that learners are able to track regularities of multimodal stimuli, yet it is unknown whether the statistical computations are performed on integrated representations or on separate, unimodal representations. In the present study, we investigated the ability of adults to integrate audio and visual input during statistical learning. We presented learners with a speech stream synchronized with a video of a speaker’s face. In the critical condition, the visual (...) (e.g. /gi/) and auditory (e.g. /mi/) signals were occasionally incongruent, which we predicted would produce the McGurk illusion, resulting in the perception of an audiovisual syllable (e.g. /ni/). In this way, we used the McGurk illusion to manipulate the underlying statistical structure of the speech streams, such that perception of these illusory syllables facilitated participants’ ability to segment the speech stream. Our results therefore demonstrate that participants can integrate audio and visual input to perceive the McGurk illusion during statistical learning. We interpret our findings as support for modality-interactive accounts of statistical learning. (shrink)

A growing number of children in the United States are exposed to multiple languages at home from birth. However, relatively little is known about the early process of word learning—how words are mapped to the referent in their child-centered learning experiences. The present study defined parental input operationally as the integrated and multimodal learning experiences as an infant engages with his/her parent in an interactive play session with objects. By using a head-mounted eye tracking device, we (...) recorded visual scenes from the infant’s point of view, along with the parent’s social input with respect to gaze, labeling, and actions of object handling. Fifty-one infants and toddlers from an English monolingual or a diverse bilingual household were recruited to observe the early multimodal learning experiences in an object play session. Despite that monolingual parents spoke more and labeled more frequently relative to bilingual parents, infants from both language groups benefit from a comparable amount of socially coordinated experiences where parents name the object while the object is looked at by the infant. Also, a sequential path analysis reveals multiple social coordinated pathways that facilitate infant object looking. Specifically, young children’s attention to the referent objects is directly influenced by parent’s object handling. These findings point to the new approach to early language input and how multimodal learning experiences are coordinated socially for young children growing up with monolingual and bilingual learning contexts. (shrink)

This is an excerpt from a report that highlights and explores five questions which arose from the workshop on perceptual learning and perceptual recognition at the University of Toronto, Mississauga on May 10th and 11th, 2012. This excerpt explores the question: What are the origins of multimodal associations?

The learning style myth is a commonly held myth that matching instruction to a student's “learning style” will result in improved learning, while providing mismatched instruction will result in suboptimal learning. The present study used a short online reasoning exercise about the efficacy of multimodal instruction to investigate the nature of learning styles beliefs. We aimed to: understand how learning style beliefs interact with beliefs about multimodal learning; characterize the potential complexity (...) of learning style beliefs and understand how this short exercise might influence endorsements of learning styles. Many participants who believed in the learning style myth supported the efficacy of multimodal learning, and many were willing to revise their belief in the myth after the exercise. Personal experiences and worldviews were commonly cited as reasons for maintaining beliefs in learning styles. Findings reveal the complexity of learning style beliefs, and how they interact with evidence in previously undocumented ways. (shrink)

There is little room left for doubt or even debate at the severity of the ecological, indeed planetary crises that we find ourselves in during this period coined the Anthropocene. As educators working in the face of these crises, we have asked ourselves the question ‘how do we carry on?’ We reflect on a set of sensory, multimodal, meditative and arts-based pedagogical activities that bridge the geographical, biological, sociological and environmental dimensions of learning using the concepts from Hannah (...) Arendt and John Dewey, in a higher-education context in Sweden. The first-cycle (undergraduate level) course in which these activities are conducted—Teaching sustainability from a global perspective - engages students in a range of pedagogical activities designed to encourage greater awareness of sustainability issues in diverse educational settings. In this paper we reflect on pedagogical practices as praxis in a sustainability focused course, and use the theory of practice architectures to question what people do in a particular time and place. We explore how educating for the future using multimodal and arts-based pedagogies could be a strategy to support new ways for reconfiguring environmental sustainability education in the Anthropocene. (shrink)

Although multimodal input has the potential to lead to more sound learning outcomes, it carries the risk of causing cognitive overload, making it difficult to determine the exact effects of multimodal input on the second language phrase learning. This study tests the efficacy of multimodal input on L2 phrase learning. It adopts a mixed-method approach by utilizing both quantitative and qualitative data. The experimental design is a 2 × 3 mixed model, with a group (...) [the experimental group and the control group ] as the between-subject factor and time as the within-subject factor. A total of 66 participants were divided into two groups. All materials incorporated three aspects of phrase knowledge, but the materials of the CG were unimodal in that they were offered only on paper, and of the EG were multimodal in that they included pictures, audio recordings, and video clips. After the treatment, a questionnaire and a semi-structured interview were given to the EG learners to explore their perceptions of using multimodal materials to learn L2 phrases. The results indicate that both groups had significant gains in learning phrases, but students with the multimodal input achieved significantly better results than those with the unimodal input. Moreover, the EG students had a generally positive attitude toward the use of multimodal resources. This study validates the efficacy of multimodal input on the acquisition of English phrases and shows that cognitive overload was avoided by sequencing the information. (shrink)

A lot of research has been done on the detection of mental workload using various bio-signals. Recently, deep learning has allowed for novel methods and results. A plethora of measurement modalities have proven to be valuable in this task, yet studies currently often only use a single modality to classify MWL. The goal of this research was to classify perceived mental workload using a deep neural network that flexibly makes use of multiple modalities, in order to allow for feature (...) sharing between modalities. To achieve this goal, an experiment was conducted in which MWL was simulated with the help of verbal logic puzzles. The puzzles came in five levels of difficulty and were presented in a random order. Participants had 1 h to solve as many puzzles as they could. Between puzzles, they gave a difficulty rating between 1 and 7, seven being the highest difficulty. Galvanic skin response, photoplethysmograms, functional near-infrared spectrograms and eye movements were collected simultaneously using LabStreamingLayer. Marker information from the puzzles was also streamed on LSL. We designed and evaluated a novel intermediate fusion multimodal DNN for the classification of PMWL using the aforementioned four modalities. Two main criteria that guided the design and implementation of our DNN are modularity and generalisability. We were able to classify PMWL within-level accurate on a seven-level workload scale using the aforementioned modalities. The model architecture allows for easy addition and removal of modalities without major structural implications because of the modular nature of the design. Furthermore, we showed that our neural network performed better when using multiple modalities, as opposed to a single modality. The dataset and code used in this paper are openly available. (shrink)

In the present study, we tested the effectiveness of color coding on the programming learning of students who were learning from video lectures. Effectiveness was measured using multimodal physiological measures, combining eye tracking and electroencephalography. Using a between-subjects design, 42 university students were randomly assigned to two video lecture conditions. The participants’ eye tracking and EEG signals were recorded while watching the assigned video, and their learning performance was subsequently assessed. The results showed that the color-coded (...) design was more beneficial than the grayscale design, as indicated by smaller pupil diameter, shorter fixation duration, higher EEG theta and alpha band power, lower EEG cognitive load, and better learning performance. The present findings have practical implications for designing slide-based programming learning video lectures; slides should highlight the format of the program code using color coding. (shrink)

By the time they reach early adulthood, English speakers are familiar with the meaning of thousands of words. In the last decades, computational simulations known as distributional semantic models have demonstrated that it is possible to induce word meaning representations solely from word co-occurrence statistics extracted from a large amount of text. However, while these models learn in batch mode from large corpora, human word learning proceeds incrementally after minimal exposure to new words. In this study, we run a (...) set of experiments investigating whether minimal distributional evidence from very short passages suffices to trigger successful word learning in subjects, testing their linguistic and visual intuitions about the concepts associated with new words. After confirming that subjects are indeed very efficient distributional learners even from small amounts of evidence, we test a DSM on the same multimodal task, finding that it behaves in a remarkable human-like way. We conclude that DSMs provide a convincing computational account of word learning even at the early stages in which a word is first encountered, and the way they build meaning representations can offer new insights into human language acquisition. (shrink)

Cognitive Science, Volume 46, Issue 4, April 2022.

Traditional medical Artificial Intelligence models, approved for clinical use, restrict themselves to single-modal data e.g. images only, limiting their applicability in the complex, multimodal environment of medical diagnosis and treatment. Multimodal Transformer Models in healthcare can effectively process and interpret diverse data forms such as text, images, and structured data. They have demonstrated impressive performance on standard benchmarks like USLME question banks and continue to improve with scale. However, the adoption of these advanced AI models is not without (...) challenges. While multimodal deep learning models like Transformers offer promising advancements in healthcare, their integration requires careful consideration of the accompanying ethical and environmental challenges. (shrink)

In the era of eLearning 4.0, many researchers have suggested that multimodal input helps to enhance second language vocabulary learning. However, previous studies on the effects of multimodal teaching have failed to yield definitive conclusions. Furthermore, only few studies on the multimodal input of vocabulary learning have aimed at junior high school students and have focused on explicit vocabulary instruction in class. To explore the effects of multimodal input on English as a foreign language (...) learners’ vocabulary learning and summarize effective methods, this study adopts a mixed-method approach. Based on dual coding theory and cognitive load theory, the teaching materials in this study were designed using the resources provided by the multimodal corpus iWeb and other websites. A total of 60 junior high school students who learned EFL and had a similar English proficiency level were divided into an experimental group and a control group. Target words were selected through questionnaire I. During the experiment, the CG learned from monomodal materials while the EG received multimodal input, and an immediate post-test was delivered to the two groups. Questionnaire II was distributed in the EG, and five students of the EG were randomly selected for an interview. One week later, a delayed post-test was conducted on the EG and CG. The results showed that the EG performed better in the post-test but did worse than the CG in the delayed post-test. The results of the questionnaire and the interview suggest that students held both positive and negative attitudes toward the multimodal input approach in vocabulary learning. The study concludes with some implications for choosing a multimodal input approach in vocabulary learning, along with a number of suggestions on how to optimize its positive influence and minimize its negative effects. (shrink)

Digitization of every daily procedure requires trustworthy verification schemes. People tend to overlook the security of the passwords they use, i.e. they use the same password on different occasions, they neglect to change them periodically or they often forget them. This raises a major security issue, especially for elderly people who are not familiar with modern technology and its risks and challenges. To overcome these drawbacks, biometric factors were utilized, and nowadays, they have been widely adopted due to their convenience (...) of use and ease of hardware installation. Many biometric-based authentication schemes were proposed, but despite the advantages that they offer, recent research has shown that biometrics by itself cannot be considered as an inviolable technique. Recently, we have proposed StegoPass, a novel method that obtains the 68 facial points of a user and utilizes them as a stego message to an image. This produced stego image was the password. Although the experiments conducted showed maximum security, it would be challenging to enhance the robustness of the proposed model for even more attacks. This paper examines the utilization of multimodal biometrics as the secret message embedded in the image. More specifically, besides the extraction of the facial points, we extract the unique minutiae moments and combine them in a feature vector. This feature vector is then embedded in the image. Two different datasets were used, and the security of the method was tested against state-of-the-art deep learning models, i.e. generative adversarial networks, to test whether the image could be digitally synthesized and fool the proposed verification scheme. The results proved that the new enhanced version of StegoPass offers an extremely secure method as its predecessor. (shrink)

Multimodal communication research focuses on how different means of signalling coordinate to communicate effectively. This line of research is traditionally influenced by fields such as cognitive and neuroscience, human-computer interaction, and linguistics. With new technologies becoming available in fields such as natural language processing and computer vision, the field can increasingly avail itself of new ways of analyzing and understanding multimodal communication. As a result, there is a general hope that multimodal research may be at the “precipice (...) of greatness” due to technological advances in computer science and resulting extended empirical coverage. However, for this to come about there must be sufficient guidance on key (theoretical) needs of innovation in the field of multimodal communication. Absent such guidance, the research focus of computer scientists might increasingly diverge from crucial issues in multimodal communication. With this paper, we want to further promote interaction between these fields, which may enormously benefit both communities. The multimodal research community (represented here by a consortium of researchers from the Visual Communication [ViCom] Priority Programme) can engage in the innovation by clearly stating which technological tools are needed to make progress in the field of multimodal communication. In this article, we try to facilitate the establishment of a much needed common ground on feasible expectations (e.g., in terms of terminology and measures to be able to train machine learning algorithms) and to critically reflect possibly idle hopes for technical advances, informed by recent successes and challenges in computer science, social signal processing, and related domains. (shrink)

This book positions itself at the intersection of the key areas of the modern humanities. Different authors from a variety of countries take innovative approaches to investigating multimodal communication, adapting pedagogical design to digital environments and enhancing cognitive skills through transformations in teaching and learning practices. The eclectic forms under study require eclectic approaches and methodologies, and the authors cross disciplinary boundaries drawing on philosophy, linguistics, semiotics, computational linguistics, mathematics, cognitive studies and neuroaesthetics. Part I presents methods of (...) analysing multimodal communication in its different displays, covering promotional video in crowdfunding project presentations, multimodal public signs of prohibition and visuals as arguments. Part II explores varied teaching methodologies that have emerged as a result of and in response to modern technological changes and contains some practical hints for educators. It demonstrates the pedagogical potential of video games, virtual worlds, linguistic corpora and online dictionaries. Part III focuses on psychological and cognitive factors influencing success in the classroom, primarily, ways of developing students’ and teachers’ personalities. The volume sits at the intersection between Communication Studies, Digital Humanities, Discourse Analysis, Education Theory and Cognitive Studies and is useful to scholars and students of communication, languages, education and other areas of the humanities. This book should trigger scholarly discussions as well as stimulating practitioners’ interest in these fields. (shrink)

Dancers and musicians are experts in spatial and temporal processing which allows them to coordinate movement with music. This high-level processing has been associated with structural and functional adaptation of the brain for high performance sensorimotor integration. For these integration processes, adaptation does not only take place in primary and secondary sensory and motor areas but tertiary brain areas, such as the lateral prefrontal cortex (lPFC) and the intraparietal sulcus (IPS), also provide vital resources for highly specialized performance. Here, we (...) review evidence for the role of these brain areas in multimodal training protocols and integrate these findings into a new model of sensorimotor processing in complex motor learning. (shrink)

Electroencephalography and functional near-infrared spectroscopy have potentially complementary characteristics that reflect the electrical and hemodynamic characteristics of neural responses, so EEG-fNIRS-based hybrid brain-computer interface is the research hotspots in recent years. However, current studies lack a comprehensive systematic approach to properly fuse EEG and fNIRS data and exploit their complementary potential, which is critical for improving BCI performance. To address this issue, this study proposes a novel multimodal fusion framework based on multi-level progressive learning with multi-domain features. The (...) framework consists of a multi-domain feature extraction process for EEG and fNIRS, a feature selection process based on atomic search optimization, and a multi-domain feature fusion process based on multi-level progressive machine learning. The proposed method was validated on EEG-fNIRS-based motor imagery and mental arithmetic tasks involving 29 subjects, and the experimental results show that multi-domain features provide better classification performance than single-domain features, and multi-modality provides better classification performance than single-modality. Furthermore, the experimental results and comparison with other methods demonstrated the effectiveness and superiority of the proposed method in EEG and fNIRS information fusion, it can achieve an average classification accuracy of 96.74% in the MI task and 98.42% in the MA task. Our proposed method may provide a general framework for future fusion processing of multimodal brain signals based on EEG-fNIRS. (shrink)

With the development of artificial intelligence technologies, it is possible to use computer to read digital medical images. Because Alzheimer’s disease has the characteristics of high incidence and high disability, it has attracted the attention of many scholars, and its diagnosis and treatment have gradually become a hot topic. In this paper, a multimodal diagnosis method for AD based on three-dimensional shufflenet and principal component analysis network is proposed. First, the data on structural magnetic resonance imaging and functional magnetic (...) resonance imaging are preprocessed to remove the influence resulting from the differences in image size and shape of different individuals, head movement, noise, and so on. Then, the original two-dimensional ShuffleNet is developed three-dimensional, which is more suitable for 3D sMRI data to extract the features. In addition, the PCANet network is applied to the brain function connection analysis, and the features on fMRI data are obtained. Next, kernel canonical correlation analysis is used to fuse the features coming from sMRI and fMRI, respectively. Finally, a good classification effect is obtained through the support vector machines method classifier, which proves the feasibility and effectiveness of the proposed method. (shrink)

In the rapidly evolving landscape of multimodal communication research, this paper explores the transformative role of machine learning (ML), particularly using multimodal large language models, in tracking, augmenting, annotating, and analyzing multimodal data. Building upon the foundations laid in our previous work, we explore the capabilities that have emerged over the past years. The integration of ML allows researchers to gain richer insights from multimodal data, enabling a deeper understanding of human (and non-human) communication across (...) modalities. In particular, augmentation methods have become indispensable because they facilitate the synthesis of multimodal data and further increase the diversity and richness of training datasets. In addition, ML-based tools have accelerated annotation processes, reducing human effort while improving accuracy. Continued advances in ML and the proliferation of more powerful models suggest even more sophisticated analyses of multimodal communication, e.g., through models like ChatGPT, which can now “understand” images. This makes it all the more important to assess what these models can achieve now or in the near future, and what will remain unattainable beyond that. We also acknowledge the ethical and practical challenges associated with these advancements, emphasizing the importance of responsible AI and data privacy. We must be careful to ensure that benefits are shared equitably and that technology respects individual rights. In this paper, we highlight advances in ML-based multimodal research and discuss what the near future holds. Our goal is to provide insights into this research stream for both the multimodal research community, especially in linguistics, and the broader ML community. In this way, we hope to foster collaboration in an area that is likely to shape the future of technologically mediated human communication. (shrink)

This book envisions a relational and multimodal conceptualisation of higher education, which advocates for knowledge to be analyzed under three relational dimensions: social, digital and environmental. The volume draws on interdisciplinary approaches grounded in Peirceian semiotics, in exploring an integration of these dimensions in higher education theory and practice. The book situates learning in an awareness of the environment, grounded in principles of interconnectedness and flattened hierarchies. The volume features practical case studies through dialogues with higher education teachers, (...) presenting different ways of practicing multimodal and relational higher education which prioritize and embrace expression and creation through more than one dominant modality of communication. The book highlights how a relational multimodal paradigm can serve as a way forward for universities to address global challenges concerning social, technological, and environmental futures through a social justice lens. This innovative book will be of interest to students and scholars in semiotics, multimodality, and higher education, as well as stakeholders in these fields. (shrink)

This paper presents an implemented computational model of word acquisition which learns directly from raw multimodal sensory input. Set in an information theoretic framework, the model acquires a lexicon by finding and statistically modeling consistent cross‐modal structure. The model has been implemented in a system using novel speech processing, computer vision, and machine learning algorithms. In evaluations the model successfully performed speech segmentation, word discovery and visual categorization from spontaneous infant‐directed speech paired with video images of single objects. (...) These results demonstrate the possibility of using state‐of‐the‐art techniques from sensory pattern recognition and machine learning to implement cognitive models which can process raw sensor data without the need for human transcription or labeling. (shrink)

Interpreting a seemingly simple function word like “or,” “behind,” or “more” can require logical, numerical, and relational reasoning. How are such words learned by children? Prior acquisition theories have often relied on positing a foundation of innate knowledge. Yet recent neural‐network‐based visual question answering models apparently can learn to use function words as part of answering questions about complex visual scenes. In this paper, we study what these models learn about function words, in the hope of better understanding how the (...) meanings of these words can be learned by both models and children. We show that recurrent models trained on visually grounded language learn gradient semantics for function words requiring spatial and numerical reasoning. Furthermore, we find that these models can learn the meanings of logical connectives and and or without any prior knowledge of logical reasoning as well as early evidence that they are sensitive to alternative expressions when interpreting language. Finally, we show that word learning difficulty is dependent on the frequency of models' input. Our findings offer proof‐of‐concept evidence that it is possible to learn the nuanced interpretations of function words in a visually grounded context by using non‐symbolic general statistical learning algorithms, without any prior knowledge of linguistic meaning. (shrink)

An increasing number of research teams are investigating the efficacy of brain-computer interface (BCI)-mediated interventions for promoting motor recovery following stroke. A growing body of evidence suggests that of the various BCI designs, most effective are those that deliver functional electrical stimulation (FES) of upper extremity (UE) muscles contingent on movement intent. More specifically, BCI-FES interventions utilize algorithms that isolate motor signals—user-generated intent-to-move neural activity recorded from cerebral cortical motor areas—to drive electrical stimulation of individual muscles or muscle synergies. BCI-FES (...) interventions aim to recover sensorimotor function of an impaired extremity by facilitating and/or inducing long-term motor learning-related neuroplastic changes in appropriate control circuitry. We developed a non-invasive, electroencephalogram (EEG)-based BCI-FES system that delivers closed-loop neural activity-triggered electrical stimulation of targeted distal muscles while providing the user with multimodal sensory feedback. This BCI-FES system consists of three components: (1) EEG acquisition and signal processing to extract real-time volitional and task-dependent neural command signals from cerebral cortical motor areas, (2) FES of muscles of the impaired hand contingent on the motor cortical neural command signals, and (3) multimodal sensory feedback associated with performance of the behavioral task, including visual information, linked activation of somatosensory afferents through intact sensorimotor circuits, and electro-tactile stimulation of the tongue. In this report, we describe device parameters and intervention protocols of our BCI-FES system which, combined with standard physical rehabilitation approaches, has proven efficacious in treating UE motor impairment in stroke survivors, regardless of level of impairment and chronicity. (shrink)

Mary Kalantzis was from 2006 to 2016 Dean of the College of Education at the University of Illinois, Urbana-Champaign. Bill Cope is a Professor in the College of Education at the University of Illinois, Urbana-Champaign. They are co-authors of multiple books including Making Sense: Reference, Agency, and Structure in a Grammar of Multimodal Meaning (Cambridge, forthcoming), New Learning: Elements of a Science of Education (Cambridge, 2008, 2012), Literacies (Cambridge 2012, 2016) and e-Learning Ecologies (2017).

Open peer commentary on the article “A Cybernetic Approach to Contextual Teaching and Learning” by Philip Baron. Upshot: Universities in South Africa and elsewhere are being challenged to prepare graduates for the world of work and to recognize the multitude of resources that students bring with them to the higher-education experience. A multimodal approach to pedagogy may assist in both of these regards.

Esta herramienta didáctica con tecnología de realidad aumentada ha aparecido recientemente como un nuevo recurso en los últimos años. En esta investigación cualitativa pretendemos conocer qué percepciones obtienen 72 futuros docentes de Educación Infantil una vez que experimentan por sí mismos estos formatos digitales multimodales. Entre las conclusiones, consideran que el cuento de RA es una herramienta multimodal con muchas cualidades, apta para ser introducida a los infantes, jugando un papel motivador para el desarrollo de la lectura. Además, estos (...) cuentos en 3D hacen que los niños aprendan mientras se divierten. (shrink)

A teacher’s use of classroom space via embodied movement enacts a specific spatial pedagogy and has a significant impact on the nature of teaching and learning that can take place. The ubiquity and large expenditure in renovating university learning environments have not been accompanied by well-informed research due to a tendency to background embodied movement in pedagogic investigations. This paper explores the semiotic potentials and pedagogic functions of one teacher’s embodied movement in an Active Learning Classroom in (...) a tertiary setting. It maps out the interrelationship of embodied movement and space as an entanglement of materiality and practice, which complements existing research that relates movement to space only in terms of layout and composition. By conducting a nuanced multimodal analysis of one teacher’s co-patterns of movement and speech in a tertiary lesson, this paper finds that co-patterns of embodied movement and speech in the classroom can segment phases of pedagogic discourse into five secondary phases and that these phases congruently enact different types of space to modulate the teacher-student relationship and enact different pedagogic roles. The paper concludes with a discussion of the implication of embodied movement on classroom design and pedagogic practice. (shrink)

While influential works since the 1970s have widely assumed that imitation is an innate skill in both human and non-human primate neonates, recent empirical studies and meta-analyses have challenged this view, indicating other forms of reward-based learning as relevant factors in the development of social behavior. The visual input translation into matching motor output that underlies imitation abilities instead seems to develop along with social interactions and sensorimotor experience during infancy and childhood. Recently, a new visual stream has been (...) identified in both human and non-human primate brains, updating the dual visual stream model. This third pathway is thought to be specialized for dynamics aspects of social perceptions such as eye-gaze, facial expression and crucially for audio-visual integration of speech. Here, we review empirical studies addressing an understudied but crucial aspect of speech and communication, namely the processing of visual orofacial cues and its integration with vocal auditory cues. Along this review, we offer new insights from our understanding of speech as the product of evolution and development of a rhythmic and multimodal organization of sensorimotor brain networks, supporting volitional motor control of the upper vocal tract and audio-visual voices-faces integration. (shrink)

Emotion recognition is an important part of building an intelligent human-computer interaction system and plays an important role in human-computer interaction. Often, people express their feelings through a variety of symbols, such as words and facial expressions. A business ecosystem is an economic community based on interacting organizations and individuals. Over time, they develop their capabilities and roles together and tend to develop themselves in the direction of one or more central enterprises. This paper aims to study a multimodal (...) ER method based on attention mechanism. It analyzes the current emotional state of consumers and the development direction of enterprises through multi-modal ER of human emotions and analysis of market trends, so as to provide the most appropriate response or plan. This paper firstly describes the related methods of multimodal ER and deep learning in detail, and briefly outlines the meaning of enterprise strategy in the business ecosystem. Then, two datasets, CMU-MOSI and CMU-MOSEI, are selected to design the scheme for multimodal ER based on self-attention mechanism. Through the comparative analysis of the accuracy of single-modal and multi-modal ER, the self-attention mechanism is applied in the experiment. The experimental results show that the average recognition accuracy of happy under multimodal ER reaches 91.5%. (shrink)

The purpose of this study is to discuss the effects of audiovisual input on second language acquisition and the factors that influence the difficulty of audiovisual learning through a systematic literature review. Prior to this systematic review, in this paper, we searched papers on related topics for the past 10 years from 2012 to 2022, and found 46 journal papers that met the research criteria. They can basically represent the scholarly work in this field. The 46 studies were published (...) in journals indexed in Google Scholar, Eric, Scopus, and Wiley Library. Databases were selected according to a set of inclusion and exclusion criteria. The following conclusions are drawn from the literature review: Audiovisual input can provide more authentic language input and more adequate and richer multimodal cultural and situational contexts, which can better promote learners' understanding of the content and stimulate learners' interest in participating in listening comprehension tasks. The influencing factors of multimodal input on listening difficulty include subtitles, video type, and the relationship between the audio and visual input. (shrink)

The social brain hypothesis implies that humans and other primates evolved “modules“ for representing social knowledge. Alternatively, no such cognitive specializations are needed because social knowledge is already present in the world — we can simply monitor the dynamics of social interactions. Given the latter idea, what mechanism could account for coalition formation? We propose that statistical learning can provide a mechanism for fast and implicit learning of social signals. Using human participants, we compared learning of social (...) signals with arbitrary signals. We found that learning of social signals was no better than learning of arbitrary signals. While coupling faces and voices led to parallel learning, the same was true for arbitrary shapes and sounds. However, coupling versus uncoupling social signals with arbitrary signals revealed that faces and voices are treated with perceptual priority. Overall, our data suggest that statistical learning is a viable domain-general mechanism for learning social group structure. Keywords: social brain; embodied cognition; distributed cognition; situated cognition; multisensory; audiovisual speech; crossmodal; multimodal. (shrink)

What is the experience of creating a synergistic approach to arts and sciences practice in a learning community focused on the notion of sustainable development? In this article, I answer this question through an evolutionary approach to societal transformation. My social research inquiry integrates the arts and sciences, a learning and design community, sustainable development, and Internet networking. Codesigners created the conditions to explore in multimodal dialogue and engage, guide, and design the emergence of what I call (...) evolutionary artscience through portfolios and a new systems design space-the ArtSciLAB (an online artscience laboratory). Eighteen individuals working in nine countries participated in the Evolutionary Learning Community Network. Our learning and design experience, guided by Evolutionary Systems Design inquiry supports General Evolution Theory. We guided the Evolutionary Learning Community Network and designed the ArtSciLAB to incubate localized knowledge for development in learning regions worldwide. (shrink)

How can imitation lead to free musical expression? This article explores the role of auditory imitation in jazz. Even though many renowned jazz musicians have assessed the method of imitating recorded music, no systematic study has hitherto explored how the method prepares for aural jazz improvisation. The article picks up an assumption presented by Berliner (1994), suggesting that learning jazz by aural imitation is “just like” learning a mother tongue. The article studies three potential stages in the method, (...) comparing with imitative, rhythmic, multimodal, and protosymbolic behavior of infant perception (building on the works of Stern, Trevarthen, and Merleau-Ponty). The demonstrations of the aural-imitation method draw on pedagogic experiences accumulated since 1979 in the Jazz Program at the Norwegian University of Science and Technology. By analyzing structures of behavior suggested by the method, the article indicates key traits that render aural jazz improvisation possible, such as a fundamental sense of rhythm, formation of symbolic behavior, joint musical attention, and the facility to “hear via the other.” In conclusion, we critically address a frequent theoretical model describing musical improvisation as a synthesis of discrete elements or building blocks. (shrink)

Emotion recognition plays an important role in intelligent human–computer interaction, but the related research still faces the problems of low accuracy and subject dependence. In this paper, an open-source software toolbox called MindLink-Eumpy is developed to recognize emotions by integrating electroencephalogram and facial expression information. MindLink-Eumpy first applies a series of tools to automatically obtain physiological data from subjects and then analyzes the obtained facial expression data and EEG data, respectively, and finally fuses the two different signals at a decision (...) level. In the detection of facial expressions, the algorithm used by MindLink-Eumpy is a multitask convolutional neural network based on transfer learning technique. In the detection of EEG, MindLink-Eumpy provides two algorithms, including a subject-dependent model based on support vector machine and a subject-independent model based on long short-term memory network. In the decision-level fusion, weight enumerator and AdaBoost technique are applied to combine the predictions of SVM and CNN. We conducted two offline experiments on the Database for Emotion Analysis Using Physiological Signals dataset and the Multimodal Database for Affect Recognition and Implicit Tagging dataset, respectively, and conducted an online experiment on 15 healthy subjects. The results show that multimodal methods outperform single-modal methods in both offline and online experiments. In the subject-dependent condition, the multimodal method achieved an accuracy of 71.00% in the valence dimension and an accuracy of 72.14% in the arousal dimension. In the subject-independent condition, the LSTM-based method achieved an accuracy of 78.56% in the valence dimension and an accuracy of 77.22% in the arousal dimension. The feasibility and efficiency of MindLink-Eumpy for emotion recognition is thus demonstrated. (shrink)

Early number skills represent critical milestones in children's cognitive development and are shaped over years of interacting with quantities and numerals in various contexts. Several connectionist computational models have attempted to emulate how certain number concepts may be learned, represented, and processed in the brain. However, these models mainly used highly simplified inputs and focused on limited tasks. We expand on previous work in two directions: First, we train a model end-to-end on video demonstrations in a synthetic environment with (...) class='Hi'>multimodal visual and language inputs. Second, we use a more holistic dataset of 35 tasks, covering enumeration, set comparisons, symbolic digits, and seriation. The order in which the model acquires tasks reflects input length and variability, and the resulting trajectories mostly fit with findings from educational psychology. The trained model also displays symbolic and non-symbolic size and distance effects. Using techniques from interpretability research, we investigate how our attention-based model integrates cross-modal representations and binds them into context-specific associative networks to solve different tasks. We compare models trained with and without symbolic inputs and find that the purely non-symbolic model employs more processing-intensive strategies to determine set size. (shrink)

In this article, the formation and transformation of knowledge and the role of designs for learning will be elaborated and discussed in relation to the introduction of national curricula and school textbooks during the beginning of the industrialized era vs. the introduction of individual curricula and new digital learning resources in the post-industrialized era of globalization and multiculturalism. Quite different teaching and learning strategies have been emphasized, which I will call here “designed information and teaching” vs. “designs (...) for learning”. It seems obvious that our current society is in a stage of change that requires a new understanding of knowledge, learning and identity formation. The new position and role of the learner underlines the productive and constructive aspect of learning. Pupils not only read texts, they also produce texts, pictures, film and music and they compile and edit virtual texts. Multimodal texts, as well as the information flow of the Internet, are the consequences of, and at the same time a vehicle for, new social patterns. “Learning Design Sequences” (LDS) is introduced as a theoretical map for the purpose of analyzing critical incidents in (a creative) learning process, using different genres, modes and media in a process of meaning-making. (shrink)

There is substantial variation in language experience, yet there is surprising similarity in the language structure acquired. Constraints on language structure may be external modulators that result in this canalization of language structure, or else they may derive from the broader, communicative environment in which language is acquired. In this paper, the latter perspective is tested for its adequacy in explaining robustness of language learning to environmental variation. A computational model of word learning from cross-situational, multimodal information (...) was constructed and tested. Key to the model's robustness was the presence of multiple, individually unreliable information sources to support learning. This “degeneracy” in the language system has a detrimental effect on learning, compared to a noise-free environment, but has a critically important effect on acquisition of a canalized system that is resistant to environmental noise in communication. (shrink)

Reading the Visual is an essential introduction that focuses on what teachers should know about multimodal literacy and how to teach it. This engaging book provides theoretical, curricular, and pedagogical frameworks for teaching a wide-range of visual and multimodal texts, including historical fiction, picture books, advertisements, websites, comics, graphic novels, news reports, and film. Each unit of study presented contains suggestions for selecting cornerstone texts and visual images and launching the unit, as well as lesson plans, text sets, (...) and analysis guides. These units are designed to be readily adapted to fit the needs of a variety of settings and grade levels. (shrink)