ABSTRACTSeveral eye-tracking studies have revealed that spider phobic patients show a typical hypervigilance-avoidance pattern when confronted with images of spiders. The present experiment investigated if this pattern can be changed via placebo treatment. We conducted an eye-tracking experiment with 37 women with spider phobia. They looked at picture pairs for 7 s each in a retest design: once with and once without a placebo pill presented along with the verbal suggestion that it can reduce phobic symptoms. (...) The placebo was labelled as Propranolol, a beta-blocker that has been successfully used to treat spider phobia. In the placebo condition, both the fixation count and the dwell time on the spider pictures increased, especially in the second half of the presentation time. This was associated with a slight decrease in self-reported symptom severity. In summary, we were able to show that a placebo was able to positively influence visual avoidance in spider phobia.... (shrink)

This study investigates the online process of reading and analyzing of sketchnotes (visual notes containing a handwritten text and drawings) on Russian language material. Using the eye-tracking method, we compared the processing of different types of sketchnotes (‘path’ (trajectory), linear, and radial) and the processing of a verbal text. Biographies of Russian writers were used as the material. In a preliminary experiment, we asked 89 college students to read the biographies and to evaluate each text or sketch using (...) five scales (from -2 to +2). The best example for each of three formats of sketchnotes and a verbal text were chosen. In the main experiment, twenty-one secondary school students examined four different biographies in four different formats (three sketchnotes and a verbal text), answered to the factual and analytical questions to these texts and estimated the difficulty of each text. We measured the total dwell time, the total fixation count, the average fixation duration for each stimulus as well as for separate zones inside the sketches including verbal and non-verbal information. Our results show that readers process the information better and faster while reading sketchnotes than a verbal text. In the trajectory sketchnotes, the readers followed the order of elements aimed by the author of the sketchnotes better than in the radial and linear sketchnotes. The analysis of participants’ eye movements while processing the stimuli made it possible to propose several recommendations for creating effective sketchnotes. (shrink)

Multimodal Critical Discourse Analysis (MCDA) as a sub-discipline of Critical Discourse Analysis (CDA) emerged from the availability of social semiotic frameworks describing multimodal meaning making. However, weaknesses of these frameworks have raised concerns and prompted recent methodological reflections in MCDA. Inspired by these reflections, this paper critically assesses MCDA research on advertising persuasion and identifies a lack of attention in studies to account for the social and ideological impact of advertising. This shortcoming is argued to be attributable to the weak (...) empirical grounding of social semiotic frameworks which in turn is rooted in a ‘discursive gap’ (Bateman, Citation2019, p. 300) in multimodality as a discipline. The authors then propose methodological solutions involving eye tracking experiments and higher order data analytics which can strengthen the empirical basis of theoretical assumptions made in MCDA research on advertising. Drawing on methodological guidance for empirical multimodal research, the authors discuss specific issues in realizing this proposal, focusing on a strategy addressing operationalization challenges in experimental design and the affordance of a data cluster algorithm that calculates the common viewing path of a group of viewers. (shrink)

Installation art is one of the most important and provocative developments in the visual arts during the last half century and has become a key focus of artists and of contemporary museums. It is also seen as particularly challenging or even disliked by many viewers, and—due to its unique in situ, immersive setting—is equally regarded as difficult or even beyond the grasp of present methods in empirical aesthetic psychology. In this paper, we introduce an exploratory study with installation art, utilizing (...) a collection of techniques to capture the eclectic, the embodied, and often the emotionally-charged viewing experience. We present results from an investigation of two pieces, both part of Olafur Eliasson’s exhibition “Baroque, Baroque” held at the Belvedere museum in Vienna. These were assessed by pre- and post-viewing questionnaires focusing on emotion, meaning-making, and appraisals, in tandem with mobile eye tracking to consider viewers’ attention to both installed artworks and/or to the museum environment. The data showed differences in participants’ emotional states, appraisals, and visual exploration, which together paint a picture of the aesthetic reactions to the works. These differences also showed how viewers’ appraisal strategies, meaning making, and physical actions facilitated relatively more or less deep engagement with, and enjoyment of, the art. The results are discussed in terms of their implications for museum studies, art education, and theory in empirical aesthetics. (shrink)

Semantic studies on diagrammatic notations (Barwise & Etchemendy, ; Shimojima, ; Stenning & Lemon, ) have revealed that the “non-deductive,” “emergent,” or “perceptual” effects of diagrams (Chandrasekaran, Kurup, Banerjee, Josephson, & Winkler, ; Kulpa, ; Larkin & Simon, ; Lindsay, ) are all rooted in the exploitation of spatial constraints on graphical structures. Thus, theoretically, this process is a key factor in inference with diagrams, explaining the frequently observed reduction of inferential load. The purpose of this study was to examine (...) the empirical basis for this theoretical suggestion, focusing on the reality of the constraint-exploitation strategy in actual practices of diagrammatic reasoning. Eye movements were recorded while participants used simple position diagrams to solve three- or four-term transitive inference problems. Our experiments revealed that the participants could exploit spatial constraints on graphical structures even when (a) they were not in the position of actually manipulating diagrams, (b) the semantic rule for the provided diagrams did not match their preferences, and (c) the constraint-exploitation strategy invited a partly adverse effect. These findings indicate that the hypothesized process is in fact robust, with the potential to broadly account for the inferential advantage of diagrams. (shrink)

Background: Traditional residential architecture often integrates intricate decorative patterns, vital in shaping the overall aesthetic experience. Aim: The purpose of this research is to investigate the visual perception of traditional residential architectural decorations using eye-tracking experiments, focusing on various design elements influence aesthetic appreciation. Methodology: In this study, we used eye-tracking equipment to record participants' gaze patterns and fixation lengths while they viewed images of traditional architectural decorations. 95 participants from diverse demographic backgrounds in the Leizhou Peninsula were (...) involved in the experiments. The eye tracking data was analysed to determine how participants' visual attention varied based on the attractiveness and complexity of the decorative patterns. We examined eye movements to evaluate fixation counts, duration, and pupil dilation using electroencephalogram (EEG). Additionally, subjective feedback was collected through surveys to gauge participants' aesthetic preferences. Result and Conclusion: Findings indicate that more elaborate and visually appealing traditional decorations elicited longer fixation durations and higher levels of engagement compared to simpler designs. This study highlights the significant impact of traditional decorative patterns on visual perception and aesthetic appreciation. These findings can guide the design of modern residential products, ensuring they resonate with cultural aesthetics while enhancing the user experience. (shrink)

This study examined the time-course of context effects on spoken word recognition during Chinese sentence processing. We recruited 60 native Mandarin listeners to participate in an eye-tracking experiment. In this eye-tracking experiment, listeners were told to listen to a sentence carefully, which ended with a Chinese homophone, and look at different visual probes presented concurrently on the computer screen naturally. Different types of context and probe types were manipulated in the experiment. The results showed that (...) preceding sentence context had an early effect on spoken word recognition processes and phonological information of the distracters had only a negligible effect on the spoken word recognition processes. Finally, the patterns of eye-tracking results seemed to favor an interactive approach in spoken word recognition. (shrink)

Experimental aesthetics has shed light on the involvement of pre-motor areas in the perception of abstract art. However, the contribution of texture perception to aesthetic experience is still understudied. We hypothesized that digital screen-based art, despite its immateriality, might suggest potential sensorimotor stimulation. Original born-digital works of art were selected and manipulated by the artist himself. Five behavioral parameters: Beauty, Liking, Touch, Proximity, and Movement, were investigated under four experimental conditions: Resolution (high/low), and Magnitude (Entire image/detail). These were expected to (...) modulate the quantity of material and textural information afforded by the image. While the Detail condition afforded less content-related information, our results show that it augmented the image’s haptic appeal. High Resolution improved the haptic and aesthetic properties of the images. Furthermore, aesthetic ratings positively correlated with sensorimotor ratings. Our results demonstrate a strict relation between the aesthetic and sensorimotor/haptic qualities of the images, empirically establishing a relationship between beholders’ bodily involvement and their aesthetic judgment of visual works of art. In addition, we found that beholders’ oculomotor behavior is selectively modulated by the perceptual manipulations being performed. The eye-tracking results indicate that the observation of the Entire, original images is the only condition in which the latency of the first fixation is shorter when participants gaze to the left side of the images. These results thus demonstrate the existence of a left-side bias during the observation of digital works of art, in particular, while participants are observing their original version. (shrink)

Eye-tracking technology has become a powerful tool for biomedical-related applications due to its simplicity of operation and low requirements on patient language skills. This study aims to use the machine-learning models and deep-learning networks to identify key features of eye movements in Alzheimer's Disease (AD) under specific visual tasks, thereby facilitating computer-aided diagnosis of AD. Firstly, a three-dimensional (3D) visuospatial memory task is designed to provide participants with visual stimuli while their eye-movement data are recorded and used to build (...) an eye-tracking dataset. Then, we propose a novel deep-learning-based model for identifying patients with Alzheimer's Disease (PwAD) and healthy controls (HCs) based on the collected eye-movement data. The proposed model utilizes a nested autoencoder network to extract the eye-movement features from the generated fixation heatmaps and a weight adaptive network layer for the feature fusion, which can preserve as much useful information as possible for the final binary classification. To fully verify the performance of the proposed model, we also design two types of models based on traditional machine-learning and typical deep-learning for comparison. Furthermore, we have also done ablation experiments to verify the effectiveness of each module of the proposed network. Finally, these models are evaluated by four-fold cross-validation on the built eye-tracking dataset. The proposed model shows 85% average accuracy in AD recognition, outperforming machine-learning methods and other typical deep-learning networks. (shrink)

This study investigated the relative role of sub-syllabic components in spoken word recognition of Mandarin Chinese using an eye-tracking experiment with a visual world paradigm. Native Mandarin speakers were presented with four pictures and an auditory stimulus. They were required to click the picture according to the sound stimulus they heard, and their eye movements were tracked during this process. For a target word, nine conditions of competitors were constructed in terms of the amount of their phonological overlap (...) with the target: consonant competitor, rime competitor, tone competitor, consonant plus rime competitor, consonant plus tone competitor, rime plus tone competitor, cohort competitor, cohort plus tone competitor, and baseline competitor. A growth curve analysis was conducted with the fixation to competitors, targets, and distractors, and the results showed that competitors with consonant or rime overlap can be adequately activated, while tone overlap plays a weaker role since additional tonal information can strengthen the competitive effect only when it was added to a candidate that already bears much phonological similarity with the target. Mandarin words are processed in an incremental way in the time course of word recognition since different partially overlapping competitors could be activated immediately; like the pattern found in English, both cohort and rime competitors were activated to compete for lexical activation, but these two competitors were not temporally distinctive and mainly differed in the size of their competitive effects. Generally, the gradation of activation based on the phonological similarity between target and candidates found in this study was in line with the continuous mapping models and may reflect a strategy of native speakers shaped by the informative characteristics of the interaction among different sub-syllabic components. (shrink)

Cognitive architectures have often been applied to data from individual experiments. In this paper, I develop an ACT-R reader that can model a much larger set of data, eye-tracking corpus data. It is shown that the resulting model has a good fit to the data for the considered low-level processes. Unlike previous related works, the model achieves the fit by estimating free parameters of ACT-R using Bayesian estimation and Markov-Chain Monte Carlo techniques, rather than by relying on the mix (...) of manual selection + default values. The method used in the paper is generalizable beyond this particular model and data set and could be used on other ACT-R models. (shrink)

A number of studies in the extant literature report findings that suggest asymmetry in the way reflexive and pronoun anaphors are interpreted in the early stages of processing: that pronouns are less sensitive to structural constraints, as formulated by Binding Theory, than reflexives, in the initial antecedent retrieval process. However, in previous visual world paradigm eye-tracking studies, these conclusions were based on sentences that placed the critical anaphors within picture noun phrases or prepositional phrases, which have independently been shown (...) not to neatly conform to the Binding Theory principles. We present results from a visual world paradigm eye-tracking experiment that show that when critical anaphors are placed in the indirect object position immediately following a verb as a recipient argument, pronoun and reflexive processing are equally sensitive to structural constraints. (shrink)

Modern consumers face a dramatic rise in web-based technological advancements and have trouble making rational and proper decisions when they shop online. When they try to make decisions about products and services, they also feel pressured against time when sorting among all of the unnecessary items in the flood of information available on the web. In this sense, they need to use consumer decision-making creativity to make rational decisions. However, unexplored research questions on this subject remain. First, in what ways (...) do task difficulty and time constraints affect visual attention on exploitative and exploratory activities differently? Second, how does the location of the reference influence the level of visual attention to exploitative and exploratory activities depending on affordance theory? Third, how do exploratory and exploitative activities affect CDMC? Eye-tracking experiments were conducted with 70 participants to obtain relevant metrics such as total fixation duration, fixation count, and visit count to answer these research questions. Our findings suggest that task difficulty influences exploitative activity, whereas time constraint is related to the exploratory activity. The result of the location of hints aligns with the affordance theory for the exploitative activity. Besides, exploratory activity positively affected CDMC, but exploitative activity did not show any effect. (shrink)

Two studies investigated the interaction between utterance and scene processing by monitoring eye movements in agent–action–patient events, while participants listened to related utterances. The aim of Experiment 1 was to determine if and when depicted events are used for thematic role assignment and structural disambiguation of temporarily ambiguous English sentences. Shortly after the verb identified relevant depicted actions, eye movements in the event scenes revealed disambiguation. Experiment 2 investigated the relative importance of linguistic/world knowledge and scene information. When (...) the verb identified either only the stereotypical agent of a (nondepicted) action, or the (nonstereotypical) agent of a depicted action as relevant, verb‐based thematic knowledge and depicted action each rapidly influenced comprehension. In contrast, when the verb identified both of these agents as relevant, the gaze pattern suggested a preferred reliance of comprehension on depicted events over stereotypical thematic knowledge for thematic interpretation. We relate our findings to language comprehension and acquisition theories. (shrink)

Theorists have debated whether our ability to understand sarcasm (pertaining here to verbal irony) is principally determined by the context or by properties of the comment itself. The current research investigated an alternative view that broadens the focus on the comment itself, suggesting that mitigating a highly positive concept by using negation generates sarcastic interpretations by default. In the current study, pretests performed on the target utterances presented in isolation established their default interpretations; novel affirmative phrases (e.g., He is the (...) best lawyer) were interpreted literally, whereas equally novel negative counterparts (e.g., He isn’t the best lawyer) were interpreted sarcastically. In Experiment 1 (an eye-tracking study), prior context biased these utterances toward literal or sarcastic interpretations. Results showed that target utterances were easier to process in contexts supporting their default interpretations, regardless of affirmation/negation. Results from a second eye-tracking experiment suggested that readers’ tendency to interpret negative phrases sarcastically is related to their own tendency to use malicious humor. Our findings suggest that negation leads to certain ambiguous utterances receiving sarcastic interpretations by default and that this process may be further intensified by personality factors. (shrink)

The aim of this study was to find out how people process the dialectal variation encountered in the daily linguistic input. We conducted an eye tracking study (Visual Word Paradigm) that targeted the predictive processing of grammatical gender markers. Three different groups of Norwegian speakers took part in the experiment: one group of students from the capital Oslo, and two groups of dialect speakers from the Western Norwegian town Sogndal. One Sogndal group was defined as ``stable dialect speakers'', (...) and one as ``unstable dialect speakers'', based on a background questionnaire. The students participated in two eye tracking experiments each, one conducted in the their own dialect, and one in the other dialect (i.e., Sogn dialect for the Oslo students, and Oslo dialect for the Sogndal students). The gender systems in the two dialects differ: the Sogn dialect makes an obligatory three-gender split (Masculine, Feminine and Neuter) whereas the Oslo dialect only obligatorily makes a two gender distinction. The research question was whether speakers could use gender markers to predict the upcoming target noun in both local and non-local dialect mode, and furthermore, if they correctly could adjust their expectations based on dialect mode. The results showed that the Sogn speakers could predict upcoming linguistic material both in the local and Oslo dialect, but only the stable group were able to adjust their predictions based on the dialect mode. The unstable group applied a more general Oslo-compatible parsing to both the local and the non-local dialect. The Oslo speakers on the other hand were able to use gender markers as predictors only in their own dialect. We argue that the stable Sogn group should be treated as a bilingual group, as they show native-like skills in both varieties, while the unstable Sogn group can be seen as accommodated monolinguals, in that they treat the two varieties as sharing an underspecified grammar. The Oslo group on the other hand lacks sufficient competence in the other dialect to make use of grammatical markers to make predictions. (shrink)

Two visual world experiments investigated the processing of the implicature associated with some using a “gumball paradigm.” On each trial, participants saw an image of a gumball machine with an upper chamber with orange and blue gumballs and an empty lower chamber. Gumballs dropped to the lower chamber, creating a contrast between a partitioned set of gumballs of one color and an unpartitioned set of the other. Participants then evaluated spoken statements, such as “You got some of the blue gumballs.” (...) Experiment 1 investigated the time course of the pragmatic enrichment from some to not all when the only utterance alternatives available to refer to the different sets were some and all. In Experiment 2, the number terms two, three, four, and five were also included in the set of alternatives. Scalar implicatures were delayed relative to the interpretation of literal statements with all only when number terms were available. The results are interpreted as evidence for a constraint-based account of scalar implicature processing. (shrink)

In cue-based content-addressable approaches to memory, a target and its competitors are retrieved in parallel from memory via a fast, associative cue-matching procedure under a severely limited focus of attention. Such a parallel matching procedure could in principle ignore the serial order or hierarchical structure characteristic of linguistic relations. I present an eye tracking while reading experiment that investigates whether the sentential position of a potential antecedent modulates the strength of similarity-based interference, a well-studied effect in which increased (...) similarity in features between a target and its competitors results in slower and less accurate retrieval overall. The manipulation trades on an independently established Locality bias in sluiced structures to associate a wh -remnant ( which ones ) in clausal ellipsis with the most local correlate ( some wines ), as in The tourists enjoyed some wines, but I don't know which ones. The findings generally support cue-based parsing models of sentence processing that are subject to similarity-based interference in retrieval, and provide additional support to the growing body of evidence that retrieval is sensitive to both the structural position of a target antecedent and its competitors, and the specificity or diagnosticity of retrieval cues. (shrink)

Existing research shows that people can improve their decision skills by learning what experts paid attention to when faced with the same problem. However, in domains like financial education, effective instruction requires frequent, personalized feedback given at the point of decision, which makes it time‐consuming for experts to provide and thus, prohibitively costly. We address this by demonstrating an automated feedback mechanism that allows amateur decision‐makers to learn what information to attend to from one another, rather than from an expert. (...) In the first experiment, eye movements of N = 100 subjects were recorded while they repeatedly performed a standard behavioral finance investment task. Consistent with previous studies, we found that a significant proportion of subjects were affected by decision bias. In the second experiment, a different group of N = 100 subjects faced the same task but, after each choice, they received individual, machine learning‐generated feedback on whether their pre‐decision eye movements resembled those made by Experiment 1 subjects prior to good decisions. As a result, Experiment 2 subjects learned to analyze information similarly to their successful peers, which in turn reduced their decision bias. Furthermore, subjects with low Cognitive Reflection Test scores gained more from the proposed form of process feedback than from standard behavioral feedback based on decision outcomes. (shrink)

Both linguistic and extralinguistic consultations are essential in translation practice and have been commonly investigated as an integral topic in previous studies. However, since extralinguistic information is usually longer in extent and not specifically designed for a linguistic purpose, extralinguistic consultations involve different search strategies compared with linguistic consultations. Drawing on eye-tracking and screen-recording data, this study compares linguistic and extralinguistic consultations in terms of cognitive resources allocation and information processing patterns in English–Chinese translation. It also explores the differences (...) among 17 language learners, 20 student translators, and 21 professional translators, and the effect of extralinguistic consultation on their translation quality. The findings are as follows: all participants allocate more attention and lower cognitive load to extralinguistic consultations than to linguistic consultations; participants’ translation experience levels and their attention allocated to extralinguistic consultation show an inverted U-shaped relationship; and participants who consult extralinguistic information before drafting or devote more attention to extralinguistic consultation produce target texts with significantly higher scores. (shrink)

This article presents two studies that deepen the theme of how soundtracks shape our interpretation of audiovisuals. Embracing a multivariate perspective, Study 1 ( N = 118) demonstrated, through an online between-subjects experiment, that two different music scores (melancholic vs. anxious) deeply affected the interpretations of an unknown movie scene in terms of empathy felt toward the main character, impressions of his personality, plot anticipations, and perception of the environment of the scene. With the melancholic music, participants felt empathy (...) toward the character, viewing him as more agreeable and introverted, more oriented to memories than to decisions, while perceiving the environment as cozier. An almost opposite pattern emerged with the anxious music. In Study 2 ( N = 92), we replicated the experiment in our lab but with the addition of eye-tracking and pupillometric measurements. Results of Study 1 were largely replicated; moreover, we proved that the anxious score, by increasing the participants’ vigilance and state of alert (wider pupil dilation), favored greater attention to minor details, as in the case of another character who was very hard to be noticed (more time spent on his figure). Results highlight the pervasive nature of the influence of music within the process of interpretation of visual scenes. (shrink)

Immersive 360-degree video has become a new learning resource because of its immersive sensory experience. This study examined the effects of textual and visual cues on learning and attention in immersive 360-degree video by using eye-tracking equipment integrated in a virtual reality head-mounted display. Participants were randomly assigned to one of four conditions: no cues, textual cues in the initial field of view, textual cues outside the initial FOV, and textual cues outside the initial FOV + visual cues. The (...) results showed that the cues helped learners achieve better learning outcomes and spend more time focusing on the areas with cues. In addition, the study found a serious imbalance in the distribution of learners’ attention in each region of the video. The attention directed to textual cues in the initial FOV is much higher than the attention directed to textual cues outside the initial FOV. Adding visual cues can effectively direct attention to textual cues outside the initial FOV and alleviate the imbalance of attention distribution. Consequently, adding cues to immersive 360-degree video can be an appropriate approach to promote learning and guide attention in immersive 360-degree video learning environments. This study provided new insights into the design and development of immersive 360-degree video instructional resources. (shrink)

In this study, we investigate the processing of object-denoting emoji in sentences using eye tracking. We hypothesize that (a) such emoji are more difficult to process when used as word replacement; and (b) their processing is subject to ambiguity constraints similarly to what happens with words. We conduct two experiments in which participants have to read sentences in which an emoji either follows or replaces a word. Control stimuli not containing emoji are also tested. In the second experiment, (...) the emoji are presented in two different disambiguating contexts. First fixation duration, total visit duration, and total revisit duration are modeled in the various conditions using linear mixed models. Both our hypotheses are supported. We observe longer total visit time for non-redundant emoji, and higher values for all three measures for ambiguous emoji. We conclude that lexical access may be more difficult for emoji especially when they are used as word replacements and their meaning is not immediately clear. Furthermore, we also conclude that non-redundant emoji are more difficult to integrate in the processing of the sentence than the equivalent words, or emoji used in a redundant way. In turn, our results indicate that emoji may not always be as immediate and easy to process for readers in spite of their popularity. (shrink)

Recent advances in neuroscience indicate that analysis of bio-signals such as rest state electroencephalogram and eye-tracking data can provide more reliable evaluation of children autism spectrum disorder than traditional methods of behavior measurement relying on scales do. However, the effectiveness of the new approaches still lags behind the increasing requirement in clinical or educational practices as the “bio-marker” information carried by the bio-signal of a single-modality is likely insufficient or distorted. This study proposes an approach to joint analysis of (...) EEG and eye-tracking for children ASD evaluation. The approach focuses on deep fusion of the features in two modalities as no explicit correlations between the original bio-signals are available, which also limits the performance of existing methods along this direction. First, the synchronization measures, information entropy, and time-frequency features of the multi-channel EEG are derived. Then a random forest applies to the eye-tracking recordings of the same subjects to single out the most significant features. A graph convolutional network model then naturally fuses the two group of features to differentiate the children with ASD from the typically developed subjects. Experiments have been carried out on the two types of the bio-signals collected from 42 children. The results indicate that the proposed approach can achieve an accuracy of 95% in ASD detection, and strong correlations exist between the two bio-signals collected even asynchronously, in particular the EEG synchronization against the face related/joint attentions in terms of covariance. (shrink)

We investigated the effects of everyday language exposure on the prediction of orthographic and phonological forms of a highly predictable word during listening comprehension. Native Japanese speakers in Tokyo (Experiment 1) and Berlin (Experiment 2) listened to sentences that contained a predictable word and viewed four objects. The critical object represented the target word (e.g., /sakana/;fish), an orthographic competitor (e.g., /tuno/;horn), a phonological competitor (e.g., /sakura/;cherry blossom), or an unrelated word (e.g., /hon/;book). The three other objects were distractors. (...) The Tokyo group fixated the target and the orthographic competitor over the unrelated objects before the target word was mentioned, suggesting that they pre-activated the orthographic form of the target word. The Berlin group showed a weaker bias toward the target than the Tokyo group, and they showed a tendency to fixate the orthographic competitor only when the orthographic similarity was very high. Thus, prediction effects were weaker in the Berlin group than in the Tokyo group. We found no evidence for the prediction of phonological information. The obtained group differences support probabilistic models of prediction, which regard the built-up language experience as a basis of prediction. (shrink)

The literature on irony processing mainly focused on contextual effect, leaving other factors untouched. The current study investigated how social status information affected the online comprehension of irony. As irony might be more damaging when a speaker uses it to a superordinate than the other way around, it is assumed that greater processing efforts would be observed in the former case. Using an eye-movement sentence reading paradigm, we recruited 36 native Mandarin speakers and examined the role of social status information (...) and literality in their irony interpretation. Our results showed ironic statements were more effortful to process than literal ones, reporting an early and consistent effect on the target regions. The social status effect followed the literality effect, with more difficulty in processing ironic statements that targeted the superordinate than the subordinate; such an effect of social status was missing with literal statements. Besides, an individual’s social skill appeared to affect the perception of status information in ironic statements, as the socially skillful readers needed more time than the socially unskillful to process irony targeting a subordinate in the second half of the experiment in the critical region. Our study suggests that irony processing might be further discussed in terms of the relative predictability of linguistic, social, and individual variabilities. (shrink)

Metamemory processes depend on different factors across the learning and memory time-scale. In the laboratory, subjects are often asked to make prospective feeling-of-knowing (FOK) judgments about target retrievability, or are asked to make retrospective confidence judgments (RCJs) about the retrieved target. We examined distinct and shared contributors to metamemory judgments, and how they were built over time. Eye movements were monitored during a face-scene associative memory task. At test, participants viewed a studied scene, then rated their FOK that they would (...) remember the associated face. This was followed by a forced choice recognition test and RCJs. FOK judgments were less accurate than RCJ judgments, showing that the addition of mnemonic experience can increase metacognitive accuracy over time. However, there was also evidence that the given FOK rating influenced RCJs. Turning to eye movements, initial analyses showed that higher cue fluency was related to both higher FOKs and higher RCJs. However, further analyses revealed that the effects of the scene cue on RCJs were mediated by FOKs. Turning to the target, increased viewing time and faster viewing of the correct associate related to higher FOKs, consistent with the idea that target accessibility is a basis of FOKs. In contrast, the amount of viewing directed to the chosen face, regardless of whether it was correct, predicted higher RCJs, suggesting that choice experience is a significant contributor RCJs. We also examined covariates of the change in RCJ rating from the FOK rating, and showed that increased and faster viewing of the chosen face predicted raising one’s confidence above one’s FOK. Taken together these results suggest that metamemory judgments should not be thought of only as distinct subjective experiences, but complex processes that interact and evolve as new psychological bases for subjective experience become available. (shrink)

Purpose This paper aims to discuss research ethics in mixed-methods research and MMR development with a focus on ethical challenges that stem from working with technical instruments such as mobile eye-trackers. Design/methodology/approach The case of an interdisciplinary mixed-methods development study that aimed at researching the impacts of emerging mobile augmented-reality technologies on the perception of public places serves as an example to discuss research-ethical challenges regarding the practical implementation of the study, data processing and management and societal implications of developing (...) instruments to track and understand human practices. Findings This study reports challenges and experiences in ethical decision-making in the practical implementation of the study regarding the relationship to research subjects, the use of mobile research instruments in public places and the interdisciplinary cooperation among research team members. Further, this paper expounds on ethical challenges and recommendations in data processing and management and with a view to societal implications of method development and the aspirations of transdisciplinarity. This study concludes that institutionalized ethics need to become more flexible, while applied ethics and reflection must make their entry into university curricula across disciplines. Originality/value Complex interdisciplinary mobile and mixed-methods projects that involve sensors and instruments such as mobile eye-trackers are on the rise. However, there is a significant lack of engagement with practical research ethical challenges, practices and requirements in both mixed-methods and method-development literature. By taking a context- and process-oriented perspective focusing on doing ethics, the paper contributes a concrete empirical case to these underdeveloped fields. (shrink)

Referential gaze has been shown to benefit language processing in situated communication in terms of shifting visual attention and leading to shorter reaction times on subsequent tasks. The present study simultaneously assessed both visual attention and, importantly, the immediate cognitive load induced at different stages of sentence processing. We aimed to examine the dynamics of combining visual and linguistic information in creating anticipation for a specific object and the effect this has on language processing. We report evidence from three visual‐world (...) eye‐tracking experiments, showing that referential gaze leads to a shift in visual attention toward the cued object, which consequently lowers the effort required for processing the linguistic reference. Importantly, perceiving and following the gaze cue did not prove costly in terms of cognitive effort, unless the cued object did not fit the verb selectional preferences. (shrink)

This study investigates how speed of motion is processed in language. In three eye-tracking experiments, participants were presented with visual scenes and spoken sentences describing fast or slow events (e.g., The lion ambled/dashed to the balloon). Results showed that looking time to relevant objects in the visual scene was affected by the speed of verb of the sentence, speaking rate, and configuration of a supporting visual scene. The results provide novel evidence for the mental simulation of speed in language (...) and show that internal dynamic simulations can be played out via eye movements toward a static visual scene. (shrink)

Two eye-tracking experiments investigated how and when pointing gestures and location descriptions affect target identification. The experiments investigated the effect of gestures and referring expressions on the time course of fixations to the target, using videos of human gestures and human voice, and animated gestures and synthesized speech. Ambiguous, yet informative pointing gestures elicited attention and facilitated target identification, akin to verbal location descriptions. Moreover, target identification was superior when both pointing gestures and verbal location descriptions were used. These (...) findings suggest that gesture not only operates as a context to verbal descriptions, or that verbal descriptions operate as a context to gesture, but that they complement one another in reference resolution. (shrink)

Subject relative clauses (SRCs) are typically processed more easily than object relative clauses (ORCs), but this difference is diminished by an inanimate head‐noun in semantically non‐reversible ORCs (“The book that the boy is reading”). In two eye‐tracking experiments, we investigated the influence of animacy on online processing of semantically reversible SRCs and ORCs using lexically inanimate items that were perceptually animate due to motion (e.g., “Where is the tractor that the cow is chasing”). In Experiment 1, 48 children (...) (aged 4;5–6;4) and 32 adults listened to sentences that varied in the lexical animacy of the NP1 head‐noun (Animate/Inanimate) and relative clause (RC) type (SRC/ORC) with an animate NP2 while viewing two images depicting opposite actions. As expected, inanimate head‐nouns facilitated the correct interpretation of ORCs in children; however, online data revealed children were more likely to anticipate an SRC as the RC unfolded when an inanimate head‐noun was used, suggesting processing was sensitive to perceptual animacy. In Experiment 2, we repeated our design with inanimate (rather than animate) NP2s (e.g., “where is the tractor that the car is following”) to investigate whether our online findings were due to increased visual surprisal at an inanimate as agent, or to similarity‐based interference. We again found greater anticipation for an SRC in the inanimate condition, supporting our surprisal hypothesis. Across the experiments, offline measures show that lexical animacy influenced children's interpretation of ORCs, whereas online measures reveal that as RCs unfolded, children were sensitive to the perceptual animacy of lexically inanimate NPs, which was not reflected in the offline data. Overall measures of syntactic comprehension, inhibitory control, and verbal short‐term memory and working memory were not predictive of children's accuracy in RC interpretation, with the exception of a positive correlation with a standardized measure of syntactic comprehension in Experiment 1. (shrink)

Visual information conveyed by iconic hand gestures and visible speech can enhance speech comprehension under adverse listening conditions for both native and non‐native listeners. However, how a listener allocates visual attention to these articulators during speech comprehension is unknown. We used eye‐tracking to investigate whether and how native and highly proficient non‐native listeners of Dutch allocated overt eye gaze to visible speech and gestures during clear and degraded speech comprehension. Participants watched video clips of an actress uttering a clear (...) or degraded (6‐band noise‐vocoded) action verb while performing a gesture or not, and were asked to indicate the word they heard in a cued‐recall task. Gestural enhancement was the largest (i.e., a relative reduction in reaction time cost) when speech was degraded for all listeners, but it was stronger for native listeners. Both native and non‐native listeners mostly gazed at the face during comprehension, but non‐native listeners gazed more often at gestures than native listeners. However, only native but not non‐native listeners' gaze allocation to gestures predicted gestural benefit during degraded speech comprehension. We conclude that non‐native listeners might gaze at gesture more as it might be more challenging for non‐native listeners to resolve the degraded auditory cues and couple those cues to phonological information that is conveyed by visible speech. This diminished phonological knowledge might hinder the use of semantic information that is conveyed by gestures for non‐native compared to native listeners. Our results demonstrate that the degree of language experience impacts overt visual attention to visual articulators, resulting in different visual benefits for native versus non‐native listeners. (shrink)

We conducted two eye-tracking experiments investigating the processing of the Mandarin reflexive ziji in order to tease apart structurally constrained accounts from standard cue-based accounts of memory retrieval. In both experiments, we tested whether structurally inaccessible distractors that fulfill the animacy requirement of ziji influence processing times at the reflexive. In Experiment 1, we manipulated animacy of the antecedent and a structurally inaccessible distractor intervening between the antecedent and the reflexive. In conditions where the accessible antecedent mismatched the (...) animacy cue, we found inhibitory interference whereas in antecedent-match conditions, no effect of the distractor was observed. In Experiment 2, we tested only antecedent-match configurations and manipulated locality of the reflexive-antecedent binding (Mandarin allows non-local binding). Participants were asked to hold three distractors (animate vs. inanimate nouns) in memory while reading the target sentence. We found slower reading times when animate distractors were held in memory (inhibitory interference). Moreover, we replicated the locality effect reported in previous studies. These results are incompatible with structure-based accounts. However, the cue-based ACT-R model of Lewis and Vasishth ( 2005 ) cannot explain the observed pattern either. We therefore extend the original ACT-R model and show how this model not only explains the data presented in this article, but is also able to account for previously unexplained patterns in the literature on reflexive processing. (shrink)

Upon hearing a scalar adjective in a definite referring expression such as “the big…,” listeners typically make anticipatory eye movements to an item in a contrast set, such as a big glass in the context of a smaller glass. Recent studies have suggested that this rapid, contrastive interpretation of scalar adjectives is malleable and calibrated to the speaker's pragmatic competence. In a series of eye‐tracking experiments, we explore the nature of the evidence necessary for the modulation of pragmatic inferences (...) in language comprehension, focusing on the complementary roles of top–down information ‐ (knowledge about the particular speaker's pragmatic competence) and bottom‐up cues (distributional information about the use of scalar adjectives in the environment). We find that bottom‐up evidence alone (e.g., the speaker says “the big dog” in a context with one dog), in large quantities, can be sufficient to trigger modulation of the listener's contrastive inferences, with or without top‐down cues to support this adaptation. Further, these findings suggest that listeners track and flexibly combine multiple sources of information in service of efficient pragmatic communication. (shrink)

Based on various stimuli, the findings for the inattentional blindness paradigm suggest that many observers do not perceive an unexpected object in a dynamic setting. In a first experiment, inattentional blindness was combined with eye tracking data from children. Observers who did not notice the unexpected object in the basketball game test by Simons and Chabris spent on average as much time looking at the unexpected object as those subjects who did perceive it. As such, individual differences that (...) are responsible for the recognition of unexpected objects have to be found as further indicators. In a second experiment, the expert-novice paradigm was used to show that the probability of seeing an unexpected object can be increased with specific previous experience. The results in the same task indicate significant differences between basketball experts and basketball novices. The ages of the subjects in both experiments are discussed in connection with the inattentional blindness paradigm. (shrink)

Psycholinguistic methods hold great promise for experimental philosophy. Many philosophical thought experiments and arguments proceed from verbal descriptions of possible cases. Many relevant intuitions and conclusions are driven by spontaneous inferences about what else must also be true in the cases described. Such inferences are continually made in language comprehension and production. This chapter explains how methods from psycholinguistics can be employed to study such routine automatic inferences, with a view to assessing intuitions and reconstructing arguments. We demonstrate how plausibility (...) ratings, pupillometry, and reading time measurements can be used to examine hypotheses about automatic inferences in speech and text comprehension. Two experiments on inferences from polysemous (perception-)verbs provide evidence of a potentially consequential ‘salience bias’. Findings help assess intuitions about unusual cases and analyse a philosophical paradox (‘argument from hallucination’). The paper thus illustrates how we can adapt psycholinguistic methods for philosophical purposes and demonstrates the methods’ philosophical usefulness. (shrink)

Despite various studies examining intertemporal choice with hypothetical rewards due to problematic real reward delivery, there remains no substantial evidence on the effect of the incentives on the decision confidence and cognitive process in intertemporal choice and no comprehensive exploration on the loss domain. Hence, this study conducts an eye-tracking experiment to examine the effect of incentive approach and measure participants' decision confidence using a between-subject design in both gain and loss domains. Results replicated previous findings which show (...) incentives do not affect intertemporal choice in the gain domain. In contrast, in the loss domain, participants in the incentivized group were more likely to choose the larger-later options than those in the non-incentivized group. Furthermore, the decision confidence and the mean fixation duration differed between the incentivized and non-incentivized groups in both gain and loss domains. These findings allow for a better understanding of the effect of incentives on intertemporal choice and provide valuable information for the design of incentives in future intertemporal experiments. (shrink)

Gastronomic experiences offer a set of stimuli that affect the customer’s perception of chef-designed food. This empirical study aims to analyze the influence on the consumer, at a cerebral level, of the stimuli characteristic of a high-level gastronomic experience, in a Michelin starred restaurant. The presentation by the waiter or chef, the plate design, the dish served, the taste of food, interaction or moment in which the food is served are the variables analyzed. Through the use of neuromarketing techniques – (...) galvanic skin response to register emotional arousal, eye tracking to identify where consumers look, and electroencephalography to interpret emotional reactions – combined with qualitative research technique (In-depth interviews with all consumers), in order to know the natural and suggested memories, the objective of this research is to determine the emotional impact of the variables analyzed against the actual taste of food, obtaining conclusions about each variable in overall experience and allowing the authors to propose a model of order design of dishes, designed by the chef, based on emotions and thereby achieving greater efficiency in results of the experience and the memory of it. Results indicate a favorable influence on emotions when the chef presents the food. Likewise, dishes with special presentation have a greater influence at the level of interest than conventional dishes. It is important to highlight that the levels of emotion and attention fall after the midway point of the experience, due to the duration of the experience. Therefore, the dishes do not have the same emotional impact, despite being as special as at the beginning of the experience. (shrink)

With two eye movements tracking experiments, present study sought to establish whether predictability of target words facilitate characters processing in Chinese reading. Target Words embedded in sentences in both experiments were composed by 2 Chinese characters. Predictability of target words were manipulated in both experiments, beyond that, frequency of targets' first characters were manipulated in Experiment 1, frequency of targets' second characters were manipulated in Experiment 2. Pervasive predictability effects were observed on almost all of eye movement (...) measures in both experiments. It also yield reliable characters frequency effect in Experiment 1 & 2, on some measures. None reliable interaction effects of predictability and first character frequency were observed in Experiment 1, but predictability and second character frequency interactively impacted on some measures of eye movement. Theoretical implications of the data are discussed. (shrink)

We report the results of one visual‐world eye‐tracking experiment and two referent selection tasks in which we investigated the effects of information structure in the form of prosody and word order manipulation on the processing of subject pronouns er and der in German. Factors such as subjecthood, focus, and topicality, as well as order of mention have been linked to an increased probability of certain referents being selected as the pronoun's antecedent and described as increasing this referent's prominence, (...) salience, or accessibility. The goal of this study was to find out whether pronoun processing is primarily guided by linguistic factors (e.g., grammatical role) or nonlinguistic factors (e.g., first‐mention), and whether pronoun interpretation can be described in terms of referents' “prominence” / “accessibility” / “salience.” The results showed an overall subject preference for er, whereas der was affected by the object role and focus marking. While focus increases the attentional load and enhances memory representation for the focused referent making the focused referent more available, ultimately it did not affect the final interpretation of er, suggesting that “prominence” or the related concepts do not explain referent selection preferences. Overall, the results suggest a primacy of linguistic factors in determining pronoun resolution. (shrink)

We investigated how two cues to contrast—beat gesture and contrastive pitch accenting—affect comprehenders' cognitive load during processing of spoken referring expressions. In two visual‐world experiments, we orthogonally manipulated the presence of these cues and their felicity, or fit, with the local (sentence‐level) referential context in critical referring expressions while comprehenders' task‐evoked pupillary responses (TEPRs) were examined. In Experiment 1, beat gesture and contrastive accenting always matched the referential context of filler referring expressions and were therefore relatively felicitous on the (...) global (experiment) level, whereas in Experiment 2, beat gesture and contrastive accenting never fit the referential context of filler referring expressions and were therefore infelicitous on the global level. The results revealed that both beat gesture and contrastive accenting increased comprehenders' cognitive load. For beat gesture, this increase in cognitive load was driven by both local and global infelicity. For contrastive accenting, this increase in cognitive load was unaffected when cues were globally felicitous but exacerbated when cues were globally infelicitous. Together, these results suggest that comprehenders' cognitive resources are taxed by processing infelicitous use of beat gesture and contrastive accenting to convey contrast on both the local and global levels. (shrink)

Motivated by controversies surrounding the continued employment of poverty porn in humanitarian business, we initiated two 2 × 2 × 2 experiments to examine the extent to which humanitarian ads that utilize poverty porn images weaponize fundraising. Informed by negative state relief and Affect Intensity Theory, the two investigations explored the effects on study participants of the inclusion within ad appeals of images of starving children, ad spokespeople of disparate gender and ethnicity, and different types of message frame. A 2 (...) (protest emotive vs. informative message) × 2 (male vs. female announcer) × 2 (White British or Black African ethnicity) between-subjects eye track experiment ( n = 236) revealed that an informative message with a White female announcer attracted the most attention. Next, a survey ( n = 667) was completed, which recorded participants’ levels of affect intensity, advertising skepticism, and donation intention. The results suggested that a White British female announcer was more likely to engage potential donors than a male and/or non-White spokesperson. The implications of the findings are discussed in light of how poverty porn might work in practice. (shrink)

The use of visual attention for evaluating consumer behavior has become a relevant field in recent years, allowing researchers to understand the decision-making processes beyond classical self-reports. In our research, we focused on using eye-tracking as a method to understand consumer preferences in children. Twenty-eight subjects with ages between seven and twelve years participated in the experiment. Participants were involved in two consecutive phases. The initial phase consisted of the visualization of a set of stimuli for decision-making in (...) an eight-position layout called Alternative Forced-choice. Then the subjects were asked to freely analyze the set of stimuli, they needed to choose the best in terms of preference. The sample was randomly divided into two groups balanced by gender. One group visualized a set of icons and the other a set of toys. The final phase was an independent assessment of each stimulus viewed in the initial phase in terms of liking/disliking using a 7-point Likert scale. Sixty-four stimuli were designed for each of the groups. The visual attention was measured using a non-obstructive eye-tracking device. The results revealed two novel insights. Firstly, the time of fixation during the last 4 visits to each stimulus before the decision-making instant allows us to recognize the icon or toy chosen from the eight alternatives with a 71.2% and 67.2% of accuracy respectively. The result supports the use of visual attention measurements as an implicit tool to analyze decision-making and preferences in children. Secondly, eye movement and the choice of liking/disliking choice are influenced by stimuli design dimensions. The icon observation results revealed how gender samples have different fixation and different visit times which depend on stimuli design dimension. The toy observations results revealed how the materials determinate the largest amount fixations, also, the visit times were differentiated by gender. This research presents a relevant empirical data to understand the decision-making phenomenon by analyzing eye movement behavior. The presented method can be applied to recognize the choice likelihood between several alternatives. Finally, children's opinions represent an extra difficulty judgment to be determined, and the eye-tracking technique seen as an implicit measure to tackle it. (shrink)

Experiments in animal models have shown that running increases neuronal activity in early visual areas in light as well as in darkness. This suggests that visual processing is influenced by locomotion independent of visual input. Combining mobile electroencephalography, motion- and eye-tracking, we investigated the influence of overground free walking on cortical alpha activity and eye movements in healthy humans. Alpha activity has been considered a valuable marker of inhibition of sensory processing and shown to negatively correlate with neuronal firing (...) rates. We found that walking led to a decrease in alpha activity over occipital cortex compared to standing. This decrease was present during walking in darkness as well as during light. Importantly, eye movements could not explain the change in alpha activity. Nevertheless, we found that walking and eye related movements were linked. While the blink rate increased with increasing walking speed independent of light or darkness, saccade rate was only significantly linked to walking speed in the light. Pupil size, on the other hand, was larger during darkness than during light, but only showed a modulation by walking in darkness. Analyzing the effect of walking with respect to the stride cycle, we further found that blinks and saccades preferentially occurred during the double support phase of walking. Alpha power, as shown previously, was lower during the swing phase than during the double support phase. We however could exclude the possibility that the alpha modulation was introduced by a walking movement induced change in electrode impedance. Overall, our work indicates that the human visual system is influenced by the current locomotion state of the body. This influence affects eye movement pattern as well as neuronal activity in sensory areas and might form part of an implicit strategy to optimally extract sensory information during locomotion. (shrink)

According to the shared signal hypothesis the impact of facial expressions on emotion processing partially depends on whether the gaze is directed toward or away from the observer. In autism spectrum disorder several aspects of face processing have been found to be atypical, including attention to eye gaze and the identification of emotional expressions. However, there is little research on how gaze direction affects emotional expression processing in typically developing individuals and in those with ASD. This question is investigated here (...) in two multimodal experiments. Experiment 1 required processing eye gaze direction while faces differed in emotional expression. Forty-seven children participated. Their Autism Diagnostic Observation Schedule scores ranged from 0 to 6 in the experiment. Event-related potentials were sensitive to gaze direction and emotion, but emotion processing did not depend on gaze direction. However, for angry faces the gaze direction effect on the N170 amplitude, as typically observed in TD individuals, diminished with increasing ADOS score. For neutral expressions this correlation was not significant. Experiment 2 required explicit emotion classifications in a facial emotion composite task while eye gaze was manipulated incidentally. A group of 22 children with ASD was compared to a propensity score-matched group of TD children. The same comparison was carried out for a subgroup of nine children with ASD who were less trained in social cognition, according to clinician’s report. The ASD group performed overall worse in emotion recognition than the TD group, independently of emotion or gaze direction. However, for disgust expressions, eye tracking data revealed that TD children fixated relatively longer on the eyes of the stimulus face with a direct gaze as compared with averted gaze. In children with ASD we observed no such modulation of fixation behavior as a function of gaze direction. Overall, the present findings from ERPs and eye tracking confirm the hypothesis of an impaired sensitivity to gaze direction in children with ASD or elevated autistic traits, at least for specific emotions. Therefore, we conclude that multimodal investigations of the interaction between emotional processing and stimulus gaze direction are promising to understand the characteristics of individuals differing along the autism trait dimension. (shrink)

Public education in museums has the interdisciplinary nature of STEAM education contemporary learning. In the contemporary learning process of the public, digital museums can rely on mobile terminals to provide people with opportunities for mobile learning. Especially since the global outbreak of COVID-19, many offline museums have been forced to close their doors or impose restrictions. How to use digital museums to better carry out the learning experience of visitors is a problem worthy of attention. Effective dissemination of visual information (...) can help digital museums conduct educational activities more efficiently. The purpose of this study was to explore a research method for visual evaluation of visitors’ learning experiences in digital museums. The visual evaluation of visitors in the digital museum learning experience involves the layout of exhibition content, the form of exhibition space, and the way of display of exhibits. Through eye-tracking engineering experiments, we obtain and analyze the visual data of visitors during the educational experience of the digital museum to form an accurate evaluation. This method can be used to assess the visual satisfaction of visitors with the educational experience of a digital museum in mobile learning. This kind of evaluation research can provide strong data support and design optimization direction guidance for the development of digital museum public education. (shrink)