Evidence from cognitive science supports the claim that humans and other animals see the world as divided into objects. Although this claim is widely accepted, it remains unclear whether the mechanisms of visual reference have representational content or are directly instantiated in the functional architecture. I put forward a version of the former approach that construes object files as icons for objects. This view is consistent with the evidence that motivates the architectural account, can respond to the key (...) arguments against representational accounts, and has explanatory advantages. I draw general lessons for the philosophy of perception and the naturalization of intentionality. (shrink)

Perception is a central means by which we come to represent and be aware of particulars in the world. I argue that an adequate account of perception must distinguish between what one perceives and what one's perceptual experience is of or about. Through capacities for visual completion, one can be visually aware of particular parts of a scene that one nevertheless does not see. Seeing corresponds to a basic, but not exhaustive, way in which one can be visually aware (...) of an item. I discuss how the relation between seeing and visual awareness should be explicated within a representational account of the mind. Visual awareness of an item involves a primitive kind of reference: one is visually aware of an item when one's visual perceptual state succeeds in referring to that particular item and functions to represent it accurately. Seeing, by contrast, requires more than successful visual reference. Seeing depends additionally on meta-semantic facts about how visual reference happens to be fixed. The notions of seeing and of visual reference are both indispensable to an account of perception, but they are to be characterized at different levels of representational explanation. (shrink)

Many have held that when a person visually attends to an object, her visual system deploys a representation that designates the object. Call the referential link between such representations and the objects they designate attentive visual reference. In this article I offer an account of attentive visual reference. I argue that the object representations deployed in visual attention—which I call attentive visual object representations —refer directly, and are akin to indexicals. Then I turn (...) to the issue of how the reference of an AVOR is determined relative to a context. After raising problems for existing accounts, I propose a mechanism of reference determination that is both causal and descriptive: For an AVOR to refer to a particular object, the object must appropriately cause the deployment of the AVOR, and the AVOR must be associated with descriptive information about some of the object's geometrical and mereological properties. (shrink)

In lieu of an abstract, here is a brief excerpt of the content:Kennedy Institute of Ethics Journal 10.2 (2000) 175-188 [Access article in PDF] Scope Note 38 Bioethics Resources on the Web * Once described as an "enormous used book store with volumes stacked on shelves and tables and overflowing onto the floor" (Pool, Robert. 1994. Turning an Info-Glut into a Library. Science 266 (7 October): 20-22, p. 20), Internet resources now receive numerous levels of organization, from basic directory listings (...) to elaborate "links" (cross-references to other Internet documents) that flash and glow in the dark. In many ways, the Internet functions as a direct extension of the traditional library. Following linked references across various sites is an electronic version of browsing library stacks, and the serendipitous discoveries made while "surfing the Net" recall the wonders of physically going "book-to-book." Although many of the Internet's best aspects augment library strengths by eliminating physical distance from sources, Net resources pose unique problems for researchers. It is not always clear who is responsible for the content of Web documents, when the document was produced, or how often (if ever) it is updated. Internet resource tools, then, must enable researchers to identify appropriate resources and to evaluate what they find when they get there. Web Research as Taking "Snapshots" Given the Internet's visual nature and fluid content, the process of conducting research on the Web has been likened to taking an informal picture of information on a specific date at a particular time. This metaphor also conveys the "zoom-in" aspect of searching through Web pages by going from general to specific categories. In the first section of this Scope Note, National Reference Center for Bioethics Literature (NRCBL) staff present a snapshot of bioethics resources taken during the spring months of 2000. Entries are divided into the categories of directories, electronic journals, full-text documents, news/current awareness, and teaching resources. In the two subsequent sections, this series of categories is repeated for entries on a specific topic within bioethics-genetics-and again for general search and evaluation tools. Where a Web site may contain pages applicable [End Page 175] to more than one category, multiple entries are made for that site with appropriate information for each category.Much as telecommunication advances have strengthened the relationship of patients and providers as partners, the dynamic nature of the Internet acknowledges that the researcher functions as a librarian whenever accessing the Web. It is our hope that BIOETHICSLINE and the other NRCBL databases not only will enable researchers to search the Web more efficiently, but also will help them to evaluate what they find when they get there. Bioethicsline on the Web: An International Plan in Progress Once only available through libraries, the U.S. National Library of Medicine (NLM), the funding agency for the ongoing development of the BIOETHICS LINE® database from the Kennedy Institute, has provided free Web access to that database of 63,000 citations since September 1998 through Internet Grateful Med. Plans are now underway to eliminate such specialty databases and incorporate all citations into PubMed (for journal articles) or LOCATORplus (for book-like materials and chapters in books). Both of these databases are likewise available at no cost on the Web. In the transition to PubMed and LOCATORplus some features of BIOETHICSLINE that presently are available may be lost. Researchers are encouraged to contact the National Reference Center for Bioethics Literature with search requests or for search strategy assistance with either the current or the future system (medethx@georgetown.edu).The inclusion of bioethics citations in PubMed and LOCATORplus will have several advantages. First, unique keywords from the Bioethics Thesaurus will be considered for inclusion in Medical Subject Headings (MeSH), the indexing vocabulary for PubMed and LOCATORplus, thereby enhancing retrieval of relevant clinical literature by offering more precise terminology covering ethical issues. Second, it will be easier to identify foreign-language materials, which are not currently included in the scope of BIOETHICSLINE. Third, the updating schedule will become more frequent. On the other hand, there may be some loss of searching precision because certain specialized features... (shrink)

When referring to a target object in a visual scene, speakers are assumed to consider certain distractor objects to be more relevant than others. The current research predicts that the way in which speakers come to a set of relevant distractors depends on how they perceive the distance between the objects in the scene. It reports on the results of two language production experiments, in which participants referred to target objects in photo-realistic visual scenes. Experiment 1 manipulated three (...) factors that were expected to affect perceived distractor distance: two manipulations of perceptual grouping (region of space and type similarity), and one of presentation mode (2D vs. 3D). In line with most previous research on visually-grounded reference production, an offline measure of visual attention was taken here: the occurrence of overspecification with color. The results showed effects of region of space and type similarity on overspecification, suggesting that distractors that are perceived as being in the same group as the target are more often considered relevant distractors than distractors in a different group. Experiment 2 verified this suggestion with a direct measure of visual attention, eye tracking, and added a third manipulation of grouping: color similarity. For region of space in particular, the eye movements data indeed showed patterns in the expected direction: distractors within the same region as the target were fixated more often, and longer, than distractors in a different region. Color similarity was found to affect overspecification with color, but not gaze duration or the number of distractor fixations. Also the expected effects of presentation mode (2D vs. 3D) were not convincingly borne out by the data. Taken together, these results provide direct evidence for the close link between scene perception and language production, and indicate that perceptual grouping principles can guide speakers in determining the distractor set during reference production. (shrink)

Speakers’ perception of a visual scene influences the language they use to describe it—which objects they choose to mention and how they characterize the relationships between them. We show that visual complexity can either delay or facilitate description generation, depending on how much disambiguating information is required and how useful the scene's complexity can be in providing, for example, helpful landmarks. To do so, we measure speech onset times, eye gaze, and utterance content in a reference production (...) experiment in which the target object is either unique or non-unique in a visual scene of varying size and complexity. Speakers delay speech onset if the target object is non-unique and requires disambiguation, and we argue that this reflects the cost of deciding on a high-level strategy for describing it. The eye-tracking data demonstrate that these delays increase when speakers are able to conduct an extensive early visual search, implying that when speakers scan too little of the scene early on, they may decide to begin speaking before becoming aware that their description is underspecified. Speakers’ content choices reflect the visual makeup of the scene—the number of distractors present and the availability of useful landmarks. Our results highlight the complex role of visual perception in reference production, showing that speakers can make good use of complexity in ways that reflect their visual processing of the scene. (shrink)

John Campbell argues that visual attention to objects is the means by which we can refer to objects, and that this is so because conscious visual attention enables us to retrieve information about a location. It is argued here that while Campbell is right to think that we visually attend to objects, he does not give us sufficient ground for thinking that consciousness is involved, and is wrong to assign an intermediary role to location. Campbell’s view on sortals (...) is also queried, as is his espousal of the so-called Referential View of Experience. (shrink)

Speakers’ readiness to describe event scenes using active or passive constructions has previously been attributed—among other factors—to the accessibility of referents. While most research has highlighted the accessibility of agents, the present study examines whether patients’ accessibility can be modulated by means of visual preview of the patient character (derived accessibility), as well as by manipulating the animacy status of patients (inherent accessibility). Crucially, we also examined whether effects of accessibility were amenable to the visuospatial position of the patient (...) by presenting the patient character either to the left or to the right of the agent. German native speakers were asked to describe drawings depicting event scenes while their gaze and speech were recorded. Our results show that making patients more accessible using derived and inherent accessibility factors led to more produced passives, shorter speech onsets, and a reduction of fixations on patients. Complementing previous research on agent accessibility, our findings demonstrate that the accessibility of patients affected both sentence production and looking behavior. While effects were observed for both inherent and derived accessibility, they appeared to be more pronounced for the latter. Regarding character position, we observed a significant effect of position on participants’ gaze patterns and structural choices, suggesting that position itself can be considered an accessibility-related factor. Importantly, the position of a patient also interacted with our manipulation of its accessibility via visual preview. Participants produced more passives after preview than no preview for left-positioned but not for right-positioned patients, demonstrating that effects of patient accessibility (i.e., visual preview) were susceptible to character position. A similar interaction was observed for participants’ viewing patterns. These findings provide the first evidence that the position of a referent is a factor that interacts with other accessibility-related factors (i.e., cueing), emphasizing the need of controlling for position effects when testing referent accessibility. (shrink)

John Campbell investigates how consciousness of the world explains our ability to think about the world; how our ability to think about objects we can see depends on our capacity for conscious visual attention to those things. He illuminates classical problems about thought, reference, and experience by looking at the underlying psychological mechanisms on which conscious attention depends.

Differences in language use and structures between signed and spoken languages have often been attributed to so-called language “modality.” Indeed, this is derived from the conception that spoken languages resort to both the oral-aural channel of speech and the visual-kinesic channel of visible bodily action whereas signed languages only resort to the latter. This paper addresses the use of enactment, a depictive communicative strategy whereby language users imitate referents in signed and spoken languages. Reviewing comparative research on enactment, this (...) paper highlights theoretical and methodological shortcomings in prior works. First, a broader set of causal explanations needs to be taken into account when interpreting differences between signing and speaking communities. A more comprehensive conceptual toolbox ensures that differences are not automatically attributed to modality. In particular, less-studied factors of language diversity, like sociolinguistic and cultural ecologies, and how they interact with other factors should be considered. Second, diversity in enactment across signed and spoken languages is shown to be inadequately and insufficiently documented. It is argued that by comparing enactment across more diverse signing and speaking communities and using large, directly comparable corpora, solid analyses can be carried out, enabling a better understanding of how and why different communities use enactment in similar or different ways. (shrink)

When producing a description of a target referent in a visual context, speakers need to choose a set of properties that distinguish it from its distractors. Computational models of language production/generation usually model this as a search process and predict that the time taken will increase both with the number of distractors in a scene and with the number of properties required to distinguish the target. These predictions are reminiscent of classic findings in visual search; however, unlike models (...) of reference production, visual search models also predict that search can become very efficient under certain conditions, something that reference production models do not consider. This paper investigates the predictions of these models empirically. In two experiments, we show that the time taken to plan a referring expression—as reflected by speech onset latencies—is influenced by distractor set size and by the number of properties required, but this crucially depends on the discriminability of the properties under consideration. We discuss the implications for current models of reference production and recent work on the role of salience in visual search. (shrink)

While speakers have been shown to adapt to the knowledge state of their addressee in choosing referring expressions, they often also show some egocentric tendencies. The current paper aims to provide an explanation for this “mixed” behavior by presenting a model that derives such patterns from the probabilistic combination of both the speaker's and the addressee's perspectives. To test our model, we conducted a language production experiment, in which participants had to refer to objects in a context that also included (...) a visually misleading object whose function was either known to both partners, or known just to the speaker but not the addressee. Modeling results indicate that the experimental findings cannot be explained by assuming that speakers tailor a referring expression solely to their own perspective or to the perspective of their addressee. Instead, accounting for the behavioral pattern requires an approach where both perspectives influence the choice of referring expressions. Nevertheless, in our situation, speakers consider their partner's perspective more than their own. (shrink)

Experiments on visually grounded, definite reference production often manipulate simple visual scenes in the form of grids filled with objects, for example, to test how speakers are affected by the number of objects that are visible. Regarding the latter, it was found that speech onset times increase along with domain size, at least when speakers refer to nonsalient target objects that do not pop out of the visual domain. This finding suggests that even in the case of (...) many distractors, speakers perform object‐by‐object scans of the visual scene. The current study investigates whether this systematic processing strategy can be explained by the simplified nature of the scenes that were used, and if different strategies can be identified for photo‐realistic visual scenes. In doing so, we conducted a preregistered experiment that manipulated domain size and saturation; replicated the measures of speech onset times; and recorded eye movements to measure speakers’ viewing strategies more directly. Using controlled photo‐realistic scenes, we find (1) that speech onset times increase linearly as more distractors are present; (2) that larger domains elicit relatively fewer fixation switches back and forth between the target and its distractors, mainly before speech onset; and (3) that speakers fixate the target relatively less often in larger domains, mainly after speech onset. We conclude that careful object‐by‐object scans remain the dominant strategy in our photo‐realistic scenes, to a limited extent combined with low‐level saliency mechanisms. A relevant direction for future research would be to employ less controlled photo‐realistic stimuli that do allow for interpretation based on context. (shrink)

I examine John Campbell’s claim that the determination of the reference of a perceptual demonstrative requires conscious visual object-based selective attention. I argue that although Campbell’s claim to the effect that, first, a complex binding parameter is needed to establish the referent of a perceptual demonstrative, and, second, that this referent is determined independently of, and before, the application of sortals is correct, this binding parameter does not require object-based attention for its construction. If object-based attention were indeed (...) required then the determination of the referent would necessarily involve the application of sortal concepts, since object-based attention initiates top-down cognitive effects on visual processing. I also examine Mohan Matthen’s claim that reference to objects is established only through the visual processing in the dorsal visual stream and argue that although it is true that processing in the dorsal stream can determine reference, a thesis that goes against Campbell’s view that the determination of the referent requires conscious attention, processing along the ventral visual stream can also establish the reference of perceptual demonstratives. It also claim that Matthen’s account of dorsal processing underestimates the kind of information processed along the dorsal stream and this has some implications regarding perceptual demonstratives reference fixing. (shrink)

The paper argues that the reference of perceptual demonstratives is fixed in a causal nondescriptive way through the nonconceptual content of perception. That content consists first in spatiotemporal information establishing the existence of a separate persistent object retrieved from a visual scene by the perceptual object segmentation processes that open an object-file for that object. Nonconceptual content also consists in other transducable information, that is, information that is retrieved directly in a bottom-up way from the scene (motion, shape, (...) etc). The nonconceptual content of the mental states induced when one uses a perceptual demonstrative constitutes the mode of presentation of the perceptual demonstrative that individuates but does not identify the object of perceptual awareness and allows reference to it. On that account, perceptual demonstratives put us in a de re relationship with objects in the world through the non-conceptual information retrieved directly from the objects in the environment. (shrink)

Previous research has shown that language comprehenders resolve reference quickly and incrementally, but not much is known about the neural processes and representations that are involved. Studies of visual short-term memory suggest that access to the representation of an item from a previously seen display is associated with a negative evoked potential at posterior electrodes contralateral to the spatial location of that item in the display. In this paper we demonstrate that resolving the reference of a noun (...) phrase in a recently seen visual display is associated with an event-related potential that is analogous to this effect. Our design was adapted from the visual world paradigm: in each trial, participants saw a display containing three simple objects, followed by a question about the objects, such as Was the pink fish next to a boat?, presented word by word. Questions differed in whether the color adjective allowed the reader to identify the referent of the noun phrase or not (i.e., whether one or more objects of the named color were present). Consistent with our hypothesis, we observed that reference resolution by the adjective was associated with a negative evoked potential at posterior electrodes contralateral to spatial location of the referent, starting approximately 333 ms after the onset of the adjective. The fact that the laterality of the effect depended upon the location of the referent within the display suggests that reference resolution in visual domains involves, at some level, a modality-specific representation. In addition, the effect gives us an estimate of the time course of processing from perception of the written word to the point at which its meaning is brought into correspondence with the referential domain. (shrink)

Most of the research done with spatial demonstratives have focused on the production, not the interpretation, of these words. In addition, emphasis has been largely on demonstrative pronouns, leaving demonstrative adverbs with relatively little research attention. The present study explores the interpretation of both demonstrative pronouns and demonstrative adverbs in Estonian—a Finno-Ugric language with two dialectal-specific demonstrative pronoun systems. In the South-Estonian dialectal region, two demonstrative pronouns, see—“this” and too—“that”, are used. In the North-Estonian region, only one, see—“this/that”, is used. (...) The aim of this study is twofold. First, we test if the distance and the visual salience of a referent have an effect on the interpretation of demonstratives. Second, we explore if there is a difference in the interpretation of demonstratives between native speakers from SE and NE. We used an interpretation experiment with 30 participants per group and compared the SE and NE group responses. The results clearly show that the distance of the referent has an effect on how demonstratives are interpreted across the two groups, while the effect of visual salience is inconclusive. There is also a difference in the interpretation of demonstratives between the two dialectal groups. When using the Estonian with an influence of the SE dialect, the NE speakers rely on demonstrative adverbs in interpreting the referential utterance that includes demonstrative pronoun and adverb combinations, whereas the SE speakers also take into account the semantics of demonstrative pronouns. We show that, in addition to an already known difference in the production, there is also a difference in the interpretation of demonstratives between the two groups. In addition, our findings support the recognition that languages that have distance neutral demonstrative pronouns enforce the spatial meaning of a referring utterance by adding demonstrative adverbs. Not only is the interpretation of demonstrative pronouns affected, but the interpretation of demonstrative adverbs as well. The latter shows the importance of studying adverbs also, not just pronouns, and contributes to further knowledge of how demonstratives function. (shrink)

In ‘Fregean Reference Defended’ (1995), Sosa presents a sophisticated descriptive theory of reference, which he calls ‘fregean’, and which he argues avoids standard counterexamples to more basic variants of this approach. What is characteristic of a fregean theory, in his sense, is the idea that what makes a person’s thought about some object, a, a thought about that particular thing, is the fact that a uniquely satisfies an appropriate individuator which is suitably operative in her thinking.1 On his (...) version, (FT), any individuating concept, or definite description, is an appropriate individuator, whether it picks out its referent entirely independently of context, and is therefore absolute (e.g. ), or it picks out its referent only with the aid of a context of use, and is therefore perspectival (e.g. ); and such an individuator, α, may be suitably operative in a person’s thinking in one of two ways. First, she may be thinking, de dicto, a proposition predicating some property φ with respect to α. Second, she may be thinking, de dicto, a proposition predicating φ with respect to another individuator, β, which is a member of a referential conception for her at that time, whose epistemic basis contains α as one of its great preponderance of individually co-referential members. The second possibility is designed to avoid standard counterexamples, in which, insofar as a person is thinking, de dicto, a proposition predicating φ with respect to some individuator, β, she is nevertheless intuitively thinking about something other than the unique satisfier of β, either because there is no such thing, just a near miss, or because, although there is, she is really thinking about something else, which may generally be believed to be the β (e.g. Donnellan, 1966; and Kripke, 1980). For Sosa argues that, in such cases, the intuitive object of thought uniquely satisfies an appropriately related α.. (shrink)

Learning a deep structure representation for complex information networks is a vital research area, and assessing the quality of stereoscopic images or videos is challenging due to complex 3D quality factors. In this paper, we explore how to extract effective features to enhance the prediction accuracy of perceptual quality assessment. Inspired by the structure representation of the human visual system and the machine learning technique, we propose a no-reference quality assessment scheme for stereoscopic images. More specifically, the statistical (...) features of the gradient magnitude and Laplacian of Gaussian responses are extracted to form binocular quality-predictive features. After feature extraction, these features of distorted stereoscopic image and its human perceptual score are used to construct a statistical regression model with the machine learning technique. Experimental results on the benchmark databases show that the proposed model generates image quality prediction well correlated with the human visual perception and delivers highly competitive performance with the typical and representative methods. The proposed scheme can be further applied to the real-world applications on video broadcasting and 3D multimedia industry. (shrink)

In lieu of an abstract, here is a brief excerpt of the content:Self-Reference:Theory and Didactics between Language and LiteratureSvend Erik Larsen (bio)Semiotics of Self-ReferenceLiterary metafiction constitutes the extreme case of self-referential texts. Therefore we can either discard it as generally irrelevant for the understanding of the cultural functions of texts, or use it as a point of departure for the formulation of both general and basic aspects of such functions. The position taken in this essay will opt for the (...) last possibility, although I know full well that already the term "metafiction" itself inevitably triggers a variety of skeptical reactions. Does it not just refer to some author's self-centered ruminations in the ivory tower? Or a topic for the nerds of literary studies with a taste for theoretical acrobatics? If so, in both cases following metafictional inclinations results in complete isolation from both the context of literature and its readers.However, one may also adopt the opposite position. The metafictional features of literature constitute one of the means by which literature reaches out to its context and also engages its readers. And critical endeavors are not necessarily devoid of anything but their own concepts. They may instead pave the way for methodological and didactic considerations that ultimately make difficult literature more accessible as a cultural phenomenon. Rather than dwelling upon subtle theoretical differences in the definition of what metafiction is, I will therefore engage in a methodological approach: what kind of questions can we put to the texts, and what kind of answers can we reasonably come up with when using metafiction in a methodological and didactic framework?I shall begin by proposing a set of definitions broader than metafiction that will allow me to zoom in on it as a relevant methodological concept that opens for didactic reflections. The notion of metalevel will constitute my starting point. In its broadest sense, a metalevel indicates a position outside a given field of objects or meanings and in relation to it. But rather than relating [End Page 13] to the content of the field in question, the metalevel is supposed to serve as a platform enabling us to observe and evaluate basic principles and function of the field. This is the role of, for example, the theory of science or of epistemology in relation to the particular knowledge of various sciences. We may call it an epistemological metalevel.In a more restricted sense, a metalevel comprises a level in relation to a given sign system — rather than to a given field of objects — from which we can observe and evaluate the basic principles defining the function, range and validity of the sign system. This is the role of, for example, musicology in relationto the acoustic arts, art criticism in relation to the visual arts, medicine in relation to bodily symptoms. We may call it an inter-semiotic metalevel.Finally, in its most restricted sense, a metalevel points toward a level inside a given sign system from which we can refer to this sign system itself,but not in relation to a given field of objects or in relation to another sign system. We may call this level an intra-semiotic metalevel. In this context, language is a unique system, which in this capacity shapes literature as a particular linguistic phenomenon. Only inside the semiotic system of language can the intra-semiotic metalevel have two different functions:The first function is as a science of language, linguistics, which is itself a verbal sign system. In verbal scientific descriptions and arguments we may scrutinize the natural or everyday language as a medium, that is, its basic linguistic structures and principles, without taking into consideration its content, which it may share with other media. Texts and pictures may refer to the same phenomenon even if such references do not overlap completely. All other sciences differ from linguistics in this respect, because they cannot deal scientifically with their non-verbal objects through the same non-verbal medium as that which makes up the object, but need the support of a linguistic metalevel or at least of signs that are linguistically anchored. Even mathematics or other strictly formal sign systems are based on natural languages. We may refer to... (shrink)

It has often been observed that color is a highly preferred attribute for use in distinguishing descriptions, that is, referring expressions produced with the purpose of identifying an object within a visual scene. However, most of these observations were based on visual displays containing only colors that were maximally different in hue and for which the language of experimentation possessed basic color terms. The experiments described in this paper investigate whether speakers’ preference for color is reduced if the (...) color of the target referent is similar to that of the distractors. Because colors that look similar are often also harder to distinguish linguistically, we also examine the impact of the codability of color values. As a third factor, we investigate the salience of available alternative attributes and its impact on the use of color. The results of our experiments show that, while speakers are indeed less likely to use color when the colors in a display are similar, this effect is mostly due to the difficulty in naming similar colors. Color use for color with a basic color term is affected only when the colors of target and distractors are very similar. The salience of our alternative attribute size, manipulated by varying the difference in size between target and distractors, had no impact on the use of color. (shrink)

Past research has sought to elucidate how speakers and addressees establish common ground in conversation, yet few studies have focused on how visual cues such as co-speech gestures contribute to this process. Likewise, the effect of cognitive constraints on multimodal grounding remains to be established. This study addresses the relationship between the verbal and gestural modalities during grounding in referential communication. We report data from a collaborative task where repeated references were elicited, and a time constraint was imposed to (...) increase cognitive load. Our results reveal no differential effects of repetition or cognitive load on the semantic-based gesture rate, suggesting that representational gestures and speech are closely coordinated during grounding. However, gestures and speech differed in their execution, especially under time pressure. We argue that speech and gesture are two complementary streams that might be planned in conjunction but that unfold independently in later stages of language production, with speakers emphasizing the form of their gestures, but not of their words, to better meet the goals of the collaborative task. (shrink)

Establishing and maintaining reference is a crucial part of discourse. In spoken languages, differential linguistic devices mark referents occurring in different referential contexts, that is, introduction, maintenance, and re-introduction contexts. Speakers using gestures as well as users of sign languages have also been shown to mark referents differentially depending on the referential context. This article investigates the modality-specific contribution of the visual modality in marking referential context by providing a direct comparison between sign language and co-speech gesture with (...) speech in elicited narratives. Across all forms of expression, we find that referents in subject position are referred to with more marking material in re-introduction contexts compared to maintenance contexts. Furthermore, we find that spatial modification is used as a modality-specific strategy in both DGS and German co-speech gesture, and that the configuration of referent locations in sign space and gesture space corresponds in an iconic and consistent way to the locations of referents in the narrated event. However, we find that spatial modification is used in different ways for marking re-introduction and maintenance contexts in DGS and German co-speech gesture. The findings are discussed in relation to the unique contribution of the visual modality to reference tracking in discourse when it is used in a unimodal system with full linguistic structure versus in a bimodal system that is a composite of speech and gesture. (shrink)

The perceptual system estimates distal conditions based upon proximal sensory input. It typically exploits information from multiple cues across and within modalities: it estimates shape based upon visual and haptic cues; it estimates depth based upon convergence, binocular disparity, motion parallax, and other visual cues; and so on. Bayesian models illuminate the computations through which the perceptual system combines sensory cues. I review key aspects of these models. Based on my review, I argue that we should posit co-referring (...) perceptual representations corresponding to distinct sensory cues. For example, the perceptual system represents a distal size using a representation canonically linked with vision and a distinct representation canonically linked with touch. Distinct co-referring perceptual representations represent the same denotation, but they do so under different modes of presentation. Bayesian cue combination models demonstrate that psychological explanation of perception should attend to mode of presentation and not simply to denotation. (shrink)

The thesis that the visual system represents objects has garnered empirical support from a variety of sources in recent decades. But what kinds of things qualify as “objects” in the relevant sense? Are they ordinary three-dimensional bodies? Are they the facing surfaces of three-dimensional bodies? I argue that there is no fact of the matter: what we have are equally acceptable ways of assigning extensions to the relevant visual states. The view I defend bears obvious similarities to Quine’s (...) thesis that linguistic reference is inscrutable. Importantly, though, I argue that even if Quine was wrong about inscrutability as a thesis about language and thought, the case for the inscrutability of visual reference remains strong. (shrink)

It has often been observed that color is a highly preferred attribute for use in distinguishing descriptions, that is, referring expressions produced with the purpose of identifying an object within a visual scene. However, most of these observations were based on visual displays containing only colors that were maximally different in hue and for which the language of experimentation possessed basic color terms. The experiments described in this paper investigate whether speakers’ preference for color is reduced if the (...) color of the target referent is similar to that of the distractors. Because colors that look similar are often also harder to distinguish linguistically, we also examine the impact of the codability of color values. As a third factor, we investigate the salience of available alternative attributes and its impact on the use of color. The results of our experiments show that, while speakers are indeed less likely to use color when the colors in a display are similar, this effect is mostly due to the difficulty in naming similar colors. Color use for color with a basic color term is affected only when the colors of target and distractors are very similar (yet still distinguishable). The salience of our alternative attribute size, manipulated by varying the difference in size between target and distractors, had no impact on the use of color. (shrink)

In referential communication, Grice's Maxim of Quantity is thought to imply that utterances conveying unnecessary information should incur comprehension difficulties. There is, however, considerable evidence that speakers frequently encode redundant information in their referring expressions, raising the question as to whether such overspecifications hinder listeners’ processing. Evidence from previous work is inconclusive, and mostly comes from offline studies. In this article, we present two event‐related potential (ERP) experiments, investigating the real‐time comprehension of referring expressions that contain redundant adjectives in complex (...) visual contexts. Our findings provide support for both Gricean and bounded‐rational accounts. We argue that these seemingly incompatible results can be reconciled if common ground is taken into account. We propose a bounded‐rational account of overspecification, according to which even redundant words can be beneficial to comprehension to the extent that they facilitate the reduction of listeners’ uncertainty regarding the target referent. (shrink)

Human instructors often refer to objects and actions involved in a task description using both linguistic and non-linguistic means of communication. Hence, for robots to engage in natural human-robot interactions, we need to better understand the various relevant aspects of human multi-modal task descriptions. We analyse reference resolution to objects in a data collection comprising two object manipulation tasks and find that 78.76% of all referring expressions to the objects relevant in Task 1 are verbally underspecified and 88.64% of (...) all referring expressions are verbally underspecified in Task 2. The data strongly suggests that a language processing module for robots must be genuinely multi-modal, allowing for seamless integration of information transmitted in the verbal and the visual channel, whereby tracking the speaker’s eye gaze and gestures as well as object recognition are necessary preconditions. (shrink)

This chapter examines the role of the situation model in referential processing and how it can link what appear to be incompatible results from studies of monologue and dialogue as well as studies of reading and visual-world eye tracking. It shows that data from experiments on pronoun resolution in reading indicate a two-step model, in which candidate antecedents for an anaphor are first identified on the basis of gender matching and number matching, then evaluated with respect to the overall (...) situation model. In similar visual-world studies, these stages are seemingly collapsed together, which can be explained in terms of differences in how the situation model is related to the text versus the visual display of potential referents. The chapter suggests that comparing these different types of studies can elucidate the processes of reference resolution. (shrink)