Abstract

We cannot directly uncover the contents of our mind but we can come closer to cognitive processes via overt manifestations. I am interested in how speakers perceive, conceptualise and spontaneously describe complex pictures on higher levels of discourse and I use eye tracking methodology along with verbal protocols and a multimodal scoring method to study these mental processes. In the studies on picture viewing and picture description, the combination of visual and verbal data revealed how picture objects are focused on and conceptualised at different levels of specificity and how objects’ location, activities and attributes are evaluated. We can witness a process of stepwise specification, evaluation, interpretation and re-conceptualisation of picture elements and of the picture as a whole. During their successive picture discovery, viewers describe not only scene-inherent objects with spatial proximity but also clustered elements distributed across the scene and create new mental groupings based on abstract concepts. The process of mental zooming in and out is documented, where concrete objects are re-fixated and viewed with another concept in mind. In sum, the comparison of visual and verbal foci in the process of picture viewing and picture description shows us the ways information is acquired and processed in the human mind. As the studies on mental imagery suggest, inner visualisations and mental images are important for speakers who describe pictures and visual scenes. When analysing attentional shifts in mental image inspection, a significant similarity was found between the eye movement patterns during picture viewing and those produced during picture description. The eye movements closely reflected the content and the spatial relations of the original picture, suggesting that the informants created a sort of mental image as an aid for their descriptions from memory. Eye movements were thus not dependent on a present visual scene but on a mental record of the scene. In addition, even verbal scene descriptions evoked mental images and elicited eye movements that reflect spatiality. In sum, picture viewing, picture description and mental imagery can be used as windows to the mind showing how language and vision, in concert, elucidate covert mental processes.