Two experiments are reported that employed think-aloud methods to test predictions concerning relevance effects and rationalisation processes derivable from Evans' (1996) heuristic-analytic theory of the selection task. Evans' account proposes that card selections are triggered by relevance-determining heuristics, with analytic processing serving merely to rationalise heuristically cued decisions. As such, selected cards should be associated with more references to both their facing and their hidden sides than rejected cards, which are not subjected to analytic rationalisation. Experiment 1 used (...) a standard selection-task paradigm, with negative components permuted through abstract conditional rules. Support was found for all heuristic-analytic predictions. This evidence was shown to be robust in Experiment 2, where "select - don't select" decisions were enforced for all cards. Both experiments also clarify the role played by secondary heuristics in cueing the consideration of hidden card values during rationalisation. We suggest that whilst Evans' heuristic-analytic model and Oaksford and Chater's (e.g., 2003) optimal data selection model can provide compelling accounts of our protocol findings, the mental models theory fares less well as an explanation of our full dataset. (shrink)

The standard interpretation of cognitive reflection tests assumes that correct responses are reflective and lured responses are unreflective. However, prior process-tracing of mathematical reflection tests has cast doubt on this interpretation. In two studies (N = 201), we deployed a validated think-aloud protocol in-person and online to test how this assumption is satisfied by the new, validated, less familiar, and less mathematical verbal Cognitive Reflection Test (vCRT). Importantly, thinking aloud did not disrupt test performance compared to (...) a control group. Moreover, verbalized thoughts in both studies revealed that most (but not all) correct responses involved reflection and that most (but not all) lured responses lacked reflection. These data suggest that the vCRT usually satisfies the standard interpretation of the reflection tests (albeit not without exceptions) and that the vCRT can be a good measure of the construct theorized by the two-factor explication of ‘reflection’ (as deliberate and conscious). (shrink)

It has been widely theorized and empirically proven that self-regulated learning (SRL) is related to more desired learning outcomes, e.g., higher performance in transfer tests. Research has shifted to understanding the role of SRL during learning, such as the strategies and learning activities, learners employ and engage in the different SRL phases, which contribute to learning achievement. From a methodological perspective, measuring SRL using think-aloud data has been shown to be more insightful than self-report surveys as it helps (...) better in determining the link between SRL activities and learning achievements. Educational process mining on the basis of think-aloud data enables a deeper understanding and more fine-grained analyses of SRL processes. Although students’ SRL is highly contextualized, there are consistent findings of the link between SRL activities and learning outcomes pointing to some consistency of the processes that support learning. However, past studies have utilized differing approaches which make generalization of findings between studies investigating the unfolding of SRL processes during learning a challenge. In the present study with 29 university students, we measured SRLviaconcurrent think-aloud protocols in a pre-post design using a similar approach from a previous study in an online learning environment during a 45-min learning session, where students learned about three topics and wrote an essay. Results revealed significant learning gain and replication of links between SRL activities and transfer performance, similar to past research. Additionally, temporal structures of successful and less successful students indicated meaningful differences associated with both theoretical assumptions and past research findings. In conclusion, extending prior research by exploring SRL patterns in an online learning setting provides insights to the replicability of previous findings from online learning settings and new findings show that it is important not only to focus on the repertoire of SRL strategies but also on how and when they are used. (shrink)

Two studies examined the use of think aloud (TA) protocol as a means for collecting data of cognitive processes during performance in golf. In study 1, TA was employed to examine if different verbalisation (Level 2 or Level 3 TA) instructions influence performance of high and low skilled golfers. Participants performed 30 putts using TA at either Level 2, Level 3, or no verbalization condition. Although Level 3 verbalization produced a higher volume of verbal data than Level (...) 2, TA at either level 2 or 3 did not impair putting performance compared to no verbalization. Study 2 examined the congruence between data collected via TA at Level 3 and cued retrospective recall of cognitive processes during golf performance. Experienced golfers performed six holes of golf whilst engaging in Level 3 TA. After performance, three semi-structured retrospective interviews were conducted (ten minutes after performance, 24 hours after performance and 48 hours after performance). A comparison of the themes identified large discrepancies between the information reported during TA and at interview, with only 38-41% similarity in variables reported to influence decision making on each hole. Both studies suggest TA is a valuable method for recording cognitive processes of individuals during task performance. TA provides richer verbal data regarding decisions than cued retrospective recall, and TA does not negatively impact performance. (shrink)

The use of remember–know judgments to assess subjective experience associated with memory retrieval, or as measures of recollection and familiarity processes, has been controversial. In the current study we had participants think aloud during study and provide verbal reports at test for remember–know and confidence judgments. Results indicated that the vast majority of remember judgments for studied items were associated with recollection from study , but this correspondence was less likely for high-confidence judgments . Instead, high-confidence judgments were (...) more likely than remember judgments to be associated with incorrect recollection and a lack of recollection. Know judgments were typically associated with a lack of recollection , but still included recollection from the study context . Thus, although remember judgments provided fairly accurate assessments of retrieval including contextual details, know judgments did not provide accurate assessments of retrieval lacking contextual details. (shrink)

The squared paper or graphs are grid-based design representations used in engineering, industrial and craft design practices wherein designs are drawn over symmetrical grids. This paper reports grid-processing strategies undertaken by actors in a native craft practice, viz. Kashmiri carpet-weaving having three task contexts: (1) _design_, wherein designs are drawn on graph sheets and color scheme given by assigning practice-specific symbolic codes to the motifs by designers; (2) _coding_, wherein a cryptic script, called _talim_, is generated from these encoded graphs (...) by talim-writers; and (3) _weaving_, wherein this script is read, decoded and communicated in an equally cryptic trade-language by weavers among their teams. 32 video-recorded think-aloud sessions were conducted with talim-writers to understand their graph-processing and resultant coding strategies. The observations and qualitative analysis of transcripts revealed highly _situated_ strategies undertaken by experts keen on imbibing visuality of design embedded in the grids, in contrast to _embodied_ strategies undertaken by less-experienced coders keen on processing grids’ structural features and getting overwhelmed by grid-clutter in the process. In this landscape, the findings revealed a negotiated nature of errors instead of being deviations from the underlying grid and thereby objectively assessable. The paper makes recommendations for CAD systems to include more of those grid features exploited by experts during their graph processing to facilitate designers. (shrink)

Einstellung effects designate the phenomenon where established routines can prevent people from finding other, possibly more efficient solutions. Here we investigate the mechanism behi...

Thinking aloud is the concurrent verbalization of thoughts while performing a task. The study of thinking-aloud protocols has a long tradition in cognitive psychology, the field of education, and the industrial-organizational context. It has been used rarely in cultural and cross-cultural psychology. This paper will describe thinking aloud as a useful method in cultural and cross-cultural psychology referring to a few studies in general and one study in particular to show the wide applications of this method. Thinking- (...) class='Hi'>aloud protocols can be applied for a) improving the validity of cross-cultural surveys, b) process analysis of thoughts and the analysis of changes over time, c) theory development across cultures, d) the study of cultural meaning systems, and e) individual as well as group level analyses allowing hypothesis testing cross-culturally. Limitations of the thinking-aloud method are also discussed. (shrink)

This article presents an exploratory analysis of the metaphoric structure of five artistic paintings within “Think aloud” protocols, in which a group of 14 English speakers with a low self-rated level of expertise in art and history of art expertise were asked to verbalize all their thoughts, ideas and impressions of the artworks. The main findings of this study can be summarized as follows: (1) multiple interpretations for the same artwork are possible, (2) the interpretations of the metaphorical (...) structures described by the participants often diverge from those advanced by the researchers. These findings challenge the methods by which metaphor identification and analysis in pictorials is currently approached. As a matter of fact, most of the research in pictorial metaphors tends to reduce stimuli such as artistic paintings to unique metaphoric interpretations generally produced by a single researcher by means of introspection. By addressing this methodological problem in metaphor research, this article contributes to the development of a theoretical and operational participant-based framework that takes into account the role of metaphoric conceptualization within the domain of art and art cognition. (shrink)

This article presents a cognitive map proposed to be associated with understanding of the “system concept,” one component of the physics principle of Newton's second low. A definition of the concept is followed by the results of a problem‐solving experiment designed to investigate whether or not good problem solvers possess cognitive structures similar to the one proposed. Think‐aloud protocols were collected as subjects solved a series of physics problems involving Newton's second law. Coding schemes were used to analyze (...) these protocols and to develop a quantitative index intended to reflect the extent to which subjects possessed certain components of the proposed structure. This index proved a highly significant predictor of performance on Newton's second law transfer problems contained in a written exam. In contrast, performance on a set of familiar problems in the same written exam was not a good predictor of performance on transfer problems. These results indicate that cognitive structures that connect the equation ΣF = ma and the concept of choosing appropriate systems are at least part of what constitutes understanding of the principle of Newton's second law. Implications for physics education and problem‐solving research are presented. (shrink)

Research in the fields of complex problem solving (CPS) and dynamic decision making using microworlds has been mainly conducted in Western industrialized countries. This study analyzes the CPS process by investigating thinking‐aloud protocols in five countries. Participants were 511 students from Brazil, Germany, India, the Philippines, and the United States who worked on two microworlds. On the basis of cultural‐psychological theories, specific cross‐national differences in CPS strategies were hypothesized. Following theories of situatedness of cognition, hypotheses about the specific frequency (...) of problem‐solving strategies in the two microworlds were developed. Results of the verbal protocols showed (a) modification of the theoretical CPS model, (b) task dependence of CPS strategies, and (c) cross‐national differences in CPS strategies. Participants’ CPS processes were particularly influenced by country‐specific problem‐solving strategies. (shrink)

In spite of the wide variety of approaches to ethics training it is still debatable which approach has the highest potential to enhance professionals’ integrity. The current effort assesses a novel curriculum that focuses on metacognitive reasoning strategies researchers use when making sense of day-to-day professional practices that have ethical implications. The evaluated trainings effectiveness was assessed by examining five key sensemaking processes, such as framing, emotion regulation, forecasting, self-reflection, and information integration that experts and novices apply in ethical decision-making. (...) Mental models of trained and untrained graduate students, as well as faculty, working in the field of physical sciences were compared using a think-aloud protocol 6 months following the ethics training. Evaluation and comparison of the mental models of participants provided further validation evidence for sensemaking training. Specifically, it was found that trained students applied metacognitive reasoning strategies learned during training in their ethical decision-making that resulted in complex mental models focused on the objective assessment of the situation. Mental models of faculty and untrained students were externally-driven with a heavy focus on autobiographical processes. The study shows that sensemaking training has a potential to induce shifts in researchers’ mental models by making them more cognitively complex via the use of metacognitive reasoning strategies. Furthermore, field experts may benefit from sensemaking training to improve their ethical decision-making framework in highly complex, novel, and ambiguous situations. (shrink)

This exploratory study aims at integrating the psychometric approach to studying creativity with an eye-tracking methodology and thinking-aloud protocols to potentially untangle the nuances of the creative process. Wearing eye-tracking glasses, one hundred adults solved a drawing creativity test – The Test of Creative Thinking-Drawing Production (TCT-DP) – and provided spontaneous comments during this process. Indices of visual activity collected during the eye-tracking phase explained a substantial amount of variance in psychometric scores obtained in the test. More importantly, however, (...) clear signs of methodological synergy were observed when all three sources (psychometrics, eye-tracking, and coded thinking-aloud statements) were integrated. The findings illustrate benefits of using a blended methodology for a more insightful analysis of creative processes, including creative learning and creative problem-solving. (shrink)

The study examined the development of Chinese as a second language learners’ formulaic knowledge through comparing the processing of Chinese idioms versus non-idiomatic formulaic sequences (FSs) by advanced-level learners (ALs), super-advanced learners (SLs), and native speakers (NSs). Using two phrase acceptability judgment tasks with and without think-aloud protocols, we collected data on participants’ processing accuracy, processing speed, and processing strategies of reading the two types of FSs. Four processing patterns emerged from the analyses of the datasets. First, learners’ (...) processing accuracy and speed increased along with their proficiency. Second, learners’ idiom processing ability was generally lower than that of non-idiom processing ability, but they demonstrated an improving trend as their proficiency level increased. Third, learners’ use of processing strategies did not change much as proficiency rose and demonstrated a categorical difference from NSs. Fourth, all three groups exhibited poorer productive idiom knowledge than productive non-idiom knowledge. The overall findings denote that second language learners’ formulaic knowledge can evolve beyond the lexical plateau as learners move from the advanced to a higher proficiency level, but the productive idiom knowledge can be a long-term problem. The findings provide implications for measuring and teaching Chinese formulaic knowledge at the higher-than-advanced stage. (shrink)

This paper presents a cognitive model of the planning process. The model generalizes the theoretical architecture of the Hearsay‐ll system. Thus, it assumes that planning comprises the activities of a variety of cognitive “specialists.” Each specialist can suggest certain kinds of decisions for incorporation into the plan in progress. These include decisions about: (a) how to approach the planning problem; (b) what knowledge bears on the problem; (c) what kinds of actions to try to plan; (d) what specific actions to (...) plan; and (e) how to allocate cognitive resources during planning. Within each of these categories, different specialists suggest decisions at different levels of abstraction. The activities of the various specialists are not coordinated in any systematic way. Instead, the specialists operate opportunistically, suggesting decisions whenever promising opportunities arise. The paper presents a detailed account of the model and illustrates its assumptions with a “thinking aloud” protocol. It also describes the performance of a computer simulation of the model. The paper contrasts the proposed model with successive refinement models and attempts to resolve apparent differences between the two points of view. (shrink)

Interest in thought experiments (TEs) derives from the paradox: “How can findings that carry conviction result from a new experiment conducted entirely within the head?” Historical studies have established the importance of TEs in science but have proposed disparate hypotheses concerning the source of knowledge in TEs, ranging from empiricist to rationalist accounts. This article analyzes TEs in think‐aloud protocols of scientifically trained experts to examine more fine‐grained information about their use. Some TEs appear powerful enough to discredit (...) an existing theory—a disconfirmatory purpose. In addition, confirmatory and generative purposes were identified for other TEs. One can also use details in transcript data, including imagery reports and gestures, to provide evidence for a central role played by imagistic simulations in many TEs, and to suggest that these simulations can generate new knowledge using several sources, including the “extended application” of perceptual motor schemas, implicit prior knowledge, and spatial reasoning operations, in contrast to formal arguments. These sources suggest what it means for TEs to be grounded in embodied processes that can begin to explain the paradox above. This leads to a rationalistic view of TEs as using productive internal reasoning, but the view also acknowledges the historical role that experience with the world can play in forming certain schemas used in TEs. Understanding such processes could help provide a foundation for developing a larger model of scientific investigation processes grounded on imagistic simulation (Clement, 2008). (shrink)

The study examines the relative merits of a noncompatibility and a restructuring explanation of the recurrent empirical finding that a prominent attribute looms larger in choices than in judgments. Pairs of equally attractive options were presented to 72 undergraduates who were assigned to six conditions in which they performed (1) only preference judgments or choices, (2) preference judgments or choices preceded by judgments of attractiveness of attribute levels, or (3) preference judgments or choices accompanied by think-aloud reports. The (...) results replicated the prominence efTect for choices, but a prominence effect was also found for preference judgments. In accordance with the restructuring explanation, the think-aloud protocols indicated that options were more often restructured in choices than in preference judgments. However, restructuring could not explain the prominence effect observed for preference judgments. A modified compatibility hypothesis is offered as an alternative explanation. (shrink)

How do people imagine the possibilities in which an assertion would be true and the possibilities in which it would be false? We argue that the mental representation of the meanings of connectives, such as "and", "or", and "if", specify how to construct the true possibilities for simple assertions containing just a single connective. It follows that the false possibilities are constructed by inference from the true possibilities. We report converging evidence supporting this account from four experiments in which the (...) participants had to list the true and the false possibilities for various sorts of assertion. Their systematic errors, response times, and think-aloud protocols corroborated the theory's predictions. Not even true possibilities are immediately available for complex assertions containing two or more different connectives. The task of listing the true possibilities then becomes difficult, and the task of listing the false possibilities almost impossible. (shrink)

Current theories of reasoning such as mental models or mental logic assume a universal cognitive mechanism that underlies human reasoning performance. However, there is evidence that this is not the case, for example, the work of Ford (1995), who found that some people adopted predominantly spatial and some verbal strategies in a syllogistic reasoning task. Using written and think-aloud protocols, the present study confirmed the existence of these individual differences. However, in sharp contrast to Ford, the present study (...) found few differences in reasoning performance between the two groups, in terms of accuracy or type of conclusion generated. Hence, reasoners present an outward appearance of ubiquity, despite underlying differences in reasoning processes. These findings have implications for theoretical accounts of reasoning, and for attempts to model reasoning data. Any comprehensive account needs to account for strategic differences and how these may develop in logically untrained individuals. (shrink)

Three studies were conducted to examine the relation of spatial visualization to solving kinematics problems that involved either predicting the two‐dimensional motion of an object, translating from one frame of reference to another, or interpreting kinematics graphs. In Study 1, 60 physics‐naíve students were administered kinematics problems and spatial visualization ability tests. In Study 2, 17 (8 high‐ and 9 low‐spatial ability) additional students completed think‐aloud protocols while they solved the kinematics problems. In Study 3, the eye movements (...) of fifteen (9 high‐ and 6 low‐spatial ability) students were recorded while the students solved kinematics problems. In contrast to high‐spatial students, most low‐spatial students did not combine two motion vectors, were unable to switch frames of reference, and tended to interpret graphs literally. The results of the study suggest an important relationship between spatial visualization ability and solving kinematics problems with multiple spatial parameters. (shrink)

Research investigating the intricacies of how self-regulated writing strategies are used in a finely focused area of the second language writing process is still lacking. This study takes a mixed-methods approach to explore Chinese English as a Foreign Language learners’ use of self-regulated writing strategies when revising based on automated, peer, and teacher feedback in an online EFL writing context. Thirty-six Chinese university learners filled in three questionnaires. In addition, four learners followed a think-aloud protocol while revising (...) and responding to a stimulated recall interview to provide further data. The results revealed that learners employed an array of self-regulated writing strategies to attain their feedback revision goals. Learners used more cognitive strategies when revising based on automated feedback compared with peer and teacher feedback and more motivational strategies when revising based on teacher feedback. The think-aloud data and stimulated recall interviews coincided with the quantitative findings. Textual analysis revealed that feedback type and quantity were associated with self-regulated writing strategy use. (shrink)

To add to the currently limited research on the degree of cultural uniqueness of Chinese EFL learners’ anxiety and the multidimensional nature of second language writing anxiety (SLWA), the present qualitative study used think-aloud protocol and interview to examine Chinese EFL learners’ three dimensions of SLWA and the related variables, so as to probe into this problem that could pose an obstacle to sustainable second language writing. Findings showed that Chinese EFL learners experienced much Cognitive Anxiety, but (...) relatively little Avoidance Behavior. Learner-internal, teacher-related, and human-external variables interacted with SLWA in a dynamic way. To a certain degree, Chinese EFL learners showcased uniqueness in some aspects of SLWA, under cultural influences, regarding ethnic culture, local culture, academic culture, and disciplinary culture. (shrink)

The Role of Motivation in Complex Problem SolvingPrevious research on Complex Problem Solving (CPS) has primarily focused on cognitive factors as outlined below. The current paper discusses the role of motivation during CPS and argues that motivation, emotion, and cognition interact and cannot be studied in an isolated manner. Motivation is the process that determines the energization and direction of behavior (Heckhausen, 1991).Three motivation theories and their relation to CPS are examined: McClelland’s achievement motivation, Maslow’s hierarchy of needs, and Dörner’s (...) needs as outlined in PSI-theory. We chose these three theories for several reasons. First, space forces us to be selective. Second, the three theories are among the most prominent motivational theories. Finally, they are need theories postulating several motivations and not just one. A thinking-aloud protocol is provided to illustrate the role of motivational and cognitive dynamics in CPS. Problems are part of all the domains of human life. The field of CPS investigates problems that are complex, dynamic, and non-transparent (Dörner, 1996). Complex problems consist of many interactively interrelated variables. Dynamic ones change and develop further over time, regardless of whether the involved people take action. And non-transparent problems have many aspects of the problem situation that are unclear or unknown to the involved people. CPS researchers focus exactly on such kinds of problems. Under a narro... (shrink)

Taboo terms offer a playground for linguistic creativity in language after language, and sign languages form no exception. The present paper offers the first investigation of taboo terms in sign languages from a cognitive linguistic perspective. We analyze the linguistic mechanisms that introduce offense, focusing on the combined effects of cognitive metonymy and iconicity. Using the Think Aloud Protocol, we elicited offensive or crass signs and dysphemisms from nine signers. We find that German Sign Language uses a (...) variety of linguistic means to introduce and enhance offense, many of which rely on iconic properties of the taboo sign. In conjunction with cross-linguistically common metonymic word-formation strategies, the degree of visual explicitness of a sign increases its potential to offend. Semantically similar taboo signs based on the same metonymic anchor but differing in their degree of iconicity also differ in offensiveness. This allows for creating dysphemisms and euphemisms via phonological changes to a sign. We further show that embodiment creates modality-enhanced ‘vicarious embarrassment’ in the viewer that results in the respective signs being judged obscene or offensive. Further, lexical blending and non-manual enhancement play a role in the creation of dysphemisms in DGS. Lastly, we propose that iconicity as a cognitive structuring principle of linguistic expressions constrains the possible semantic extensions of iconic taboo terms. (shrink)

Power and organizational hierarchies are ubiquitous to social institutions that form the foundation of modern society. Power differentials may act to constrain or enhance people’s ability to make good ethical decisions. However, little scholarly work has examined perceptions of this important topic. The present effort seeks to address this issue by interviewing academics about hypothetical ethical problems that involve power differences among those involved. Academics discussed what they would do in these scenarios, often drawing on their own experiences. Using a (...) think-aloud protocol, participants were prompted to discuss their reasoning and thinking behind their ethical decisions. These interview data were content analyzed using a semantic analysis program that identified a number of distinct ways that academics think about power differences and abuses in ethical situations. Implications of these findings are discussed. (shrink)

Reasoning patterns found in Galileo’s treatise on machines, On Mechanics, are compared with patterns identified in case studies of scientifically trained experts thinking aloud, and many similarities are found. At one level the primary patterns identified are ordered analogy sequences and special diagrammatic techniques to support them. At a deeper level I develop constructs to describe patterns that can support embodied, imagistic, mental simulations as a central underlying process. Additionally, a larger hypothesized pattern of ‘progressive imagistic generalization’—Galileo’s development of (...) a model or mechanism that becomes more and more general with each machine while still being imagistically projectable into many machines—provides a way to think about his progress toward a modern explanatory model of torque. By unpacking his arguments, we gain an appreciation of his skillful ability to foster imagistic processes underlying scientific thinking. (shrink)

Reflective thought, as sustained in writing, is attributed to two‐way communication between a content problem space and a rhetorical problem space. An instructional experiment involving sixth‐graders aimed at helping them sustain such a two‐way process independently, in place of the more typical one‐way process of generating content and writing it out. Instruction included modeling of thinking aloud, both by instructors and students, use of cues to stimulate self‐questioning during planning monologues, and direct strategy instruction emphasizing dialectical synthesis of conflicting (...) ideas. Increased numbers of reflective statements in thinking‐aloud protocols and rated reflectiveness of compositions indicate gains were made at the level of reflection on individual ideas. (shrink)

_To develop reading skills of students, teachers have been advised to provide metacognitive reading strategy instruction by researchers. However, previous research has provided limited understanding of how teachers could foster metacognition of reading strategies in a ‘real’ classroom setting. Moreover, previous research pre-supposed an a priori list of teaching practices in the lessons and did not take into account the needs of the students and the context during metacognitive reading strategy instruction. Studies facilitating metacognitive reading strategy instruction in an ESL (...) setting are also rare. This paper reports on an action research that provides a detailed data-led understanding of how metacognition of reading strategies could be promoted in an ecologically valid university level ESL classroom setting. The study participants of this study were eight undergraduate university students. Data for the study was collected through learner diaries, interviews, think aloud protocols, note-taking, researcher journal and end of class feedback in two action research cycles over an academic year. The findings revealed that providing explicit instruction and opportunities for collaborative discussion about strategy use as well as promoting students’ interest in reading helped raise students’ awareness, use and regulation of reading strategies. Implications of the study are discussed. _. (shrink)

1. EXPLICATING THE CONCEPT OF REFLECTION (under review) -/- To understand how ‘reflection’ is used, I consider ordinary, philosophical, and scientific discourse. I find that ‘reflection’ seems to refer to reasoning that is deliberate and conscious, but not necessarily self-conscious. Then I offer an empirical explication of reflection’s conscious and deliberate features. These explications not only help explain how reflection can be detected; they also distinguish reflection from nearby concepts such as ruminative and reformative reasoning. After this, I find that (...) reflection is not obviously limited to only practical or only theoretical reasoning. The chapter ends with reasons to prefer ‘unreflective’ and ‘reflective’ to dual process theorists’ labels about systems or types. -/- 2. BOUNDED REFLECTIVISM & EPISTEMIC IDENTITY (in Metaphilosophy) -/- Reflection features centrally in philosophy (e.g., Korsgaard, 1996; Rawls, 1971; Sosa, 1991) and psychology (e.g., Pennycook, 2018). Bounded reflectivism is an empirically adequate model of reflection that explains reflection’s capacity to either help or hinder reasoning—e.g., reflective equilibrium vs. reflective polarization (e.g., Kahan et al., 2017). One innovation of the model is epistemic identity: an identity that involves particular beliefs—e.g., religious and political identities. When we feel that our epistemic identity is threatened, we can reflectively defend its beliefs rather than update our beliefs according to the best arguments and evidence. The solution, I argue, is not to suppress epistemic identity but embrace it by appealing to shared, superordinate epistemic identities. -/- 3. ALL MEASURES ARE NOT CREATED EQUAL (under review) -/- A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? “10 cents” comes to most people’s minds immediately. However, upon reflection, we can realize that “5 cents” is the correct answer. There are many reflection tests. Each has its limitations. Some reflection tests are mathematical—like the bat-and-ball problem (e.g., Frederick, 2005) and others are logical (e.g., Janis & Frick, 1943). So there are concerns that reflection tests track logical and mathematical competence rather than reflection per se. Also, some psychologists assume a priori that correct answers are reflective and lured answers are unreflective. New evidence raises additional concerns about the plausibility of these assumptions. I argue that think aloud protocols (Ericsson & Simon, 1998) and process dissociation (Jacoby, 1991) can assuage some of these concerns. -/- 4. GREAT MINDS DO NOT THINK ALIKE (in Review of Philosophy and Psychology) -/- Two large studies (N = 1299), one pre-registered, found that many correlations between reflection and philosophical beliefs among non-philosophers replicated among philosophers. For example, less reflective philosophers preferred theism to atheism and utilitarian rather than deontological responses to the trolley problem (Hannikainen & Cova, in prep.; Pennycook et al., 2016; Reynolds, Byrd, & Conway, forthcoming). However, philosophical judgments were sometimes better predicted by factors like education, gender, and personality than by reflection test performance. So although some relationships between reflection and philosophy were robust, there is more to the link between reflection and philosophy. Normative implications are also discussed—e.g., how we can infer the quality of philosophical views from their correlations with reflective or unreflective reasoning. -/- 5. WHAT WE CAN (AND CAN’T) INFER ABOUT IMPLICIT BIAS (in Synthese) -/- Contrary to some philosophers and psychologists, I argue that implicit bias is probably associative. However, I also argue that debiasing is not thereby entirely unconscious and involuntary. Indeed, strong evidence suggests that reflection can change implicitly biased behavior—e.g., conscious and deliberate counterconditioning. This has implications for the science and morality of implicit bias—e.g., evidence suggests that we can reform biases; so. intuitively, we can be responsible for biases. (shrink)

The use of traditional chicken breeds with a dual purpose has become a relevant topic in Germany mainly due to animal welfare concerns and the importance of conserving genetic variability in poultry farming. However, consumers have little knowledge about the different chicken breeds used in the industry; making it challenging to communicate traditional breeds and their advantages to consumers. Hence, this study takes the approach to look at consumers’ perceptions of different breeds. We analyze consumers’ evaluations of pictures showing four (...) dual-purpose chicken breeds. First, an eye-tracking study and think-aloud protocols were used to obtain open associations consumers make with each breed. Based on the results, an online survey was conducted to quantify consumers associations with different breeds and consumers’ interest in meat products with animal pictures on the packaging. Results show that consumers’ attention to pictures of chickens is mostly focused on their body and head, particularly with the Vorwerkhuhn. Consumers associate white breeds to white egg and meat production, while brown breeds are associated to brown egg production. Only a smaller segment of consumers, who are more engaged to animal welfare, accept pictures of animals on meat packages. We conclude that the marketing of meat products of traditional chicken breeds by using pictures is not a useful approach for the mass market. However, within smaller market concepts, such pictures can be used to communicate an alternative chicken meat production system that may lead to purchases of traditional dual-purpose breeds. (shrink)

Multi-level teaching has been proven to be more effective than a one-size-fits-all learning approach. This study aimed to develop and implement a multi-level remedial teaching scheme in various high school classes containing students of a wide range of learning levels and to determine its effect of their learning. The deterministic inputs noisy and gate model of cognitive diagnosis theory was used to classify students at multiple levels according to their knowledge and desired learning outcomes. A total of 680 senior high (...) school students from central provinces in China participated in the initial cognitive diagnostic test, and 1,615 high school sophomores from seven high schools in China participated in a formal cognitive diagnosis test. Thirty-six high school students from Southwestern China participated in the think-aloud protocols, and 258 seniors from three high schools in southwest China participated in the remedial teaching experiment. Through an analysis of students’ think-aloud protocols, cognitive errors of students at all levels were determined, and multi-level remedial teaching programs were designed to address these common cognitive errors. The remedial teaching programs were then implemented in three schools and compared with a control group. The results indicated that the students in the experimental group showed a more significant improvement. In this study, the steps of designing multi-level remedial teaching include assessment, classification, and preparing a teaching scheme, which are feasible and can have remarkable teaching effects. This process can be used for reference by teachers of various subjects. (shrink)

It is a well-acknowledged fact in legal translation studies that when searching for terminological equivalents, translators should make use of comparative conceptual analysis. Thus, legal translation trainees should be equipped with the necessary tools to carry out such analysis, but the question remains: are they? This paper is a follow-up to a study published in 2017, where modified think aloud protocols were used to explore the following research question: to what degree are university students doing a course in (...) legal and economic translation able to apply the methods of comparative conceptual analysis to translation of terms not accounted for sufficiently in legal dictionaries or terms with no straightforward equivalents. The results showed that major issues involve non-linearity of the analysis carried out and insufficient use of the resources available. The present study involves a different group of 29 BA students of the same course two years later, who were assigned the same task. As the retrospective protocols fail to simulate real-life conditions, this study uses screen recording and keystroke logging to track the processes leading to the identification of the conceptual equivalent in a more detailed and less subject-dependent manner. The results suggest that the steps most challenging for students include identification of relevant features defining the source and target language concepts, comparison of these features and selection, or creation, of an equivalent term reflecting the results of the analysis. Students also frequently show Google-driven searching, which influences the order of the steps performed in their analyses and the sources used. To address these challenges, translation training should include a range of tailor-made exercises focusing on the critical steps of the analysis as well as on improving web searching skills. (shrink)

This study investigates metaphor in its role to mediate concepts in academic textbooks and promote content understanding in the English-medium instruction (EMI) context. Of particular interest is how the language of the discourse affected and possibly hindered metaphor comprehension. Drawing on the theoretical insights found in sociocultural theory and cognitive linguistics, a stance was assumed in which language is treated as embodied and contextual, and verbalizing thoughts (languaging) assists understanding. Three pairs of Japanese students with varying English proficiency levels were (...) invited to participate in online group-based think-aloud protocols where they read and discussed selected paragraphs taken from two social and two natural science textbooks written in English. After accessing the participants’ general knowledge about the main topic of each paragraph, content understanding was accounted for in form of prompts during the think-aloud sessions and a 3-week delayed posttest. In total, over 8 hours of video data were collected, transcribed, and treated with the metaphor identification procedure MIPVU. Qualitative inspections of charged moments in discourse pinpoint metaphor as an important tool for compressing abstract entities or processes into meaningful, dense bundles of information. Participants also created their own analogies and expanded on found metaphoric expressions in textbooks when attempting to make sense of abstract phenomena in science. Further, this study confirms that the lack of English proficiency or schematic knowledge can result in non-understanding, misunderstanding, or partial understanding of metaphoric expressions which has implications for the EMI context. (shrink)

Cognitive diagnostic assessment has been developed rapidly to provide fine-grained diagnostic feedback on students’ subskills and to provide insights on remedial instructions in specific domains. To date, most cognitive diagnostic studies on reading tests have focused on retrofitting a single booklet from a large-scale assessment. Critical issues in CDA involve the scarcity of research to develop diagnostic tests and the lack of reliability and validity evidence. This study explored the development and validation of the Diagnostic Chinese Reading Comprehension Assessment for (...) primary students under the CDA framework. Reading attributes were synthesized based on a literature review, the national curriculum criteria, the results of expert panel judgments, and student think-aloud protocols. Then, the tentative attributes were used to construct three booklets of reading comprehension items for 2–6 graders at three key stages. The assessment was administered to a large population of students in grades 2–6 from 20 schools in a district of Changchun City, China. Q-matrices were compared and refined using the model-data fit and an empirical validation procedure, and five representative cognitive diagnostic models were compared for optimal performance. The fit indices suggested that a six-attribute structure and the G-DINA model were best fitted for the reading comprehension assessment. In addition, diagnostic reliability, construct, internal and external validity results were provided, supporting CDM classifications as reliable, accurate, and useful. Such diagnostic information could be utilized by students, teachers, and administrators of reading programs and instructions. (shrink)

While translation competence assessment has been playing an increasingly facilitating role in translation teaching and learning, it still failed to offer fine-grained diagnostic feedback based on certain reliable translation competence standards. As such, this study attempted to investigate the feasibility of providing diagnostic information about students’ translation competence by integrating China’s Standards of English with cognitive diagnostic assessment approaches. Under the descriptive parameter framework of CSE translation scales, an attribute pool was established, from which seven attributes were identified based on (...) students’ and experts’ think-aloud protocols. A checklist comprising 20 descriptors was developed from CSE translation scales, with which 458 students’ translation responses were rated by five experts. In addition, a Q-matrix was established by seven experts. By comparing the diagnostic performance of four widely-used cognitive diagnostic models, linear logistic model was selected as the optimal model to generate fine-grained information about students’ translation strengths and weaknesses. Furthermore, relationships among translation competence attributes were discovered and diagnostic results were shown to differ across high and low proficiency groups. The findings can provide insights for translation teaching, learning and assessment. (shrink)

This paper considers methodological issues of awareness during adult second language acquisition. Specifically, the paper deals with the issue of instructional orientations, the issue of biases in knowledge measurement, and the issue of reactivity in the online think-aloud protocol. Detailed reviews of prominent SLA research that has investigated the possibility of implicit SLA reveal that the instruction on implicit learning does not guarantee that learners engage in the implicit learning mode, that the majority of SLA research has (...) employed only tests of “explicit learning” such as untimed grammaticality judgment, and that there is some evidence that the online think-aloud protocol causes negative reactivity particularly when it is metalinguistic in nature. (shrink)

Metacognition, the cognition about cognition, is closely linked to intelligence and therefore understanding the metacognitive processes underlying intelligence test performance, specifically on Raven’s Progressive Matrices, could help advance the knowledge about intelligence. The measurement of metacognition, is often done using domain-general offline questionnaires or domain-specific online think-aloud protocols. This study aimed to investigate the relationship between metacognitive awareness and intelligence via the design and use of a novel Meta-Cognitive Awareness Scale – Domain Specific that encourages reflection of task (...) strategy processes. This domain-specific scale was first constructed to measure participants’ awareness of their own metacognition linked to Raven’s Progressive Matrices. Following discriminatory index and Exploratory Factor Analysis, a 15-item scale was derived. Exploratory Factor Analysis showed five factors: Awareness of Engagement in Self-Monitoring, Awareness of Own Ability, Awareness of Responding Speed/Time, Awareness of Alternative Solutions and Awareness of Requisite Problem-Solving Resources. The intelligence level of ninety-eight adults was then estimated using Raven’s Standard Progressive Matrices. Participants also completed the MCAS-DS, and further items that examined their test-taking behavior and Confidence level. Metacognitive awareness was positively correlated to standardized IQ scores derived from the SPM whilst Over-Confidence derived using the Confidence level measure was negatively correlated to SPM. Despite some limitations, this study shows promise for elucidating the relationship between metacognitive awareness and intelligence using the task-specific scale. (shrink)

The ethical decision making of researchers has historically been studied from an individualistic perspective. However, researchers rarely work alone, and they typically experience ethical dilemmas in a team context. In this mixed-methods study, 67 scientists and engineers working at a public R1 (very high research activity) university in the United States responded to a survey that asked whether they had experienced or observed an ethical dilemma while working in a research team. Among these, 30 respondents agreed to be interviewed about (...) their experiences using a think-aloud protocol. A total of 40 unique ethical incidents were collected across these interviews. Qualitative data from interview transcripts were then systematically content-analyzed by multiple independent judges to quantify the overall ethicality of team decisions as well as several team characteristics, decision processes, and situational factors. The results demonstrated that team formalistic orientation, ethical championing, and the use of ethical decision strategies were all positively related to the overall ethicality of team decisions. Additionally, the relationship between ethical championing and overall team decision ethicality was moderated by psychological safety and moral intensity. Implications for future research and practice are discussed. (shrink)

The technological possibilities for new data sources in media psychology, such as online live recordings, called Live Streaming, are growing continuously. These sources do not only offer plentiful quantitative material but also a fairly new access to ecologically valid and unobtrusive observation of problem-solving and decision-making processes. However, to exploit these potentials, epistemological and methodological reflection should guide research. The availability of Big Data and naturally occurring data sets allows to revise the historical controversies on the eligibility of self-description. Drawing (...) on such reflections, media psychology can contribute to renovate well established research methods, such as think aloud protocols, in order to enhance their empirical claims. Apart from other confident attempts to improve these methods, phenomenology and ethnomethodology offer a fruitful account to develop innovative data sources for self-description. Yet, this approach does not support a recurrence of self-description’s previous application but proposes an epistemological shift towards more subtle observations. In order to convey the potentials of media psychology, the risk of repeating classical mistakes, such as introspectionism, have to be regarded. Beyond these fallacies, however, modern digital technology holds encouraging potentials which have already partly been sighted by video gaming research. Due to the similarity of digital environments to laboratory setups, there remains to be a continuity from offline to online research, from traditional data to Big Data. Nevertheless, a true advance into new possibilities requires understanding the qualitative meaning of such data. (shrink)

In this paper we argue that the protocols of subjects recalling the names of their high school classmates, as well as an army of traditional memory phenomena, can be understood from an information processing analysis which interprets retrieval as a problem‐solving process. This characterization of retrieval focuses on the reconstructive and recursive nature of the process of remembering. Retrieval is viewed as a process in which some information about a target item is used to construct a description of the item (...) and this description is used in attempts to recover new fragments of information. A new description i s formed and the process continues recursively. Three subprocesses are identified: FIND A CONTEXT, in which a proper environment for conducting a search i s established: SEARCH, in which bits and pieces of information appropriate to the context are recovered until an adequate description can be formed within the search context; and VERIFY, in which recovery information is checked for consistency.From the perspective this characterization, we examine the protocols of subjects asked to think aloud while recalling the names of their classmates. Subjects' retrievals continued for a surprisingly long period and were characterized by the phenomena of overshoot, systematic hypothesizing, fabrications, establishment of search contexts, self‐corrections, and the use of a number of basic search strategies. The view of memory as a problem‐solving process not only leads us to the identification of these phenomena but also to a framework for a coherent interpretation for them. (shrink)

When, how, and why students use conceptual knowledge during math problem solving is not well understood. We propose that when solving routine problems, students are more likely to recruit conceptual knowledge if their procedural knowledge is weak than if it is strong, and that in this context, metacognitive processes, specifically feelings of doubt, mediate interactions between procedural and conceptual knowledge. To test these hypotheses, in two studies (Ns = 64 and 138), university students solved fraction and decimal arithmetic problems while (...) thinking aloud; verbal protocols and written work were coded for overt uses of conceptual knowledge and displays of doubt. Consistent with the hypotheses, use of conceptual knowledge during calculation was not significantly positively associated with accuracy, but was positively associated with displays of doubt, which were negatively associated with accuracy. In Study 1, participants also explained solutions to rational arithmetic problems; using conceptual knowledge in this context was positively correlated with calculation accuracy, but only among participants who did not use conceptual knowledge during calculation, suggesting that the correlation did not reflect “online” effects of using conceptual knowledge. In Study 2, participants also completed a nonroutine problem‐solving task; displays of doubt on this task were positively associated with accuracy, suggesting that metacognitive processes play different roles when solving routine and nonroutine problems. We discuss implications of the results regarding interactions between procedural knowledge, conceptual knowledge, and metacognitive processes in math problem solving. (shrink)

The goal of this study was to assess the relationships between computational approaches to analyzing constructed responses made during reading and individual differences in the foundational skills of reading in college readers. We also explored if these relationships were consistent across texts and samples collected at different institutions and texts. The study made use of archival data that involved college participants who produced typed constructed responses under thinking aloud instructions reading history and science texts. They also took assessments of (...) vocabulary knowledge and proficiency in comprehension. The protocols were analyzed to assess two different ways to determine their cohesion. One approach involved assessing how readers established connections with themselves. The other approach involved assessing connections between the constructed responses and the texts that were read. Additionally, the comparisons were made by assessing both lexical and semantic comparisons. The result showed that both approaches for analyzing cohesion and making the comparisons were correlated with vocabulary knowledge and comprehension proficiency. The implications of the results for theory and practice are discussed. (shrink)

One of the most influential studies in all expertise research is de Groot’s (1946) study of chess players, which suggested that pattern recognition, rather than search, was the key determinant of expertise. Many changes have occurred in the chess world since de Groot’s study, leading some authors to argue that the cognitive mechanisms underlying expertise have also changed. We decided to replicate de Groot’s study to empirically test these claims and to examine whether the trends in the data have changed (...) over time. Six Grandmasters, five International Masters, six Experts, and five Class A players completed the think-aloud procedure for two chess positions. Findings indicate that Grandmasters and International Masters search more quickly than Experts and Class A players, and that both groups today search substantially faster than players in previous studies. The findings, however, support de Groot’s overall conclusions and are consistent with predictions made by pattern recognition models. (shrink)

Why should musical compositions and artistic images be included among the types of expression covered by free speech principles? One way to answer this question is to show how expression in nonverbal media can be functionally similar to other types of verbal expression. But this leaves us with an intuitively unsatisfying explanation of why free speech principles cover nonverbal creative expression that does not functionally emulate literal speech. In this article, as an alternative justification, we develop and defend the idea (...) that musical and pictorial expression‚ much like literal speech, can be media through which people think aloud, as opposed to mere tools for the transmission of thought. We use this proposal to provide a more robust justification for including nonverbal creative expression in the scope of free speech coverage, and we outline some of the practical policy implications that come with adopting this justificatory strategy. (shrink)

Validated self-report measures of post-stroke fatigue are lacking. The Dutch Multifactor Fatigue Scale (DMFS) was translated into Danish, and response process evidence of validity was evaluated. DMFS consists of 38 Likert-rated items distributed on five subscales: Impact of fatigue (11 items), Signs and direct consequences of fatigue (9), Mental fatigue (7), Physical fatigue (6), and Coping with fatigue (5). Response processes to DMFS were investigated using a Three-Step Test-Interview (TSTI) protocol, and data were analyzed using Framework Analysis. Response processes (...) were indexed on the following categories: (i) “congruent,” response processes were related to the subscale construct; (ii) “incongruent,” response processes were not related to the subscale construct; (iii) “ambiguous,” response processes were both congruent and incongruent or insufficient to evaluate congruency; and (iv) “confused,” participants did not understand the item. Nine adults were recruited consecutively 10–34 months post-stroke (median = 26.5) at an outpatient brain injury rehabilitation center in 2019 [five females, mean age = 55 years (SD= 6.3)]. Problematic items were defined as <50% of response processes being congruent with the intended construct. Of the 38 items, five problematic items were identified, including four items of Physical fatigue and one of Mental fatigue. In addition, seven items posed various response difficulties to some participants due to syntactic complexity, vague terms, a presupposition, and a double-barrelled statement. In conclusion, findings elucidate the interpretative processes involved in responding to DMFS post-stroke, strengthen the evidence base of validity, and guide revisions to mitigate potential problems in item performance. (shrink)