Digital Module 12

ITEMS Portal Menu

Digital Modules Home Print Modules

Think-aloud Interviews and Cognitive Labs

Module Overview

In this digital ITEMS module, Dr. Jacqueline Leighton and Dr. Blair Lehman review differences between think-aloud interviews to measure problem-solving processes and cognitive labs to measure comprehension processes. Learners are introduced to historical, theoretical, and procedural differences between these methods and how to use and analyze distinct types of verbal reports in the collection of evidence of test-taker response processes. The module includes details on (a) the different types of cognition that are tapped by different interviewer probes, (b) traditional interviewing methods and new automated tools for collecting verbal reports, and (c) options for analyses of verbal reports. This includes a discussion of reliability and validity issues such as potential bias in the collection of verbal reports, ways to mitigate bias, and inter-rater agreement to enhance credibility of analysis. A novel digital tool for data-collection called the ABC tool is presented via illustrative videos. As always, the module contains audio-narrated slides, quiz questions with feedback, a glossary, and curated resources.

Module

View Module

Meet the Instructors

Jacqueline P. Leighton, University of Alberta

Jackie is a Registered Psychologist and Professor of School and Clinical Child Psychology at the University of Alberta. She completed her graduate degrees in Psychology at the University of Alberta and postdoctoral fellowship studies in Psychology at Yale University. Her research and teaching is driven by the overarching goal to enhance fairness in testing. In pursuit of this goal, she has increasingly focused her research on investigating the interplay between cognitive and emotional processes underlying learning and academic achievement. For example, she investigates variables that can cognitively or emotionally bias participants’ response processes in testing situations, leading to misrepresentations in performance and weaknesses in validity arguments of test inferences. Overall, she is interested in methods designed to enhance diagnosis of achievement, validation of test inferences, and theoretical understanding of human learning.

Blair Lehman, ETS

Blair is a research scientist at Educational Testing Service. She completed her graduate degrees in Cognitive Psychology and certificate in Cognitive Science at the University of Memphis. Her research focuses on understanding students’ emotional and motivational processes during educational activities to design activities that maximize the experience for all students. Her research has focused on the task design of learning and assessment activities as well as the design of adaptive systems that consider both student cognition and motivation. For example, she has explored specific design features of game-based assessments in an effort to understand how to maintain measurement validity while also maximizing student motivation.