Cognitive pretesting

Cognitive interviewing is a field research method used primarily in pre-testing survey instruments developed in collaboration by psychologists and survey researchers. It allows survey researchers to collect verbal information regarding survey responses and is used in evaluating whether the question is measuring the construct the researcher intends. The data collected is then used to adjust problematic questions in the questionnaire before fielding the survey instrument to the full sample.

Although survey researchers do not totally agree as to what a cognitive interview entails, it in general collects the following information from participants: evaluations on how the subject constructed his or her answers; explanations on what the subject interprets the questions to mean; reporting of any difficulties the subject had in answering the questions; and anything else that reveals the circumstances to the subject’s answers.

In general, there are two methods practiced when conducting a cognitive interview. The first method, called the think-aloud method, encourages participants participating in the interview to verbalize their thoughts while responding to the survey questions. This method is considered purer as it reduces the possibility of the interviewer introducing any bias into the participants’ answers. In contrast, the disadvantage to this method is that it does require training on the part of the participant on the think-aloud process which can be burdensome to the interviewee. The second method has the interviewer ask detailed probes after subjects answer a survey question (called the probing-method). An example of a probe question is: “In your own words, what is this question asking?” or “How did you arrive at your answer?” Advocates for this method suggest that follow-up probes do not interfere with the actual process of responding to survey questions while requiring very little training on the part of the respondent.

In order to conduct a cognitive interview on a survey instrument, the researcher should recruit a minimum of 10 participants to a maximum of 25 participants. The participants recruited should reflect the diversity in the population being studied.

Cognitive interviewing is regularly practiced by U.S. Federal Agencies, including the Census Bureau, National Center for Health Statistics and the Bureau of Labor Statistics.