We addressed the challenge of scoring cognitive interviews in
research involving multiple cultural groups. We interviewed 123
fourth- and fifth-grade students from three cultural groups to
probe how they related a mathematics item to their personal
lives. Item meaningfulness—the tendency of students to relate
the content and/or context of an item to activities in which they
are actors—was scored from interview transcriptions with a
procedure similar to the scoring of constructed-response tasks.
Generalizability theory analyses revealed a small amount of
score variation due to the main and interaction effect of rater but
a sizeable magnitude of measurement error due to the
interaction of person and question (context). Students from
different groups tended to draw on different sets of contexts of
their personal lives to make sense of the item. In spite of
individual and potential cultural communication style
differences, cognitive interviews can be reliably scored by
well-trained raters with the same kind of rigor used in the
scoring of constructed-response tasks. However, to make valid
generalizations of cognitive interview-based measures, a
considerable number of interview questions may be needed.
Information obtained with cognitive interviews for a given
cultural group may not be generalizable to other groups. |