Analysing and interpreting cognitive interview data a qualitative

Analysing and interpreting cognitive interview data: a qualitative approach

Presentation structure • background • aims and objective of QT • design of cognitive interviews • review of methods of analysis • Nat. Cen approach • issues for discussion

Background • aims and objectives of QT • does test question meet measurement objectives • if not, what problems arise • implications of problems for survey • design of cognitive interviews • probe sheet • interviewing techniques • sampling strategy

Review of methods of analysis • little written on analysis of cognitive interview data • two main methods cited: • standardised coding scheme • qualitative analysis • standardised coding scheme approaches are documented (usually) • qualitative analysis descriptions are often scant on detail

Standardised coding schemes • can be used to as stand alone question appraisal tool (e. g. QAS) • can incorporate some elements of behaviour coding • interviewer has a problem reading the question or recording the answer • focus on cognitive Q & A model • • • comprehension/communication recall/computation bias/sensitivity (judgement issues) response category plus ‘logical issues’

Issues with standardised schemes PROS • lend themselves to presenting data quantitatively (x Rs had this problem) • perceived (by some) to be more robust • process is replicable • useful in cross-national/ cross cultural settings, where standardisation important CONS • need a lot of detailed codes under each main heading (particularly comprehension) • time consuming • loose context: why did R interpret Q in that way? • lend themselves to presenting data quantitatively (x Rs had this problem)

Qualitative approach “Just naming and classify what is out there is usually not enough. We need to understand the patterns, the recurrences, the whys. As Kaplan (1964) remarks, the bedrock of inquiry is the researcher’s quest for ‘repeatable regularities’. ” Miles & Huberman (1994)

Approaches to qualitative analysis Ethnographic accounts • detailed ‘thick’ descriptions of cultures or organisations Life histories • analysed as individual cases or mined for common themes Content analysis • identifies content and context of documents, often involves counting (not strictly qual) Grounded theory • generates analytical categories and the links between them through an iterative process of collecting and analysing data

Approaches to qualitative analysis Narrative analysis • examines how a story is told and the intention of the teller Conversation analysis • examines the structure of (usually) naturally occurring conversations Discourse analysis • focuses on how knowledge is produced through the use of language Interpretative analysis • attempts to present and re-present the world of those studied, by identifying and describing substantive themes, and searching for patterns between them

Key stages of the analytical process Data management • identifying themes • sorting and reducing data Generation of findings • describing • classifying • finding linkages and patterns • identifying explanations Characteristics of ‘good’ analysis system • Remains grounded in the data • Transparent data reduction process • Facilitates and displays ordering • Permits within and between case analysis

The analytical hierarchy in qualitative research Seeking wider applications Developing explanations Explanatory analysis Detecting patterns of association Establishing typologies ? Identifying elements & dimensions Descriptive analysis Summarising / synthesising data Sorting data Tagging data Identifying initial concepts / themes Primary data Data management Data collection

Thematic analysis - purposes and principles Structured display of data by theme (Q) across all cases Creating categories and classifying data within them* Demonstrating range and diversity Using examples to illustrate and amplify Must be comprehensive Labelling and categorising must be valid

Carrying out thematic analysis - Nat. Cen approach Familiarisation with data Identification of factors • highlight, summarise, provisionally label Categorisation • is this a different manifestation of that • is this a subset of that • is this of the same order as that Iterative process of refinement Start close to the data - become more abstract and interpretative Must be comprehensive Aim is analytical coherence

Looking for explanations Informed by: • hunches and hypotheses • reflections during fieldwork and analysis • other research or theories Process involves: • detailed within case analysis • comparison between cases • repeated interrogation of the data milking data moving back and forth between cases searching for rival explanations Expect multiplicity Must be comprehensive

Summary of Nat. Cen approach Detailed notes made on interviews Notes reviewed Chart set up Notes charted (chart revised) Charts reviewed Data interpreted Findings emerge Recommendations made Report written

Example chart

Issues to consider • better documentation of qualitative analysis approach • integration of code frames within thematic approach • development of best practice for analysis of cognitive data • cognitive interview data as one component of testing strategy • collaborate findings using other data sources (e. g. split ballot experiments)