UserCentered Design and Development Instructor Franz J Kurfess

User-Centered Design and Development Instructor: Franz J. Kurfess Computer Science Dept. Cal Poly San Luis Obispo FJK 2005

Chapter 12 Observing Users FJK 2005

Chapter Overview • • • Goals, Questions, and Paradigms Observation Data Collection Tracking Activities Analysis, Interpretation, and Presentation of Data FJK 2005

Motivation • the actual observation of users as they perform tasks with the system may reveal information that is difficult to obtain otherwise • technologies like video cameras and audio recorders make it fairly easy to capture many aspects of the observation • the activities of users can also be tracked indirectly • the analysis, interpretation and presentation is essential after the actual observation FJK 2005

Objectives • be familiar with the main observation methods • be aware of the different perspectives of an on -looker, a participant, and an ethnographer • appreciate the importance of analyzing, interpreting, and presenting data obtained from observations • gain experience in performing and evaluating observations FJK 2005

What and when to observe • Goals & questions determine the paradigms and techniques used. • Observation is valuable any time during design. • Quick & dirty observations early in design • Observation can be done in the field (i. e. , field studies) and in controlled environments (i. e. , usability studies)

Roles of Observers • outsiders looking on – sometimes through a window or one-way mirror – should the participant be aware of it? • participating observers – the observer works with other participants – emphasis of the observer should be on observation, not on activities to perform the task • ethnographers – intentional immersion into the “world” of the participants FJK 2005

Observation Frameworks • The person. • The place. • The thing. Who? Where? What? Framework Examples – Goetz and Le. Compte (1984) – Robinson (1993)

The Goetz and Le. Compte (1984) Framework • • Who is present? What is their role? What is happening? When does the activity occur? Where is it happening? Why is it happening? How is the activity organized? 15

The Robinson (1993) Framework • Space – • Actors – • What kind of event is it? Goals – • What are individuals doing? Events – • What objects are present? Acts – • What are they doing? Objects – • Who is involved? Activities – • What is the physical space like? What do they to accomplish? Feelings – What is the mood of the group and of individuals? 16

Aspects to Consider • • • goals and questions selection of framework, techniques data collection equipment to use acceptance and trust by participants handling of sensitive issues involvement of informants data analysis need for triangulation – different perspectives

Observing as an Outsider • same technique as in usability testing • more objective than participant observation • often done in usability labs – layout of the facilities – equipment is in place • • recording is continuous analysis & observation almost simultaneous danger of drowning in data analysis – can be coarse or fine grained • presentation – Video clips can be powerful for telling astory

Participant Observation and Ethnography • debate about differences • participant observation is key component of ethnography • co-operation of people observed • informants can be useful – but also sensitive • • data analysis is continuous interpretivist technique questions get refined as understanding grows reports usually contain examples

Activity: Observation in RPGs • is it ethical to join a community for the main purpose of observation – with or without the knowledge of other participants – human observer or computer agent • do you know of any incidents in games you’re participating in FJK 2005

Data Collection Techniques • notes – often selective, and include some interpretation • visual data – still camera – video • audio data – tape recorder – usually included in video – computer • tracking users: – diaries – interaction logging

Observation Activity: Review • data collection techniques used – appropriate – worth the effort • constraints imposed by the selected techniques – – missed data aspects not captured observer bias differences between observers • analysis and interpretation – bias, differences, jumping to conclusions FJK 2005

Data Analysis • qualitative data – interpreted and used to tell the ‘story’ about what was observed – categorized using techniques such as content analysis • quantitative data – collected from interaction and video logs – presented as values, tables, charts, graphs and treated statistically.

Interpretive Data Analysis • key events – drive the group’s activity • patterns of behavior • triangulation – testing of data sources against each other • presentation – report findings in a convincing and honest way • ‘rich’ or ‘thick descriptions’ – include quotes, pictures, and anecdotes • software tools can be useful – NUDIST, n. Vivo (see http: //www. qsr. com. au/) – Ethnograph (see http: //www. qualisresearch. com/) augmented FJK 2005

Looking for Patterns • critical incident analysis – coarse analysis of the overall data set to identify important events – only those events are analyzed in more detail • content analysis – systematic categorization of events – requires orthogonal categories (no overlap) • discourse analysis – interpretation of dialog – also considers meaning, intention, context • quantitative analysis (statistics) – means, standard deviation, etc. augmented FJK 2005

Key Points • observation from outside or as a participant • analyzing video and data logs can be timeconsuming • participant observation includes collections of comments, incidents, and artifacts • ethnography is a philosophy with a set of techniques that include participant observation and interviews • ethnographers immerse themselves in the culture that they study