Experimental Design Evaluation 4 Contextual Inquiry Sunyoung Kim

Experimental Design & Evaluation 4. Contextual Inquiry Sunyoung Kim, Ph

Contextual Inquiry

Contextual Inquiry Go where the customer works, immerse yourself in the context, observe the customer (user) as he or she works, and talk to the customer about the work. Think yourself as an apprentice and the user as a master. • You want to learn about the users’ work from them • A master teaches by doing the work and talking about it while working

Conducting Contextual Inquiry Use Work-based interview when: • Product or process already exists • User is able to complete a task while you observe • Work can be interrupted

Contextual Inquiry An approach to answering the question, “What should we build to help people do their work better? Key elements • Contextual Inquiry – Gather data from users while they do their work to help decide what system should do. • Work modeling – Use data to build models of work that are explicit and sharable. • Work redesign – Use data and models to design work model for how users will work in the future. The core design problem is work design, rather than technology design. • Use work model and iterative prototyping to drive detailed user interface design.

Why Contextual Inquiry? Need to understand what will help people do their work better while fitting into their lives and matching their culture.

Summary vs. Ongoing Experience Retrospective accounts are often summaries “I got to work, checked my email and had a cup of coffee” By being present in the time and place of activity, we can access much richer data from ongoing activities “I got to work, looked over my email, answered messages from my boss, decided to have some coffee, walked to the coffee machine, found there was no coffee, so I made coffee…” Abstract vs. Concrete data

Purposes of Contextual Inquiry To obtain data from users in their context • insights about the users’ environment • insights about their many tasks • insights about the people they work with • insights about cultural influences on work (expectations, desires, policies, values, etc. ) • understanding of breakdowns in current processes To help define requirements, plans and designs and to prioritize

Attitudes during Contextual Inquiry Master/Apprentice model • You are the Apprentice; the user is the Master - keeps investigator humble, and customer in charge • Investigator is not afraid to ask questions • Teaching while doing means user doesn’t have to think in advance what to convey • Allows discovery of subtle details, which may be overlooked in formal discussion • Reduces need for a formal set of interview questions

Attitudes during Contextual Inquiry • Be open to possibilities • Be committed to expand ground the focus (more about focus soon) • Learn rather than teach • Be willing to modify your thinking • Pay attention to the participant’s needs • Pay attention to your needs

Principles of Contextual Inquiry 1. 2. 3. 4. Context: Understand users' needs in their work environment Partnership: Work with users as co-investigators Interpretation: Assign meaning to the observations Focus: Listen & probe from a clearly defined set of concerns

1. Context The interrelated conditions within which something occurs or exists Understand work in its natural environment • Go to the stakeholders (users, agents, etc. ) • Observe real work • Interview while she/he is working Interview, Survey Contextual Inquiry • Remembered experience (or summary data & abstractions • Subjective • Limited by reliability of human memory • What customers think & say they do • Ongoing experience & concrete data where work is happening • Objective • Limited by ability to observe directly • What customers do

Context: Do’s & Dont’s • Go to the workplace & see the work as it unfolds • Seek concrete data o Direct conversation to ongoing work User: "I usually record appointments in my calendar” Interviewer: "Could you go ahead & walk me through the process” o If direct observation is not possible in area of focus, elicit retrospective accounts” Think about the last time you used System-X. Take me through the steps you went through with the system to get the job done. ”

Context: Do’s & Dont’s • Vet your design ideas o CI is also a fine time to get initial feedback on design ideas “If you had a technology that did X, would that solve this problem? ” o Users will quickly understand the intent of your suggestion, and will be able to provide direct feedback o This will also demonstrate your understanding of the problem, providing opportunity for brainstorming and/or clarification

Context: Do’s & Dont’s • Avoid summary information • Avoid abstract data o Watch out for words like "usually", "generally" etc. : Abstractions are open to interpretation and can compromise the concrete data needed to guide design o Instead, do direct conversation towards ongoing work or recent past events You want responses that include words like: - “Let me show you…” - “Here’s how I do it. . . ” - “Last time, I …”

Pay attention to • • • User's work space User's work intentions User's words Tools used How people work together Business goals Organizational and cultural structure Be guided by focus (and open to changing focus if the changed focus falls within another area of concern)

2. Partnership A relationship characterized by close cooperation. Build an equitable relationship with the user • Not traditional interviewer-interviewee • Not master-apprentice model • An approximation for partnership, but has limitations • Not expert-novice (unless they are experts) • Not host-guest (be nosy) AIM FOR EQUALITY • CI Interviewer must built a partnership

Establishing Partnership 1. Share control 2. Use open-ended questions that invite users to talk: • "What are you doing? ” • "Is that what you expect? ” • "Why are you doing X. . . ? ” 3. Let the user lead the conversation (provided on focus) 4. Listen and listen! 5. Pay attention to communication that is non-verbal

Partnership: Do’s & Dont’s • Engage user as a partner in design o Be nosy -- clarify, probe, and watch o Test design ideas in work context in which they might be relevant-“would it be useful if …” o Use caution on “would it be” questions • Do not assume role of interviewer o Questions should be in context of ongoing work

Partnership: Do’s & Dont’s • Do not assume role of expert o Answering user questions about system alters their behavior o Instead: "Why don’t you go ahead & do what you'd do if I weren't here, I'll answer any questions you may have at the end. ” • Do not assume role of guest o Being overly polite may inhibit the kind of interaction that may lead to a better understanding of work. Instead, be nosy!

3. Interpretation • Check your interpretation to make sure they are accurate “I saw you just do X. Is that because of Y? ” “I believe X. Is that correct? ” • As long as you check your interpretations in-context, participants will respond accurately • Outside of context, they may be more inclined to agree or answer in generalities rather than specifics

4. Focus • Focus is a perspective • Better to make it explicit • Characteristics of focus: - Helps keep conversation on useful topics - Set of pre-conceived assumptions & beliefs - Reveals & conceals Role of focus • Directs the selection of participants • Directs questioning • Creates understanding • Provides rich data

Focus: Do’s & Dont’s • • Directs the selection of participants Directs questioning Creates understanding Provides rich data

Focus: Do’s & Dont’s • DO allow focus to steer conversation • DO allow focus to expand o Probe surprises & contradictions o Probe what you don’t know or understand o Be wary of nodding in agreement (Do you REALLY understand? ) • DO challenge focus assumptions o DON’T try to validate them • DO avoid “expert blind spot”

Focus differs in different stages Broad focus Usefulness Artifact Narrow focus Usability Early Time in system development Late

Information you want to know • Understanding people’s underlying goals - What are people trying to accomplish? • Understanding their current behavior - How do they spend their time? - What are their priorities? - What problems do they currently have? What inefficiencies or costs are they putting up with? - How do they feel about different experiences they have?

Pay attention to: • • • Flow of work: Direction of communication and coordination Sequence of work: Detailed sequence of work steps Culture of work: External influences Artifacts used in the work: Physical objects that support the work Physical environment of the work: Layout of the work environment

7 ways to screw up CI • • Not being inquisitive/nosy enough Overly disrupting the task Turning it into a regular interview Failing to respect your participants Failing to observe and take good notes Focusing on the wrong details Slipping into abstraction