HumanComputer Interaction User Study Examples and Qualitative Data

Human-Computer Interaction: User Study Examples and Qualitative Data Gathering CSCE 315 – Programming Studio Spring 2017 Project 3, Lecture 3

Review: The User Study Questions to Address • • • How to get participants? Where to gather data? What data to gather? How to gather that data? How to analyze that data? 2

How to Gather Data? Data Gathering Processes • Interview • Pre-questionnaire • Interactive System Use – Observation! – Recording: photos, audio, video – Logging – Answers found/products produced – Think-aloud • Post-questionnaire Some material from Kerne’s slides 3

User Study Examples • There a wide variety of ways user studies can be conducted • Those listed here a subset, to give a sense of the range of options 4

First Click Testing • Determines what a user would click first on the interface • Use to determine how well an interface lets someone navigate through it • Why important? – Clicking down the right path leads to success 87% of the time (compared to only 46% on a wrong first click) • Determine a task (for study prep) – Know the right order of what menus/buttons/etc. the user needs to click on • Give the user the task (e. g. , set up auto bill pay from chase. com) – Track each click (could be manual or automated) – Record time taken to make click – Can gather subjective information on how easy the user was able to do the task Usability. gov 5

System Usability Scale • 5 -point Likert scale (strongly disagree to strongly agree) • 10 questions – 5 are “positive”, 5 are “negative” • Given to users after they’ve used the system, before any debriefing • From 1986, has become an industry standard • Each question gives score 0 to 4. For the negative questions, reverse order. • Sum them up, multiply by 2. 5 and you have a scale from 0 to 100, with 100 being best • A “quick-and-dirty” approach to usability testing – Not ideal, but widely used/accepted, so easier to compare – Just gives a score on usability – no diagnosis for how to improve • That’s why, we wont use it in out Project 3 to improve UX 6

System Usability Scale Questions 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. I think that I would like to use this system frequently. I found the system unnecessarily complex. I thought the system was easy to use. I think that I would need the support of a technical person to be able to use this system. I found the various functions in this system were well integrated. I thought there was too much inconsistency in this system. I would imagine that most people would learn to use this system very quickly. I found the system very cumbersome to use. I felt very confident using the system. I needed to learn a lot of things before I could get going with this system. 7

Test by Experts • Heuristic Evaluation – Usability Experts review your UI and compare it against accepted usability principles – Nielsen’s Heuristics are most commonly used set of heuristics – This should not replace “Usability testing”, which identifies a different set of issues – Pros: Relative inexpensive, can be applied early in the process – Cons: Requires knowledge and experience that are hard to get • Expert Reviews – A less formal version of Heuristic Evaluation – Requires the same level of expertise, but a different view 8

Interview • Contextual Interview – Observer watches/listens as user works in user’s environment instead of the lab – Collects observations, typically does not include measurements – Observation is followed/interspersed with interview – You find out such things as: Task duration, Internet speed at site, preference between keyboard and mouse usage etc. • User interview – Interview is more in-depth discussion (30 -60 minutes) – Often record interview for later review/coding – Not directly in the context of using the product 9

Interviews • Unstructured - are not directed by a script. Rich but not replicable. • Structured - are tightly scripted, often like a questionnaire. Replicable but may lack richness. • Semi-structured - guided by a script but interesting issues can be explored in more depth. Can provide a good balance between richness and replicability. • Focus groups – a group interview www. id-book. com 10

Interview questions • Two types: − ‘closed questions’ have a predetermined answer format, e. g. . ‘yes’ or ‘no’ − ‘open questions’ do not have a predetermined format • Closed questions are easier to analyze • Avoid: − Long questions − Compound sentences - split them into two − Jargon and language that the interviewee may not understand − Leading questions that make assumptions e. g. . why do you like …? − Unconscious biases e. g. . gender stereotypes www. id-book. com 11

Focus Group • More efficient than one-on-one interview in terms of getting qualitative data from many people • Less efficient, but allows deeper exploration than a survey • Capture information (recording or notes) • Moderator has to be able to progress through questions and handle group dynamics – Poor moderation can lead to people not being forthright, getting into group-think, getting drowned out 12

Online surveys • Low cost • Broad audience • Identify early on: – – The purpose of the survey Where you will get respondents Software to use Who will analyze data, and now • Questions should be created to measure or get feedback on what you intend • Can mix open-ended and closed questions – Allows mix of quantitative and qualitative evaluation 13

Task Analysis • Early stage user study – Observe users in action to determine goals/scope of a project • Find how users go about some task – What their goals are, what they try to achieve – What they actually do to achieve the goals – What experiences (personal, social, cultural) they bring to the tasks – How users are influenced by physical environment – How users’ previous knowledge and experience influence: • Their thought process • Their workflow 14

Eye Tracking • Use an eye tracker to determine where people are looking on a screen during a task • Can measure several items: – Where they are looking – How long they are looking – Focus/order of examination – Where they don’t look • Can infer where people are reading/studying vs. glancing – Focus does not mean comprehension 15

Eye Tracking (2) Heat Maps Saccade Pathways 16

Qualitative Data Gathering and Analysis • Most people are less familiar than with quantitative methods, where statistics and traditional mathematics can be applied • Formalizations of methods for gathering and analyzing qualitative data have been developed – Most of these are derived from social sciences 17

Qualitative Data Gathering • Understand human activity/experience situated in context • Seeks the why and how – Makes sense of human issues – Non-statistical • Inductive approach to analysis – Not hypothesis driven (though hypotheses are created), not testing-based – Seek to pull out understanding from patterns of detail Some material from Kerne’s slides 18

Ethnography • Role of the ethnographer: takes the view of the subject of the study – Immerses into the culture – Often, the ethnographer enters or becomes part of the culture being studied, but can remain outside • Way of representing the culture of a group – In HCI terms, of how people will interact in the system – Holistic view: understanding the user from the user’s point of view, including all a user brings in 19

Ethnography Methods • • Writing culture down Thick description of all observed behavior Layers of signification and meaning Observation – In all forms, with recording in some way • • Stories Context Interpretation Interviews Some material from Kerne’s slides 20

Gathering Qualitative Data • Guiding questions: – What’s happening here? – What are basic social processes? – What are basic social psychological processes? • Close observation – Record what’s seen and heard • Intensive interviewing – Transcribe them to have good representation • Identify and name phenomena Some material from Kerne’s slides 21

Simple Qualitative Analysis • Recurring patterns or themes – Emergent from data, dependent on observation framework if used • Categorizing data – Categorization scheme may be emergent or pre-specified • Looking for critical incidents – Helps to focus in on key events www. id-book. com 22

Analyzing Qualitative Data Grounded Theory • Weave together threads drawn from data – Generalizable theoretic statements / hypotheses • Contextual Analysis – How theory should be and is observed as being represented in a particular context Some material from Kerne’s slides 23

Analyzing Qualitative Data Grounded Theory • Development of codes and categories – Derive theory from systematic analysis of data • Drawn from data collected – Back-and-forth from data collection to coding – Gradually refine codes/categories to form understanding – Apply codes to new data to see if they fit • Coding: – – Categorize statements and data points Summarize what has been observed Account for each datum Select, Separate, Sort Some material from Kerne’s slides 24

Coding Details • Beginning: – Name each segment: – Identify most significant codes – Determine granularity: word-by-word, line-by-line, incident-to-incident, etc. • Determining codes: – Naming/labeling suggested by context – Identify common characteristics through comparative analysis – Seek novel explanations in phenomena – Discern range of potential meanings • Refinement – Adjust codes over time: split, merge, drop, create Some material from Kerne’s slides 25

Coding Self-Evaluation • Do your codes help you understand what the data indicate? How? • What do your concepts add to data interpretation? • How does our coding reflect incident or described experience? • Are there clear, evident connections between data and codes? • Have you collected enough background data about persons, processes and settings to understand portray full range of contexts? • Have you gained descriptions of a range of participants’ views and actions • Are your data sufficient to reveal changes over time? • Have you gained multiple views of participants’ range of actions? • Have you gathered data that enable you to develop analytic categories? • What kind of comparisons can you make and how do they generate and inform your ideas? Some material from Kerne’s slides 26

Code book used in grounded theory analysis www. id-book. com 27

Grounded Theory: Categorization • Provide concepts, derived from data – Drawn from the coded data – Means for “chunking” concepts – Reduce number of units – Stand for phenomena • Analytic power: explain and predict • Define properties, and range of values for some properties Some material from Kerne’s slides 28

Analyzing Qualitative Data Distributed Cognition • The people, environment, and artifacts are regarded as one cognitive system • Used for analyzing collaborative work • Focuses on information propagation and transformation 29

Analyzing Qualitative Data Activity Theory • Explains human behavior in terms of our practical activity in the world • Provides a framework that focuses analysis around the concept of an “activity” and helps to identify tensions between the different elements of the system • Two key models: – One outlines what constitutes an activity – One models the mediating role of artifacts 30

References • Some slide material taken from information at: – https: //www. usability. gov/how-to-andtools/methods/index. html • Some qualitative slides taken from Andruid Kerne’s slides: – Noted in footers – Material is copyrighted – Can be used and cited freely for non-commercial education and scholarship without profit provided that credit is given to the author and this notice is included • Some topics taken from: – Interaction Design, 4 th edition, by Rogers, Sharp, and Preece, Wiley, 2014. – www. id-book. com – See chapters 7 -8 in particular www. id-book. com 31