Comp 15 Usability and Human Factors Unit 5

Comp 15 - Usability and Human Factors Unit 5 b - Usability Evaluation Methods This material was developed by Columbia University, funded by the Department of Health and Human Services, Office of the National Coordinator for Health Information Technology under Award Number 1 U 24 OC 000003.

Cognitive Walkthrough (Polson et al, 1992) • Kind of cognitive task analysis • Assesses system usability • Criteria focuses on cognitive processes needed to perform tasks • Identifying sequences of actions and subgoals to successfully complete a task • Assigning causes to usability problems • Are the cues provided by the interface sufficient to perform task? Component 15/Unit 5 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 2

Cognitive Task Analysis (CTA) • Tools and techniques for describing knowledge & strategies required for task performance – Hierarchical decomposition of goals and component tasks • Objective: – Yield information about the knowledge, thought processes, and goal structures that underlie observable task performance Component 15/Unit 5 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 3

Why Do a CTA? • Develop theory of competent performance • Understand invariant features of a task – There are invariant performance characteristics of any class of tasks • Understand process of skill acquisition • Training and instructional resources – e. g. manuals and tutorials • Develop methods for usability testing – Design – Coding scheme for data analysis Component 15/Unit 5 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 4

Cognitive Walkthrough Step 1: Preparations • Choose a set of representative tasks: set of representative –Choose Identifya population of userstasks: – • Describe the contexts of use Identify population of users – • Identify ofofactions Describesequence the contexts use for completing a • task Identify sequence of actions for completing a task • Complex tasks require a task decomposition • • Complex tasks require a task Granularity (e. g. , keystroke to decomposition complete entry) • Granularity (e. g. , keystroke to complete entry) – • Describe user’s initial goal (top-level goal) Component 15/Unit 5 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 5

Step 2: Walkthrough Process of user’s cognitive processes for • Handsimulation of user’s processes for successfully executing an action sequence to complete a asuccessfully task each action and specify: • Stepthrough action and specify: • • – Goal structure for each step – Behavior of interface the interface and its on effect on the user Behavior of the and its effect the user Actions that could be difficult to execute – Actions could be difficult to execute Source of potential problems, – Source of potential problems, • Overall objectives: – Can a user with a certain degree of knowledge perform the tasks • Can a user a certain degree of that the with system is intended to knowledge support? perform the tasks that the system is intended to support? Can a user to perform what is unknown? • – Can a user learn to perform what is unknown? Component 15/Unit 5 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 6

Step 3: Explicate Sources of Potential Problems Goal problems Can user associate specific actions with a goal? • Action problems Cues provided by interface sufficient to perform a task? • Are there incomplete goals that look accomplished? Interpret feedback from system? Monitor his/her own progress ? Transitions between subtasks handled? Component 15/Unit 5 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 7

ATMs: Goal Structure Goals: obtain cash, deposit checks, check balance, pay bills • Goals: obtain cash, deposit checks, check Task: Enter PINpay bills balance, • 1) Retrieve number if necessary • Task: • 2) Enter each digit PIN • 3) Hit enter – 1) Retrieve number if necessary Goal-Obtain-Cash 2) Enter each digit • –Indicate intention to obtain cash • Action: If unclear on step, follow prompt on screen 3) Hit enter • –Action: Indicate Checking Account • Subgoal: Obtain $40 • Action: Enter amount and hit enter • Goal-Obtain-Cash Goal-Terminate – Indicate Transaction intention • Subgoal: Retrieve card to obtain cash – Action: If unclear on step, follow prompt on screen Component 15/Unit 5 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 8

A Partial Walkthrough: ATM Goal: Obtain $80 Cash from Checking Account 1. Action: Enter card (screen 1) • System response: Enter PIN > (screen 2) 2. Subgoal: Interpret prompt & provide input 3 & 4. Actions: Enter “PIN” on numeric keypad and hit enter (press lower white button next to screen) • System response: “Do you want a printed transaction record” • Binary Option: Yes or No (screen 3) 5. Subgoal: Decide whether a printed record is necessary 6. Action: Press button next to no response System response: Select transaction - 8 choices (screen 4) 7. Subgoal: Choose between quick cash & cash withdrawal 8. Action: Press button next to cash withdrawal • System response: Select account (screen 5) 9. Action: Press button next to checking • System response: Enter dollar amounts in multiples of 20 (screen 6) 10 & 11. Enter $80 on numeric key pad and select correct Component 15/Unit 5 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 9

Cognitive Walkthrough: Measure Glucose 1: Begin measurement • Subgoal 1: Begin measurement • Action: Press blue power button – Action: Press blue power button • System response: Meter displays last blood glucose result – System response: Meter displays last blood Subgoal 2: Obtain a blood sample • • glucose result lancet Subgoal 3: Use sterile/sharp Action: Replace lancet (if necessary) Subgoal 4: Draw blood using instrument Action: Pierce finger with lancet – Subgoal 3: Use sterile/sharp • Subgoal 2: Obtain a blood sample lancet Subgoal 5: Apply blood to test strip – Action: Replace lancet (if necessary) • Subgoal 6: Locate Pink Test Area • Action: Dab Finger/Touch Strip • Potential Problem: Missing Test Area or Applying Excessive Blood • Subgoal 4: Draw blood using instrument – Action: Pierce finger with lancet Health IT Workforce Curriculum 2. 0/Spring 2011 Version • Subgoal 5: Apply blood to test strip Component 15/Unit 5 b 10

CW: Blood Glucose 2 • Subgoal 7: Determine if test worked • Subgoal 8: Locate dot – Subgoal 8: confirmation Locate confirmation dot • Action: turn over test strip – Action: turn over test strip • Action: Determine if confirmation dot back of strip is completely blue – Action: Determine if confirmation dot back of • Potential problem: Intermediate shades of blue—has the test strip? is completely blue worked – Potential problem: Intermediate shades of Subgoal 9: Take measurement with device • • • blue—has the test worked ? Subgoal 10: Determine readiness of the device Action: Look for flashing test strip on meter System Response: Code 4 System Response: test stripreadiness – Subgoal 10: Flashing Determine Subgoal 9: Take measurement with device Component 15/Unit 5 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version of the 11

CW: Blood Glucose 3 11: Insert pink strip • Subgoal 11: test. Insert pink test strip • • Action: Push test strip in firmly (pink side up) – Action: Push test strip inwaiting firmly (pink side System response: Flashing clock signals System response: (short lag) new. Flashing glucose valueclock and timesignals – System response: Potential problem: Improper insertion up) waiting – System response: (short lag) new glucose Subgoal 13: Dispose lancet value and time • Action: Point forward on the lancet • Action: Eject and dispose of lancet – Potential problem: Improper insertion Subgoal/Action 12: Read test result 14: Turnoff meter • Subgoal/Action 12: Read test result • Action: Press blue power button response: Meter turns off • • System Subgoal 13: Dispose lancet – Action: Point forward on the lancet Component 15/Unit 5 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 12

CW Glucose Results • 14 14 Subgoal/action pairings • 16 Actions • 5 Device/screen transitions • 5 Device/screen Potentialtransitions problems – Applying blood: missing test area or applying excessive Potential problems blood – Determine if test worked: intermediate shades • Applying blood: missing test area or applying excessive blood of blueif test worked: intermediate shades of blue • Determine • Insert test strip: insert pink test strip – Insert test strip: insert pink test strip Component 15/Unit 5 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 13

Usability Testing • Gold standard for usability evaluation • Set of techniques to collect empirical data – while observing representative end users using the system under study to perform representative tasks • Video-recorded • Provide information that can lead to systems that: – Easy to learn and use – Satisfying to use – Provide utility and functionality that are valued by the target population – Characterize task-specific competencies Component 15/Unit 5 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 14

Think-Aloud Protocol • Method broadly used in cognitive research and usability testing • User verbalizes his/her thoughts while performing a task – Report the contents of working memory – Session is audio and/or video recorded • Transcript of think aloud is coordinated with video analysis Component 15/Unit 5 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 15

Selection of Representative Users • Users may differ: – including age, education, gender, computer experience, etc. • Select subjects based on relevant criteria (e. g. , age, education) • Fully representative not possible • Convenience sample is less desirable Component 15/Unit 5 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 16

Development of Test Plan • Outline task and procedure – Informed by objectives – Prior testing – Constrained by time and setting – Ethical and IRB issues • Exploratory – Characterize potential problems • Controlled Experiment – Comparing 2 Interfaces Component 15/Unit 5 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 17

Role of Researcher • Neutral Observer vs. Active Participant • Researchers may play a more interactive role in field testing – Guide the subject as necessary • Skilled user will need a minimum • Novice may need step-by-step instructions – Autonomy Rule: No more guidance than necessary Component 15/Unit 5 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 18

Field Usability Testing • Hybrid Method: Lab and Ethnography/ Field Study • Naturalistic setting – Numerous constraints • Proscribed set of tasks – Quasi- Experiment • Video analysis is key – Intrusive Component 15/Unit 5 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 19

Video-Analytic Usability Testing on Location: Old School Component 15/Unit 5 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 20

Software-based Video Analysis • Provides a video of all screen activity • Captures user via a webcam • Logs a wide range of events and system interactions including mouse clicks, text entries, web-page changes and windows dialogue events (e. g. , saving a document, selecting among a set of choices). • Morae state of the art usability testing software Component 15/Unit 5 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 21

Morae Video Analytic Usability Software Component 15/Unit 5 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 22

Data Analysis: Transcript • Working document for video analysis • Verbatim and Time Stamped – Every 10 to 30 seconds • Iteratively modified document as coding categories become refined • Add field notes and observations to the transcript Component 15/Unit 5 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 23

Video Analysis: Granularity • Basic (semiformal) video review Basic (semiformal) – Identify and video review categorize observable. Microanalysis problems Macroanalysis – Measure latency of task • Identify and • Segmenting • Fine-grained analysis of short • categorize Macroanalysis session into observable events or stretches of the problems episodes interaction – Segmenting session into events or episodes • Measure latency • Analyzing dialogue and of–task dialogue andobserved behavior observed • Microanalysis behavior – Fine-grained analysis of short stretches of the interaction Component 15/Unit 5 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 24

Macro-Analysis March 18, 2002 I: … You can choose not to answer any questions. May I ask how old are you? P: 74 I: What is or was your professional occupation. P: I have always worked in stores as a salesperson I: How long have you lived in this country? P: 48, 49 years I: If I may ask what level of education do you have? P: I finished high school, but in Columbia. Before I got married, then I came here after I got married, a couple of years afterwards. I: How long have you known that you have diabetes. P: A couple of years. About two years or three. I have never felt that I have diabetes. Component 15/Unit 5 b Profile: 74 years of age Female Native Spanish speaker Originally from Colombia 48 -49 years in US High school education Worked as salesperson in stores Has two sons, one in NJ Never used computer prior to IDEATel Diabetes History: Had it for 2 -3 years Did not have monitoring device prior to IDEATel 1 y. 2 months in program Good health/Has asthma A lot of fluctuation in glucose values Does not look at them over a period of time, watches it one day to the other. Generally good memory remembers values for previous days Health IT Workforce Curriculum 2. 0/Spring 2011 Version 25

Observations: Sending Results • Time: 4: 33: 03 – • Turns on glucose without being told to do so – • Using visualcues realizes it isworking, not working, Using visual realizes it is not sends again sends • Holds her hand on mouse whole time – Holds her hand on mouse whole time • Sits comfortably close to computer – • Sits comfortably Explains what goesclose on as to shecomputer goes Recognizes without difficulty glucose – • Explains what goes on asher she goes and blood pressure values – • Recognizes difficulty her glucose and When modem without sound stops, immediately blood pressure values recognizes the cue, says, “That’s it. ” – When modem sound stops, immediately Component 15/Unit 5 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 26

Micro-Analysis Dialogue and Conversational Code Action Code System Response Comments 7: 28 R: Can you explain what you see now? P: It is calling the Internet. There it turns off, and leaves me with the system. Body Position Change (BPC): Patient moves head, looks from side to side Gesture: Points at screen with index finger Action: Takes mouse and clicks security screen away BPC: keeps hand on mouse for a couple of seconds, then off Action: Takes mouse and clicks identification screen away. Security screen/dialogue box appears Sec. Screen goes away Password screen appears The task is to access the web and patient immediately assumes the control 7: 54 R: So far, so good. Action: Takes hand off mouse, hunches over keyboard, and puts in password. Action: Takes mouse and clicks OK button and hand off mouse. Action: Takes mouse in hand immediately as DM page comes up. Password screen transitions to Diabetes Manager Page Patient performs all actions without any prompting. Time Component 15/Unit 5 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 27

Triangulate Strategy for using more than one data gathering • Strategy for using more than one data technique. For example: gathering technique. For example: • Interviews to target certain stake holders a widercertain populationstake holders – • Questionnaires Interviewstotoreach target • Cognitive walkthrough to evaluate task complexity testing employing to evaluate learnability of – • Usability Questionnaires tonovices reach a wider population system – Cognitive walkthrough to evaluate task Provides different perspectives and corroboration of complexity findings across techniques – Usability testing employing novices to evaluate learnability of system Leading to more rigorous and defensible findings • Provides different perspectives and corroboration of findings across Component 15/Unit 5 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 28