Comp 15 Usability and Human Factors Unit 5
Comp 15 - Usability and Human Factors Unit 5 a - Usability Evaluation Methods This material was developed by Columbia University, funded by the Department of Health and Human Services, Office of the National Coordinator for Health Information Technology under Award Number 1 U 24 OC 000003.
Overview • • • Usability Evaluation methods Interviews Surveys / questionnaires Usability inspection methods – Heuristic evaluation – Cognitive walkthrough • Usability testing Component 15/Unit 5 a Health IT Workforce Curriculum 2. 0/Spring 2011 Version 2
Usability Quality of a user's experience when interacting with a product or system Factors affect the user's experience: • • Ease of learning / learnability Efficiency of use Supports cognitive task Memorability Error frequency and severity Esthetics Subjective satisfaction Component 15/Unit 4 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 3
Why Usability Matters: The Clinical Story At least 40% of systems are either abandoned or fail to meet minimum requirements Healthcare information and management systems society consider poor usability of clinical information systems as possibly the most important factor hindering adoption Problems with adoption, productivity and patient safety Component 15/Unit 4 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 4
Why Usability Matters: The Story from Patients and Consumers e. Health interventions offer significant promise to bridge the digital divide • Problems with usability and poor design disproportionately affect lower computer literacy users • Exacerbate the digital divide and possibly increase health disparities Chronic illness and Aging populations have special needs • More susceptible to usability problems Consequences of poor design: • Lack of empowerment, reduced enthusiasm, limited adoption and nonproductive use of patient-centered systems Component 15/Unit 4 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 5
Usability Evaluation Methods • Interviews/Focus groups • Questionnaires/Surveys Interviews/Focus Ethnographic Questionnaires/Surveys groups Observations • Ethnographic Observations • Usability Inspection Methods • Usability Testing Usability Inspection Controlled Cognitive Usability Testing Methods • Controlled Cognitive Experiments Component 15/Unit 4 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 6
Usability Evaluation Component 15/Unit 4 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 7
Interviews: Four Main Types 1. Unstructured Interview • Exploratory • Open-ended questions • What are the advantages for accessing clincial information on mobile devices? • Useful for early phase design Component 15/Unit 4 b 2. Structured • Ask predetermined questions (similar to questionnaire) • Questions are closed: response from a limited set of alternatives. • Do you visit this website: 1) daily, 2) weekly or 3) monthly? • Useful when goals are clearly understood Health IT Workforce Curriculum 2. 0/Spring 2011 Version 8
Interviews: Four Main Types (cont. ) 3. Semi-Structured Interview 4. Group Interviews/Focus groups • Most widely used • Starts with preplanned questions • Then probes the interviewee to elaborate on emergent themes • “Can you tell me a little more about your thoughts about redesigning this site? • All interviews and focus groups are typically audio recorded and transcribed Component 15/Unit 4 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 9
Phases of the Interview 1. Introduction: The interviewer introduces himself and discusses purpose of interview 2. Warm-up session: Simple questions including demographics 3. Main session: Questions are presented in a logical sequence with the more probing questions towards the end 4. Closing session: Interviewee thanks subject and signals an end to the interview Component 15/Unit 4 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 10
Usability Interview Example Usability Interview # 15 (Upstate 4) I: So I am going to ask you a few questions, if I may start by asking your age? P: 68 I: Have you lived in this area all your life? P: Yes I: The greater Pulaski area, Syracuse region? P: I have been here 15 years, cross the road 30 years, and was born about 4 miles country side. I: Pretty close. What do you do? P: I worked as a toolmaker; I am retired now. I: What was your highest level of education? P: I got some college; I went to night school for a while, I: Was this your first computer? P: Yes I: Do you monitor your blood sugar daily? P: Yes. I: Prior to receiving this computer, did you have the home monitoring kit? P: Yes. I have had monitor ever since I have had it. I: So you pretty well been monitoring it? P: Yes, for the last 8 years. Component 15/Unit 4 b Health IT Workforce Curriculum 2. 0/Spring 2011 Profile: 68 years of age Male English Speaker Originally from Pulaski area Retired toolmaker Works part-time at saw sharpening shop Some college education System was his first computer, later bought another one Lives out on the route, greater Pulaski area Lives with wife Nice, comfortable house Has exercise machine in house Avid hunter Computer in bedroom-easy access, but no chair. Keeps devices disconnected, keeps them in a different room. Keeps diabetes diary of results and values Version 11
Focus Groups • Commonly used method in marketing and social science research • Typically 3 to 10 participants representative of the sample population – For example: attending physicians, residents, nurses, pharmacists & administrators • • Discussion led by trained facilitator Preset agenda to guide discussion Sufficient flexibility to pursue emergent issues Facilitator guides and prompts discussion – Encourages less talkative participants to speak up – Discourages more verbose participants from dominating Component 15/Unit 4 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 12
Questionnaire / Surveys • Common method for collecting user demographic data and soliciting opinions • Can reach a rather large group of people • Questions are typically closed, though open comments may be solicited • Some use a mix of negative and positive questions – Require respondent to be more attentive Component 15/Unit 4 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 13
Rating Scales: Likert • • Mostly commonly used Measure opinions, attitudes and beliefs Widely used for evaluating user satisfaction Statement representing a range of opinions – I found the system to be very easy to use • Rate response on a continuum from one extreme (e. g. , highest or strongest agreement) to another (lowest or strongest disagreement) Component 15/Unit 4 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 14
Sample Usability Likert Questions Component 15/Unit 4 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 15
Rating Scales: Semantic Differential • Explore a range of bipolar attitudes about a particular item • Participant is asked to place mark in one of a number of positions between 2 extremes to indicate agreement – I found the website to be (attractive ugly) – I found the materials to be (clear confusing) Component 15/Unit 4 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 16
Online Surveys: Two Types Web-based: Email-based: • Can be interactive and can include check boxes, pull down menus, etc. • Branching logic tailor questions to specific populations • Answers written to a database Component 15/Unit 4 b • Electronic version of paper survey • Options are limited Health IT Workforce Curriculum 2. 0/Spring 2011 Version 17
Survey Monkey • http: //www. surveymonkey. com • Illustration of online survey • Focuses on user experience • Different forms of interaction – Check boxes for multiple selection – Radio buttons for choosing among alternatives or rating items on Likert scale • Configurable and flexible Component 15/Unit 4 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 18
Usability Inspection Methods: Expert Reviews Methods for examining user interface • Performed by analysts or evaluators • Reliance on judgment as a source of feedback Cost effective Usability expert (or team of experts) methodically steps through the system • Noting usability problems and violations to usability principles • Noting the severity of problems Involves assessing the usability of a system for carrying out specific tasks (i. e. , scenarios) Heuristic Evaluation: • Display based Cognitive Walkthrough: • Task analytic Component 15/Unit 4 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 19
Heuristic Evaluation • Developed by Jakob Nielsen and Rolf Molich • Method for structuring the critique of a system using simple & general heuristics • Several analysts critique a system to identify & diagnose usability problems • Display based • Each evaluator assesses the system & notes violations of usability principles • In a second pass, analyst rates severity of problems • Can be used in all phases of design including storyboards, prototypes and fully functioning systems Component 15/Unit 4 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 20
Usability Principles (Nielsen, 1993) • • • Visibility of system status Match between system and the real world User control and freedom Consistency and standards Help users recognize, diagnose, and recover from errors • Minimize memory load – Emphasize recognition rather than recall • Flexibility and efficiency • Motivation and engagement Component 15/Unit 4 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 21
More Specific Heuristics (Gerhardt-Powals, 1996) • Automate unwanted workload: – Free cognitive resources for high-level tasks – Eliminate mental calculations, estimations, comparisons, & thinking • Reduce uncertainty; display data in a clear and obvious manner • Reduce cognitive load by bringing together lower level data into a higher-level summation • Present new information with meaningful aids to interpretation: – Use a familiar framework, everyday terms, metaphors Component 15/Unit 4 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 22
More Specific Heuristics (cont. ) • Use names that are conceptually related to function – Context-dependent – Attempt to improve recall and recognition • Group data consistently meaningful ways to decrease search time • Limit data-driven tasks: – Reduce time assimilating raw data, appropriate use of color & graphics • Include only information needed by the user at a given time Component 15/Unit 4 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 23
Heuristic Evaluation Problem Description, Recommended Solution & Commentary Heuristics Violated Consistency Standards • Problem Description, Recommended Solution & Commentary P. Picture is extraneous and eats up valuable space. Minimalism S. Summary positioned above the picture. • P. Description is clear, key points should P. An excess of text Minimalism S. Convey points more succinctly, use bullets and eliminate sentences be more visually prominent. P. User is not fully aware of scope of session and perhaps program Visibility S. • Perhaps advancebullets organizer and aor timeline would be helpful S. an. Use bolded Text P. Can’t return to session from Discussion by clicking back arrow. Control/Flexibility • Heuristics Violated: Consistency P. Change Program is a misleading label for editing preferences. Match P. Help. Standards is not sufficiently informative and context sensitive. Help S. Useful if system provides some context-sensitive guidance. P. Picture extraneous and eats upaction/Control P. • Can’t return to English afteris switching to Spanish Reversible valuable space. P. Description is clear, key points should be more visually prominent. S. Use bullets or bolded Text Component 15/Unit 4 b Health IT Workforce Curriculum 2. 0/Spring 2011 Version 24
Problem Severity Combination of 3 Factors • 1. Frequency with which a problem occurs • Commonly or rarely • 2. Impact of the problem • Easy or difficult to overcome • 3. Persistence of a problem • One-time problem or constant source of difficulty Nielsen’s Rating Scale: • • • Component 15/Unit 4 b 0: Don’t agree there is a problem 1: Cosmetic problem only 2: Minor usability problem-low priority 3: Major usability problem 4: Usability catastrophe-Fix Now! Health IT Workforce Curriculum 2. 0/Spring 2011 Version 25
- Slides: 25