Usability Evaluation Ways of Studying User Interfaces Spring

Usability Evaluation Ways of Studying User Interfaces Spring 2019

Usability Evaluation Types – User-centered Evaluation • Usability Testing • Other studies involving users (card sorting, eye tracking, quantitative surveys…) – Expert-based Evaluation • Usability inspection methods Heuristic Evaluation / Cognitive Walkthrough / Pluralistic Walkthrough Heuristic Walkthrough / Perspective-based Evaluation/ … • Review-based evaluation – Model-based Evaluation • GOMS Model (Goals, Operators, Methods, Selection rules) – CMN GOMS (Original version of GOMS by Card, Moran and Newell 1983) – KLM GOMS (Keystroke Level Model GOMS) – NGOMSL (Natural GOMS Language) – CPM GOMS (Cognitive Perceptual Motor / Critical Path Method GOMS) Usability Engineering - Ways of Studying User Interfaces (Part A) 2

Usability Testing • Evaluation of a product or service by testing it with representative users (Best way to understand how real users experience a product or service) • Involves systematic observation under controlled conditions • During a test Participants - try to complete typical tasks Observers - watch, listen and take notes. • Main goals: Collect quantitative and qualitative data Analyze data to identify any usability problems Determine the participant's satisfaction with the product. Usability Engineering - Ways of Studying User Interfaces (Part A) 3

What usability testing is not Usability testing is not simply gathering opinions on a product or service (which is market research or qualitative research) Usability testing involves watching people trying to use something for its intended purpose rather than showing users a product and asking, "Do you understand how to use it? ” Example: Testing usability of an instruction pamphlet for a Lego toy Participants should be given the instruction pamphlet plus a pile of Lego bricks to put the toy together (rather than being asked to comment on the quality and ease of the instruction pamphlet itself) Usability Engineering - Ways of Studying User Interfaces (Part A) 4

Usability testing measures Behavior, not Preference Users are notoriously bad at articulating what they want Asking a user ‘what do you want’ is rarely -if ever- helpful “If I had asked people what they wanted, they would have said faster horses. ” Henry Ford • By observing and measuring behavior, we can understand what best supports their goals and motivation Hence: • Best questions to ask: the ones exploring - what people already do - how they do it - why they do it Usability Engineering - Ways of Studying User Interfaces (Part A) 5

Usability tests are performed in order to • Learn if participants are able to complete specified tasks successfully • Identify how long it takes to complete specified tasks • Find out how satisfied participants are with the product • Identify changes required to improve user performance and satisfaction • Analyze the performance to see if it meets preset usability objectives Main tasks in performing an effective usability test 1. Development of the Test Plan 2. Recruitment of participants 3. Carrying out the Usability Test 4. Analysis and report of the findings . Usability Engineering - Ways of Studying User Interfaces (Part A) 6

Benefits of Usability Testing • Identifies problems users might encounter with a system - The earlier issues are identified, the less expensive the fixing will be ü in terms of staff time ü in terms of possible impact to the schedule If performed in early stages of development, problems are identified even before they are coded • Decreases support cost • Increases revenues • Makes happier customers • Strengthens brand name Usability Engineering - Ways of Studying User Interfaces (Part A) 7

Factors affecting the cost of a usability test • Size of the team assembled for testing • Number of participants for testing • Test duration (number of days) • Type of testing performed • Number of Usability Tests Remember: “Building” / “applying” usability on any product is an iterative process Usability Engineering - Ways of Studying User Interfaces (Part A) 8

Usability Lab A place where usability testing is taking place. • An environment where users are studied interacting with a system for the sake of evaluating the system's usability. • The user sits in front of a computer or stands in front of the systems interface, alongside a facilitator(*) who gives the user tasks to perform. (*) Facilitator: A person that helps and guides other people towards a goal, with a ‘content neutral’ behavior (neither positive nor negative attitude). In most usability tests, one of the observers also acts as facilitator Usability Engineering - Ways of Studying User Interfaces (Part A) 9

Usability Lab • A number of observers watch the interaction, make notes, and ensure the activity is recorded. • Observers are usually in a different room (observing room) and watch user through a camera or oneway mirror • Typically, sessions are videotaped and there is specialized software that logs interaction details. • In some cases the testing and the observing room are not placed alongside. In this case the video and audio observation are transmitted through the network Usability Engineering - Ways of Studying User Interfaces (Part A) 10

Usability Lab Examples cameras one-way mirror Usability Engineering - Ways of Studying User Interfaces (Part A) 11

Usability Lab Examples cameras Usability Engineering - Ways of Studying User Interfaces (Part A) 12

Portable Usability Lab (software testing) for usability tests on-the-go Minimum configuration • • A full-size laptop for the user equipped with a real mouse (no touchpad) A second laptop for the facilitator (could be smaller) A webcam to record the user's thinking aloud comments and facial expressions Appropriate software (Morae, Camtasia etc) Usability Engineering - Ways of Studying User Interfaces (Part A) 13

Usability Test Video Example In this video example the user is presented a university website and is asked to carry out the following three tasks: 1. Find the name and email of the Dean of Arts & Sciences 2. Who is the advisor for public relationships department 3. Find the location of the “Writing Center” Click on the image to play video • The moderating Technique used for this session is Concurrent Think- Aloud (CTA) • The session was recorded using ‘Morae’, a software environment for recording and processing usability tests. Usability Engineering - Ways of Studying User Interfaces (Part A) 14

Usability Testing in a Nutshell Planning / Running / Analyzing / Reporting 1. Getting Started Identify the target audience (in general consists of more than one user groups) 2. Creating Usability Test Tasks • Typically, participants will perform a set of 5 to 10 tasks within a 90 -minute session • Tasks should represet the most common user goals. If a website is tested, tasks may also represent the most important conversion goals from the owner’s perspective (e. g. making a purchase) Conversion: any action taken by a user that satisfies the website owner’s business goals. making a purchase, signing up for an email newsletter, viewing an important webpage etc. Usability Engineering - Ways of Studying User Interfaces (Part A) 15

Usability Testing in a Nutshell - 2 3. Conducting a Usability Test • The researcher reads a participant one task at a time, such as “Find out how to contact technical support, ” and allows the participant to complete the task without any guidance. • To prevent bias, the researcher follows the same “script” when explaining the task to each participant. • The researcher may also ask the participant to talk aloud as he works on a task to better understand the participant’s mental model for the task and his decision-making in real time. • When the participant has completed a task, the researcher sets up the starting point for the next task and continues the test. • Ideally, task order is counterbalanced from participant to participant. Counterbalancing: A technique in experimental design that is used to avoid the introduction of confounding variables. In usability testing, this technique is most commonly used when establishing task order. (find the sitemap / sign up for the email newsletter) Usability Engineering - Ways of Studying User Interfaces (Part A) 16

Usability Testing in a Nutshell - 3 4. Usability Test Analysis • Usability testing recording software may be used to record the computer screen and the participant’s voice and facial expressions during testing. Morae, Camtasia… • This software can also facilitate tracking of user behaviors, including mouse clicks, keystrokes, and active or open windows. • When all participants have completed the study, the researcher will compile the data to determine the severity of each usability issue that was encountered and provide prioritized recommendations for the development team to meet usability requirements. • Usability testing should be conducted at various times throughout the iterative design process to ensure that all usability requirements have been met in the final product. Usability Engineering - Ways of Studying User Interfaces (Part A) 17

Planning a Usability Test Elements of a Test Plan • Scope - Indicate what you are testing - Specify how much of the product the test will cover e. g. prototype as of a specific date / navigation /navigation and content … • Purpose - Identify concerns, questions, and goals for this test can be broad or specific “can users navigate to important information from the prototype’s home page? ” “will users easily find the search box in its present location? ” concerns drive the scenarios you choose for your tests • Schedule & Location - where / when / how long Usability Engineering - Ways of Studying User Interfaces (Part A) 18

Planning a Usability Test Elements of a Test Plan – continued • Participants - number of users: equally distributed in classes - types (classes) of users 3 - 4 different classes usually enough to cover most of the target audience • Sessions - one per user - description: Session ID / class / user ID (no real names!) - length: typically 60’ to 90 minutes - allow 30 minutes buffer time between sessions > reset the environment, > briefly review the session > compensate for late (or early) arrival of next participant Usability Engineering - Ways of Studying User Interfaces (Part A) 19

Planning a Usability Test Elements of a Test Plan – continued • Equipment & stuff - user’s equipment: desktop / laptop / smartphone / tablet… - evaluator’s equipment: PC(s), camera(s), - forms, paper, office supplies, coffee, water, small gifts of gratitude… • Scenarios - number and types of tasks in each session - typical time per task: 5’ to 10’ • Metrics - subjective: questions to be asked at various stages of the testing - quantitative: successful completion rates, error rates, time on task… • Roles - list of staff who will participate and what role each will play facilitator, evaluators, staff leader…. Usability Engineering - Ways of Studying User Interfaces (Part A) 20

Planning a Usability Test Examples of Quantitative Test Metrics • Time on task: amount of time it takes the participant to complete the task • Number and rate of successful & unsuccessful task completions - a scenario is successfully completed when the participant indicates they have found the answer or completed the task goal. • Number and rate of critical and non-critical errors - Critical: inability to complete a task of the scenario - Non- critical: recovered by the participant and do not result in the participant’s ability to successfully complete the task • Error-free rate: percentage of test participants who complete the task without any errors Usability Engineering - Ways of Studying User Interfaces (Part A) 21

Planning a Usability Test Examples of Subjective Test Metrics • Self-reported Participant Ratings − satisfaction / ease of use / ease of finding information etc. − rated on a 5 to 7 -point Likert scale • Likes, dislikes and recommendations (open-end questions) Usability Engineering - Ways of Studying User Interfaces (Part A) 22

Running a Usability Test Choosing a Moderating Technique • Concurrent Think Aloud (CTA) Participants “think aloud” while they interact with the product. Encourages a running stream of consciousness as they work. • Retrospective Think Aloud (RTA) Participants are asked to retrace their steps when the session is complete. Often participants watch a video replay of their actions • Concurrent Probing (CP) As participants work on tasks—when they say something interesting or do something unique, the researcher asks follow-up questions • Retrospective Probing (RP) Waiting until the session is complete and then asking questions about the participant’s thoughts and actions for specific points of interest Usability Engineering - Ways of Studying User Interfaces (Part A) 23

Running a Usability Test Example of a Typical Usability Test Session 1. The facilitator welcomes the participant and explain the test session, ask the participant to sign the “consent form”, and ask any pre-test or demographic questions. 2. The facilitator explains the moderating technique (e. g. CTA) and asks if the participant has any additional questions. The facilitator explains where to start. 3. The participant reads the task scenario aloud and begins working on the scenario 4. The other members of the staff take notes of the participant’s behaviors, comments, errors and completion (success or failure) on each task. Usability Engineering - Ways of Studying User Interfaces (Part A) 24

Running a Usability Test Example of a Typical Usability Test Session - continued 5. The session continues until all task scenarios are completed or a predefined time limit has reached time limits may also exist for each individual task scenario to avoid user frustration and deadlock 6. The facilitator asks the end-of session subjective questions (de-briefing), thanks the participant, gives him a small token of appreciation and escorts him out. 7. The facilitator then resets the materials and equipment, speaks briefly with the observers and calls in the next participant Usability Engineering - Ways of Studying User Interfaces (Part A) 25

Analyzing Usability Test Data Use Spreadsheets to record analyze data for each task scenario − Success rates − Task time − Critical / non critical errors − “Quantitated” qualitative data (as in Likert scales) − Participant’s demographic data to observe variations due to demographic variables (sex, age, occupation, education…) Categorize detected usability problems by severity − Critical: If not fixed, users will not be able to complete the scenario − Serious: If not fixed, many users will be frustrated and some may give up − Minor: Users are annoyed, but this does not keep them from completing the scenario. This should be revisited later. Usability Engineering - Ways of Studying User Interfaces (Part A) 26

Writing the Usability Test Report Background Summary: – what has been tested – where and when – equipment information – actions performed before, during and after the test – the testing team – brief description of problems encountered – brief description of what worked well Methodology: – description of test sessions – type of interface between user and testing team – metrics collected – overview of task scenarios – overview of participants Usability Engineering - Ways of Studying User Interfaces (Part A) 27

Writing the Usability Test Report Test Results: – summary of the successful task completion rates • by participant • by task – average success rate • by participant • by task – percentage of participants who completed each scenario – percentage of participants who completed all scenarios – average time taken to complete each scenario – participant comments (most illustrative of them). Usability Engineering - Ways of Studying User Interfaces (Part A) 28

Writing the Usability Test Report Findings and Recommendations: – three reporting options a. a single overall list of findings and recommendations b. findings and recommendations scenario by scenario c. both a list of major findings and recommendations that cut across scenarios as well as a scenario-by-scenario report. – each finding (or group of related findings) should: • have a basis in data (what you actually saw, heard and noted) • include a statement of the situation (as specific as possible) • include recommendations on what to do. Useful to also report positive findings. What is working well must be maintained through further development. Usability Engineering - Ways of Studying User Interfaces (Part A) 29

Writing the Usability Test Report What Not to include – The names of participants refer to them as User A, User B… – Descriptions of participants from which a reader could deduce identities – Generalized conclusions that are misleading due to poor sampling or other factors – Test hypotheses / unsubstantiated opinions / biases – Entire listings of all of the raw data and observations. (supply only those necessary, and provide summaries as needed) Usability Engineering - Ways of Studying User Interfaces (Part A) 30

Ethical Issues in Usability Work involving Users The important ethical concern is: Do no harm to the users. What harm can be done: Mostly psychological Users feel: – Pressure to conform Sometimes, highly specialized experts feel particularly stressed – Inadequate or stupid when they make errors – Worried about what might happen to information being gathered about them – Concerned that information falling into the wrong hands can influence managerial decision-making Avoid management involvement in usability sessions Usability Engineering - Ways of Studying User Interfaces (Part A) 31

What must you as a Usability Engineer do when working with Users? • Before the session Make appropriate preparations and checks Explain certain things to users and answer to possible questions • During the session Observe users and behave appropriately • After the session Debrief and express gratitude to users (thank you, small gift) Debriefing: a process of questioning to gain information from an individual after a mission, and to instruct the individual as to what information can be released to the public and what information is restricted. (military origin) Above points apply both to when measuring or merely observing (whether recording or not) Usability Engineering - Ways of Studying User Interfaces (Part A) 32

Things to prepare and check before the session Learn the software yourself – Have a moderate ability with most features so as to • better understand what users are doing • prepare tasks • help users if they are stuck Decide on – Whether you will be measuring time (e. g. to gather metrics) or just observing to find problems – Whether to focus on learnability or efficiency of use Effects choice of participants – Whether to have longer tasks or a series of very short tasks – Level of interaction with participants during sessions Minimize interaction if you are timing the tasks, otherwise think-aloud interaction is best Usability Engineering - Ways of Studying User Interfaces (Part A) 33

Things to prepare and check before the session (continued) Conduct high level task analysis – Make sure tasks are carefully designed so as to be understandable and within user capabilities – Run a pilot test, preferably with another member of your team Select participants (well in advance) Plan for more participants than you really need in case one or more backs out Prepare “informed consent forms” for participants to sign (more later) Make sure the test software and recording equipment actually works So as not to frustrate the user - let alone yourself Usability Engineering - Ways of Studying User Interfaces (Part A) 34

Things to explain to users before the session Tell users about the work in advance. Explain (orally and in writing): – The nature and goals of the evaluation in particular that they are not being evaluated as individuals – – – How long it will take What recording or measurement is being done What will happen to the data The fact that there are no known risks to them (presumably) If in a company: That their manager has given them permission to participate, but he/she will not see individual results. – That they are free to participate or not, and to withdraw at any time – Any other rights (e. g. the right to see the results) – Who they can complain to Give users to sign informed consent forms Usability Engineering - Ways of Studying User Interfaces (Part A) 35

Informed Consent Form Example Usability Engineering - Ways of Studying User Interfaces (Part A) 36

Things to do During the session • Keep the environment relaxed. Leave plenty of free time • Give the users one or more 'easy' tasks to do first, for warm-up easy tasks boost confidence. (exclude warm-up tasks from results) • Hand the user test tasks one at a time so users don't feel overwhelmed about having too many things to do It is not their fault if they are slow For each task: – Read it to the user – Hand it to the user so they can re-read when necessary – Ask the user to speak out loud, explaining what they are doing and why not all users are good at this, so be patient remind user from time to keep talking Usability Engineering - Ways of Studying User Interfaces (Part A) 37

During the session - continued • Avoid coaching the user, or letting him/her struggle endlessly Try to balance. Either extreme can make the user feel inadequate • Periodically, pose questions to stimulate the user to give useful information (see next slide) • Refrain from laughing or making negative comments about what the user is doing • Ensure there are no disturbances, and no 'audience'. • Limit sessions to 1. 5 hours or less with a 10 minute break in the middle Don't use the same participant more than once in a day • Provide coffee etc. for sessions longer than an hour. • Terminate the session if the user becomes overly frustrated. Usability Engineering - Ways of Studying User Interfaces (Part A) 38

During the session - continued Questions to stimulate the user – What do you want to do? They do not know; the system cannot do what they want. – What do you think would happen if. . . ? They do not know; they give wrong answer. – What do you think the system has done? They do not know; they give wrong answer. – What do you think is this information telling you? They do not know; they give wrong answer – Question Problem if. . . – Why did the system do that? They do not know; they give wrong answer. – What were you expecting to happen? They had no expectation; they were expecting something else. Usability Engineering - Ways of Studying User Interfaces (Part A) 39

After the session • Debrief the user – Let them tell you anything they want – Ask them specific questions about difficulties they had – Tell them that they helped you find problems and that their assistance is appreciated – Tell them that it may not be possible to fix all the problems, but you will try. – Remind them that the results will be confidential – Consider giving them a token of appreciation (pin, thank-you note etc. ) • Keep the data confidential – Refer to users as 'User 1', 'User 2' etc. as long as nobody can 'read between the lines' and infer who is who – When asking users for permission to publish specific video clips, be very careful to ensure they are freely agreeing. Usability Engineering - Ways of Studying User Interfaces (Part A) 40

The Importance of Video Recording • Really important to see emotional responses and hear users’ feedback • Without it, 'you see only what you want to see’ (i. e you interpret what you see based on your mental model) • In the 'heat of the moment' you miss many things • Minor details (e. g. body language) captured • You can repeatedly analyze, looking for different problems Usability Engineering - Ways of Studying User Interfaces (Part A) 41

Usability Evaluation Types – User-centered Evaluation • Usability Testing • Other usability studies (card sorting, eye tracking, quantitative surveys…) – Expert-based Evaluation • Usability inspection methods Heuristic Evaluation / Cognitive Walkthrough / Pluralistic Walkthrough (*) Heuristic Walkthrough / Perspective-based Inspection / … • Review-based evaluation – Model-based Evaluation • GOMS Model (Goals, Operators, Methods, Selection rules) – CMN GOMS (Original version of GOMS by Card, Moran and Newell 1983) – KLM GOMS (Keystroke Level Model GOMS) – NGOMSL (Natural GOMS Language) – CPM GOMS (Cognitive Perceptual Motor / Critical Path Method GOMS) Usability Engineering - Ways of Studying User Interfaces (Part A) 42

Usability Inspection Methods Family of methods used by experienced practitioners to assess usability issues • Analytic techniques - no users involvement • Generate results for a fraction of the time and cost of usability testing. • Used to supplement, NOT replace, direct user involvement Members of the Usability Inspection Methods family • Heuristic Evaluation most prominent • Cognitive Walkthrough • Pluralistic Walkthrough • • • Heuristic Walkthrough Perspective Based Evaluation Review Based Evaluation Usability Engineering - Ways of Studying User Interfaces (Part A) 43

Heuristic Evaluations / Cognitive Walkthroughs • Also referred to as Expert Reviews because they are usually performed by experts in Usability or HCI. • Best results obtained when • evaluators are Double Experts (have expertise in both HCI and the specific domain of the application) • evaluators go through the interface at least twice • First pass: get a feel for the flow of the interaction and the general scope of the system. • Second pass: focus on specific interface elements while knowing how they fit into the larger whole. Heuristic Evaluations and Cognitive Walkthroughs should be done prior to and in addition to user-testing, not instead of user-testing Usability Engineering - Ways of Studying User Interfaces (Part A) 44

Heuristic Evaluation − Usually part of an iterative design process − Basis for evaluation: a specific list of design guidelines, frequently called ‘heuristics’. − System reviewed by a small number of evaluators - often UI experts or designers − Evaluators comment on usability problems in relation to each heuristic − Evaluators decide on their own how they want to proceed with evaluating the interface How many evaluators? – The more the best – obviously – But: if cost is taken into account, we must find the optimal number Usability Engineering - Ways of Studying User Interfaces (Part A) 45

Lists of Heuristics There is not a single set of Heuristics. – Nielsen's 10 famous heuristics (remember them? ) the ones mostly used but there also others: – Gerhardt-Powals 10 Cognitive Engineering Principles – Bastien and Scapin 18 Ergonomic criteria – Connell & Hammond's 30 Usability Principles – Smith & Mosier's 944 guidelines for the design of user-interfaces – and more… Usability Engineering - Ways of Studying User Interfaces (Part A) 46

Original List of Nielsen Heuristics Proposed by Nielsen and Molich in 1990 – contains 9 items (Replaced in 1993 by a 10 item revised list ) 1. Simple and natural dialog Simple means no irrelevant or rarely used information. Natural means an order that matches the task. 2. Speak the user’s language Use concepts from the user’s world Don’t use system-specifc engineering terms. 3. Minimize user memory load Don’t make the user remember things from one action to the next. Leave information on the screen until it is no longer needed. 4. Be consistent Action sequences learned in one part of the system should apply in other parts. Usability Engineering - Ways of Studying User Interfaces (Part A) 47

Original List of Nielsen Heuristics - 2 5. Provide feedback Let users know what effect their actions have on the system. 6. Provide clearly marked exits. If users get into a part of the system that doesn’t interest them, they should be able to get out quickly without damaging anything. 7. Provide shortcuts Help experienced users avoid lengthy dialogs and informational messages they don’t need. 8. Good error messages Let the user know what the problem is and how to correct it. 9. Prevent errors Whenever you discover an error message; ask if that error could have been prevented. Usability Engineering - Ways of Studying User Interfaces (Part A) 48

Revised List of Nielsen Heuristics (1993) Most common list of heuristics in use today 1. Visibility of system status The system should always keep users informed about what is going on, through appropriate feedback within reasonable time. 2. Match between system and real world. The system should speak the users’ language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order. 3. User control and freedom Users often choose system functions by mistake and will need a clearly marked “emergency exit” to leave the unwanted state without having to go through an extended dialogue. Support undo and redo. 4. Consistency and standards Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions. 5. Error prevention Even better than good error messages is a careful design, which prevents a problem from occurring in the first place. Usability Engineering - Ways of Studying User Interfaces (Part A) 49

Revised List of Nielsen Heuristics (1993) - 2 6. Recognition rather than recall Make objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate. 7. Flexibility and efficiency of use Accelerators -- unseen by the novice user -- may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions 8. Aesthetic and minimalist design Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility. 9. Help users recognize, diagnose and recover from errors. Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution. 10. Help and documentation Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user’s task, list concrete steps to be carried out, and not be too large. Usability Engineering - Ways of Studying User Interfaces (Part A) 50

Gerhardt-Powals’ cognitive engineering principles (1996) Similar to Nielsen’s heuristics but take a more holistic approach to evaluation 1. Automate unwanted workload: • • free cognitive resources for high-level tasks. eliminate mental calculations, estimations, comparisons, and unnecessary thinking. 2. Reduce uncertainty: • display data in a manner that is clear and obvious. 3. Fuse data: • reduce cognitive load by bringing together lower level data into a higher-level summation. 4. Present new information with meaningful aids to interpretation: • • use a familiar framework, making it easier to absorb. use everyday terms, metaphors, etc. Usability Engineering - Ways of Studying User Interfaces (Part A) 51

Gerhardt-Powals’ cognitive engineering principles (1996) - 2 5. Use names that are conceptually related to function: • • Context-dependent. Attempt to improve recall and recognition. 6. Group data in consistently meaningful ways to decrease search time 7. Limit data-driven tasks: • • Reduce the time spent assimilating raw data. Make appropriate use of color and graphics. 8. Display only that information needed by the user at a given time. • • Allow users to remain focused on critical data Exclude extraneous information that is not relevant to current tasks 9. Provide multiple coding of data when appropriate. 10. Practice judicious redundancy (to resolve the possible conflict between heuristics 6 and 8). Usability Engineering - Ways of Studying User Interfaces (Part A) 52

Cognitive Walkthrough • Focus: Evaluation of system's learnability for new or infrequent users. • Originally designed as a tool to evaluate walk-up-and-use systems where users would have little or no training ATMs, Info kiosks, interactive exhibits in museums… • Has been employed successfully with more complex systems like CAD software and software development tools to understand the first experience of new users. • The evaluation can be performed by the system's designers in the early stages of design, before user testing is possible • Generates results quickly with low cost (compared to usability testing) Cognitive Walkthrough is Task-specific Heuristic Evaluation takes a Holistic view Usability Engineering - Ways of Studying User Interfaces (Part A) 53

Cognitive Walkthrough One or more evaluators “walk” through a set of the most typical user tasks supported by the system, one-step-at-a-time asking themselves a set of questions from the perspective of the user • Does the user know what to do at this step? • If user does the right thing, will that user know if he has made progress toward the task goal? – Will the user actually be trying to do the right action ? Will the user know what to do? – Is the control for the action visible ? Will the user be able to physically see the control on the interface? – Is there a strong link between the control and the action? Will the user understand that what he is seeing is actually what he needs to use? – Is there appropriate feedback after the initiation of the action? Will the user understand the feedback he gets after performing the action? If answer is “no” to any of these questions then a usability defect is recorded. Usability Engineering - Ways of Studying User Interfaces (Part A) 54

Typical CW Task Form Prepared & filled by evaluators For each task of the walkthrough Task 4: View and print most recent bank transactions using the ATM Will user see what to do? Will user understand what to do? 1. Insert card & Yes No wait for Comments: instructions Yes No Comments: 2 Enter Pin Yes No Comments: 3 Select menu action Yes No Comments: Step 4. …. . Will user know what to do? …. . Usability Engineering - Ways of Studying User Interfaces (Part A) Will user understand the feedback they get? …. . 55

Cognitive Walkthrough • CW also takes into account the user profile, – especially the users' knowledge of the task domain and interface. • Best results obtained when evaluators are Double Experts (Similarly to Heuristic Evaluation) • For each task to be examined, an optimal sequence of actions needed to complete is prepared (Happy Path) • Sometimes alternate paths are also explored describing what happens when mistakes occur – deviation from happy path. Happy Path a. k. a. "sunny day" scenario The sequence of activities that will be executed towards a task if everything goes as expected without exceptions Usability Engineering - Ways of Studying User Interfaces (Part A) 56

Happy Path • Created for each task to be “walked-through” • Contains the complete list of the actions needed to complete the task in an optimal way Example: Happy path for creating a customized voicemail message on an i. Phone: 1. Tap Voicemail. 2. Tap Greeting. 3. Tap Custom. 4. Tap Record and speak your greeting. 5. When you finish, tap Stop. 6. To listen to your greeting, tap Play. 7. To re-record, repeat steps 4 and 5. 8. Tap Save. Usability Engineering - Ways of Studying User Interfaces (Part A) 57

Un-happy Path ? • Sometimes, creating the happy path is all you need to do to realize there is a problem with the interface. • For example, configuring email on a Nokia 6233 requires 78 separate steps! • If your happy path has this many actions, there's no need to continue with the review: you've found a serious problem already. Usability Engineering - Ways of Studying User Interfaces (Part A) 58

Nokia 6233 email configuration, all 78 steps! 1. Press the Menu key. 2. Scroll to Settings and press Select. 3. Scroll up to Configuration and press Select. 4. Scroll up to Personal Configuration Settings and press Select. 5. Press Options. 6. Scroll to Add New and press Select. 7. Scroll to E-Mail and press Select. 8. Account name will be highlighted, press Select. 9. Enter Vodafone Mail and press OK. 10. Scroll to My Name and press Select. 11. Enter your name (i. e. John) and press OK. 12. Scroll to E-mail Address and press Select. 13. Enter your email address (i. e. john. smith@vodafone. net) and press Ok. 14. Scroll to Include Signature and enter a signature, if you wish. 15. Scroll to Reply-to Address and press Select. 16. Enter your email address (i. e. john. smith@vodafone. net) and press Ok. 17. Scroll to SMTP Secure Login, leave unless required by your Email provider. 18. Scroll to SMTP User Name and press Select. 19. Enter the Account Username (i. e. john. smith) press Ok. 20. Scroll to SMTP Password and press Select. 21. Enter your email account password and press Ok. Then verify the Email password and press Ok. 22. Scroll to Outgoing SMTP Server and press Select. 23. Enter send. vodafone. net and press Ok. 24. Scroll to Outgoing SMTP Port and press Select. 25. Enter 25 unless given a different port number by your email account provider and press Ok. 26. Scroll to Use Preferred access point and press Select. 27. Scroll to No and press Select. 28. At Define access point Settings press OK. 29. Scroll to SMTP Access Point and press Select. 30. Proxy will be displayed, press Select. 31. Scroll to Disable and press Select. 32. Scroll to Data Bearer and press Select. 33. Scroll to Packet Data and press Select. 34. Scroll to Bearer Settings and press Select. 35. Packet Data Access point will be displayed, press Select. 36. Enter internet and press Ok. 37. Scroll to Network Type and press Select. 38. Scroll to IPv 4 and press Select. 39. Scroll to Authentication Type and press Select. 40. Scroll to Normal and press Select. 41. Scroll to Username and press Select. 42. Enter web and press OK. 43. Scroll to Password and press Select. 44. Enter web and press OK, to verify password reenter web again and press the select key under OK. 45. Press the Right Select key under Back twice. 46. Scroll to Incoming Server Type and press Select. 47. Scroll to POP 3 or IMAP 4 (depending on email account type use IMAP 4 for. Vodafone Mail) and press Ok. 48. Scroll to Incoming Mail Settings and press Select. 49. Retrieve E-mails will be displayed, make no changes. 50. Scroll to POP 3 secure Login/IMAP secure Login and set to enabled/disabled as required by you ISP. If required by your ISP you will now need to enter your Username again at POP 3 user name/IMAP user name SMTP and Password again at POP 3 password/IMAP password. 51. Scroll to POP 3 User name/IMAP 4 User name 52. Enter the Account Username (i. e. john. smith) and press Ok. 53. Scroll to POP 3 Password/IMAP 4 User name and press Select. Usability Engineering - Ways of Studying User Interfaces (Part A) 54. Enter the Email password set by the user and press Ok. Then Verify the Email password and Press Ok. Note: The password and username can be case sensitive. 55. Scroll to Incoming POP 3 Server/Incoming IMAP 4 Server and press Select. 56. Enter the incoming mail server provided by your email provider for. Vodafone Mail enter imap. vodafone. net and press Ok. 57. Scroll to Incoming POP 3 port/Incoming IMAP 4 Port and enter: For POP 3: 110 For IMAP 4: 143 58. Scroll to Use Preferred access point and press Select. 59. Scroll to No and press Select. 60. Scroll to POP 3/IMAP 4 Access Point and press Select. 61. Proxy will be displayed, press Select. 62. Scroll to Disable and press Select. 63. Scroll to Data Bearer and press Select. 64. Scroll to Packet Data and press Select. 65. Scroll to Bearer Settings and press Select. 66. Packet Data Access point will be displayed, press Select. 67. Enter internet for contract or enter pp. vodafone. co. uk for PAYT, press Ok. 68. Scroll to Network Type and press Select. 69. Scroll to IPv 4 and press Select. 70. Scroll to Authentication Type and press Select. 71. Scroll to Normal and press Select. 72. Scroll to Username and press Select. 73. Enter web and press OK. 74. Scroll to Password and press Select. 75. Enter web and press OK, re-enter web and press OK. 76. Press the Right Select key under Back three times. 77. Scroll to the new email settings and press Options. 78. Scroll to Activate and Press Select. 59

What HE and CW have in Common Do not employ user testing Offer a potentially cheaper way of identifying problems at all stages of the development process Uncover a substantial percentage of usability problems 30 to 90% (various studies) Provide early feedback, especially when design is iterative Difficult to test with users on early iteration stages of a prototype Remember however that: Heuristics Evaluations and Cognitive Walkthroughs should be done prior to and in addition to usability testing not instead of usability testing Usability Engineering - Ways of Studying User Interfaces (Part A) 60

Heuristic Walkthrough Combines aspects of heuristic evaluation and cognitive walkthrough Participants make two passes: - First pass uses "thought-provoking" questions and requires the evaluators to work through a set of prioritized tasks. - Second pass requires evaluators to use a set of heuristics to find additional problems. The assumption here is that the task-based review will enhance the heuristic review. Usability Engineering - Ways of Studying User Interfaces (Part A) 61

Perspective-based Evaluation Members of a product team evaluate a product from assigned perspectives. • Perspectives can be based on many things including: - personas - principles (e. g. , prevent errors) - age groups - frequency of use - disabilities -. . . Participants are asked to elicit (bring out) usability problems based on an understanding of the perspective. This method can complement other methods like cognitive walkthroughs and heuristic evaluations. Usability Engineering - Ways of Studying User Interfaces (Part A) 62

Pluralistic Walkthrough In between method combining user-centered & expert-based evaluation – Collaborative usability evaluation involving both users and members of the product team. – Employed to generate early design evaluation Method: • Users are guided through tasks simulated on hard-copy panels indicating how they would proceed. • Developers and other members of the product team address concerns or questions about the interface. • For each step in a task - participants are presented with a draft image of the interface and asked to indicate what they would do get to the next step - After the participants have indicated their actions for each step, the facilitator reveals the “correct answer” and invites discussion. See an example video here Usability Engineering - Ways of Studying User Interfaces (Part A) 63

Review Based Evaluation Relies on experimental results and empirical evidence from the literature (psychology, HCI, etc. ) in order to support or disprove parts of the user interface design. • It’s expensive to repeat experiments continually • An expert review of relevant literature can avoid the need to do so • Caution needed to ensure that literature results are transferable to our design Example An experiment testing the usability of a particular help system using novice participants may -or may not- provide accurate evaluation of a help system designed for expert users Usability Engineering - Ways of Studying User Interfaces (Part A) 64

End of presentation Questions? Usability Engineering - Ways of Studying User Interfaces (Part A) 65