CHI 99 Panel Comparative Evaluation of Usability Tests

CHI 99 Panel Comparative Evaluation of Usability Tests Presentation by Rolf Molich Dialog. Design Denmark molich@dialogdesign. dk

CHI 99 Panel Comparative Evaluation of Usability Tests Take a web-site. Take nine professional usability teams. Let each team usability test the web-site. Are the results similar?

What Have We Done? l Nine teams have usability tested the same web-site – Seven professional teams – Two student teams l Test web-site: www. hotmail. com Free e-mail service

Panel Format l Introduction (Rolf Molich) l Five minute statements from five participating teams l The Customer’s point of view (Meeta Arcuri, Hotmail) l Conclusions (Rolf Molich) l Discussion - 30 minutes

Purposes of Comparison l Survey the state-of-the art within professional usability testing of websites. l Investigate the reproducibility of usability test results

NON Purposes of Comparison l To pick a winner l To make a profit

Basis for Usability Test l Web-site address: www. hotmail. com l Client scenario l Access to client through intermediary l Three weeks to carry out test

What Each Team Did l Run standard usability test l Anonymize the usability test report l Send the report to Rolf Molich

Problems Found l Total number of different usability problems found 300 l Found by seven teams six teams five teams four teams three teams two teams one team l l l 1 1 4 4 15 49 226 (75%)

Comparative Usability Evaluation 2 l Barbara Karyukina, SGI (USA) l Klaus Kaasgaard & Ann D. Thomsen, KMD (Denmark) l Lars Schmidt and others, Networkers (Denmark) l Meghan Ede and others, Sun Microsystems, Inc. , (USA) l Wilma van Oel, P 5 (The Netherlands) l Meeta Arcuri, Hotmail, Microsoft Corp. (USA) (Customer) l Rolf Molich, Dialog. Design (Denmark) (Coordinator)

Comparative Usability Evaluation 2 l Joseph Seeley, Nova. NET Learning Inc. (USA) l Kent Norman, University of Maryland (USA) l Torben Norgaard Rasmussen and others, Technical University of Denmark l Marji Schumann and others, Southern Polytechnic State University (USA)

CHI 99 Panel Comparative Evaluation of Usability Tests Presentation by Barbara Karuykina SGI, Wisconsin USA barbarak@sgi. com

Challenges: Twenty functional areas + User preferences questions

Possible Solutions: l Two usability tests l Surveys l User notes l Focus groups

Results: 26 tasks + 10 interview questions 100 findings

Challenges: Twenty functional areas + User preferences questions

Problems Found l Total number of different usability problems found 300 l Found by seven teams six teams five teams four teams three teams two teams one team l l l 1 1 4 4 15 49 226 (75%)

CHI 99 Panel Comparative Evaluation of Usability Tests Presentation by Klaus Kaasgaard Kommunedata Denmark kka@kmd. dk

Slides currently not available

CHI 99 Panel Comparative Evaluation of Usability Tests Presentation by Lars Schmidt Framtidsfabriken Networkers Denmark ls@networkers. dk

Team E Framtidsfabriken Networkers Testlab, Denmark

Key learnings CUE-2 l Setting up the test – Insist on dialog with customer – Secure complete understanding of user groups and user tasks – Narrow down test goals l Writing the report – Use screendumps – State conclusions - skip the premises – Test the usability of the usability report

Improving Test Methodology l Searching for usability and usefulness – Hook up with different methodologies (e. g. interviews) l Focus on website context – Test against e. g. Yahoo. Mail – Test against softwarebased email clients

CHI 99 Panel Comparative Evaluation of Usability Tests Presentation by Meghan Ede Sun Microsystems California, USA meghan. ede@sun. com

Hotmail Study Requests l 18 Specific Features · e. g. Registration, Login, Compose. . . l 6 Questions · e. g. "How do users currently do email? " l 24 Potential Study Areas

Usability Methods l Expert Review · 6 Reviewers · 6 Questions l Usability Study · 6 Participants (3 + 3) · 5 Tasks (with sub-tasks)

Report Description 1. Executive Summary - 4 Main High-Level Themes - Brief Study Description 2. Debriefing Meeting Summary - 7 Areas (e. g. overall, navigation, power features, . . . ) 3. Findings - 31 Sections - Study Requests, Extra Areas, Bugs, Task Times, Study Q & A 4. Study Description Total: 36 Pages - 150 Findings

Lessons Learned l Importance of close contact with product team l Consider including: · severity ratings · more specific recommendations · screen shots

Discussion Issues l How can we measure the usability of our reports? l How to deal with the difference between number of problems found and number included in report?

CHI 99 Panel Comparative Evaluation of Usability Tests Presentation by Wilma van Oel P 5 The Netherlands w. vanoel@p 5 -adviseurs. nl

Wilma van Oel P 5 adviseurs voor produkt-& kwaliteitsbeleid quality & product management consultants Amsterdam, the Netherlands

Structure of Presentation l 1. Introduction l 2. Deviations in approach – Test design – Results and recommendations l 3. Lessons for the future – Change in approach? – Was it worth the effort?

Introduction • Company: P 5 Consultants • Personal background: psychologist

Test design l l Subjects: n=11, pilot, ‘critical users’, 1 hour session Data collection: log software, video recording Methods: lab evaluation + informal approach Techniques: exploration, task execution, think aloud, interview, questionnaire Tool: SUS

A Test Session

Results and recommendations

Lessons for the future l Change in approach? – Methods: add a usability inspection method – Procedure: extensive analysis, add session time – Results: less general, severity? l Was it worth the effort? – Company: to get experience & benchmarking – Personally: to improve skills, knowledge

CHI 99 Panel Comparative Evaluation of Usability Tests Presentation by Meeta Arcuri Microsoft Corporation California, USA meeta@hotmail. com

CUE - 2 The Customer’s Perspective Meeta Arcuri User Experience Manager Microsoft Corp. , San Jose, CA

Customer Summary of Findings l l New findings ~ 4% Validation of known issues ~ 67% – Previous finding from our lab tests – Finding from on-going inspections l Remainder - beyond Hotmail Usability – Business reasons for not changing – Out of Hotmail’s control (partner sites) – Problems generic to the web

Report Content: Positive Observations 4 4 4 Quick and Dirty results Recommendations for problem fixes Participant quotes – get tone/intensity of feedback Exact # of P who encountered each issue Background of Participants Environment (browser, speed of connection, etc. )

Additional Strengths of Reports l l l Fresh perspectives Lots of data on non-US users Recommendations from participants Trend reporting Report of outdated material on site (some help files) Appreciate positive findings, comments

Report Content: Weaknesses l l Some recommendations not sensitive to web issues (performance, security) At least one finding irreproducible (not preserving fields in Reg. Form) Frequency of issue reported was sometimes vague. Some descriptions terse, vague - had to decipher

How Hotmail Will Use Results l l l Cross-validate new findings with Hotmail Customer Service reports Lots of good data to cite in planning meetings Some good recommendations given by labs and participants

Conclusion l l l Focused, iterative testing would give better results Wide array of user data very valuable Overall - good qualitative and quantitative data to help prioritize, schedule, and improve usability of Hotmail.

CHI 99 Panel Comparative Evaluation of Usability Tests Presentation by Rolf Molich Dialog. Design Denmark molich@dialogdesign. dk

Comparison of Tests l Based only on test reports l Liberal scoring l Focus on major differences l Two generally recognized textbooks: – Dumas and Redish, ”A Practical Guide to Usability Testing” – Jeff Rubin, ”Handbook of Usability Testing”

Resources A J B C D E F 123 84 (16) 130 50 107 # Usability professionals 1 3 2 6 1 1 1 3 1 Number of tests 11 4 7 6 6 6 50 9 5 G l l l Team H Person hours used for test 136 45 218

Usability Results Team H A J B C D E F # Positive findings 14 4 0 6 8 4 7 24 25 # Problems 18 26 20 150 17 10 58 75 30 % Exclusive 56 42 60 71 24 10 57 51 33 G

Usability Results A J B C D E F 150 17 10 58 75 30 % Core problems (100%=26) 38 27 31 73 35 8 58 54 50 Person hours used for test 136 45 218 123 84 NA 130 50 107 G # Problems 18 Team H 26 20

Problems Found l Total number of different usability problems found 300 l Found by seven teams six teams five teams four teams three teams two teams one team l l l 1 1 4 4 15 49 226 (75%)

Conclusion l If Hotmail is typical, then the total number of usability problems for a typical web-site is huge, much larger than you can hope to find in one series of usability tests l Usability testing techniques can be improved l We need more awareness of the Usability of Usability work

Download Test Reports and Slides http: //www. dialogdesign. dk/cue 2. htm