Larson 6 Usability Test 1 Test Plan 1

Larson 6 Usability Test 1

Test Plan 1. 2. 3. 4. 5. 6. 7. Purpose Problem statement/test objectives Subject profile Test design Monitor Evaluation measures (data to be collected) Report contents and presentation

1. Purpose • Which type of test? – Comparison test—Which is “better? ” – Pilot test—Test the test procedure – Field test—Can real users really benefit by using the system – Beta test—Users help debug the system – Acceptance test—Validate that the system satisfy the requirements? • What do we hope to learn?

2. Test objectives • Avoid unfocused and vague problem statements • Examples of poor objectives – Is the current product usable? – Is the product ready for release or does it need more work? • Examples of good objectives – Do the screens reflect the content model? – Is help easier to access via a “hot key” or via a mouse selection?

3. Subject profile • General computer experience – Range: none to two years • Education: – Range: 10% high school, 60 % college, 20% masters 10% Phd • Age – Range: 85% ages 20 -50, 15% other • Gender • Education Major: 0% CS, 100% other

4. Test design • Detailed plan for conducting the test – Groups of subjects, e. g. , • Group A: Mary, Fred, Sam, Jose • Group B: Sue, Ron, Bob, Sally – Tests • Group A does test 1 • Group B does test 2 – Subjects read from printed instruction script

Instruction script • Orientation – Introduce yourself – Offer refreshments – Explain why subjects are here • To test a new application • Stress that the subject is not being tested – Describe the equipment the subject will use – Explain that session is being recorded • Nondisclosure form • Lists of tasks to perform. For each task: – Provide data user needs to perform the task – Do not explain what commands to widgets the user should use. • Written debriefing questionnaire

In class exercise • Write Test Instruction Script for your application 8

5. Monitor the test • • • Try to be objective Enable rather than lead the subject Don’t act too knowledgeable Don’t jump to conclusions Let the subject struggle

6. Evaluation measures • Performance – Data collected during the test • Preference – Opinions collected during debreafing

Two types of usability tests Is the application useful? Performance Measure what the user actually accomplished Validate that users achieved success Is the application enjoyable? Preference Measure users’ likes and dislikes Validate that users enjoyed the application and will uses it again

Performance Metrics Caller Task Measure Typical Criteria Locate “Modern Girl” by Sleater-Kinney Complete successfully Less than 40 seconds Purchase “Modern Girl” Complete successfully Less than 2 minutes

Performance Metrics Caller Task Measure Typical Criteria Locate “Modern Girl” by Sleater-Kinney Complete successfully Less than 40 seconds Purchase “Modern Girl” Complete successfully Write performance metrics for your application Less than 2 minutes

Preference Metrics Question Typical Criteria On a scale from 1 to 10, rate the help facility. The average caller score is greater than 7. On a scale from 1 to 10, rate the understandability of the error messages The average caller score is greater than 7. Would you recommend using this applications to a friend? Over 75% of callers respond by saying “yes. ” What would you be willing to pay to use this voice-enabled application? Over 75% of callers indicate that they are willing to pay $1. 00 or more per use.

Preference Metrics Question Typical Criteria On a scale from 1 to 10, rate the help facility. The average caller score is greater than 7. On a scale from 1 to 10, rate the understandability of the synthesized voice. The average caller score is greater than 7. Would you recommend using this applications to a friend? Over 75% of callers respond by saying “yes. ” Write preference metrics What would you be willing to pay to Over 75% of callers indicate that use this voice-enabled they are willing to pay $1. 00 or for application? your application more per use.

Preference Metrics (Open-ended Questions) • What did you like the best about the voice-enabled application? Do not change these features • What did you like the least about the voice-enabled application? Consider changing these features • What new features would you like to have added? Consider adding these features in this or a later release. • What features do you think you will never use? Consider deleting these features. • Do you have any other comments and suggestions? Pay attention to these responses. Callers frequently suggest very useful ideas.

• Write open-ended questions for your application 17

Lab Assignment • • Write a text plan Write the usability test design Conduct the usability test Record test data and summarize each of the performance and preference metrics • Place all of your documentation into your lab log and send URL to instructor