Interpretive Evaluation Agenda Simple user modeling Interpretive Evaluation

  • Slides: 34
Download presentation
Interpretive Evaluation Agenda • Simple user modeling • Interpretive Evaluation - Motivation - Methods

Interpretive Evaluation Agenda • Simple user modeling • Interpretive Evaluation - Motivation - Methods Ethnography • Fitt’s Law

Simpler User Modeling • How do attributes of users influence the design of user

Simpler User Modeling • How do attributes of users influence the design of user interfaces? • Are there some design guidelines that we can derive from different attributes? Fall 2002 CS/PSY 6750 2

User Profiles • Attributes: - attitude, motivation, reading level, typing skill, education, system experience,

User Profiles • Attributes: - attitude, motivation, reading level, typing skill, education, system experience, task experience, computer literacy, frequency of use, training, color-blindness, handedness, gender, … • Novice, intermediate, expert Fall 2002 CS/PSY 6750 3

Motivation • User • Design goal - Low motivation, discretionary use - Low motivation,

Motivation • User • Design goal - Low motivation, discretionary use - Low motivation, mandatory - High motivation, due to fear - High motivation, due to interest Fall 2002 CS/PSY 6750 - Ease of learning - Control, power - Ease of learning, robustness, control - Power, ease of use 4

Knowledge & Experience • task system - low - high - low high -

Knowledge & Experience • task system - low - high - low high - high Fall 2002 • Design goals - Many syntactic and semantic prompts - Efficient commands, concise syntax - Semantic help facilities - Lots of syntactic prompting low CS/PSY 6750 5

Job & Task Implications • Frequency of use - High - Ease of use

Job & Task Implications • Frequency of use - High - Ease of use - Low - Ease of learning & remembering • Task implications - High - Ease of use - Low - Ease of learning • System use - Mandatory - Ease of using - Discretionary - Ease of learning Fall 2002 CS/PSY 6750 6

Evaluation • Predictive - Discount Usability (Mon) - Cognitive Modeling-GOMS family (Wed) - Cognitive

Evaluation • Predictive - Discount Usability (Mon) - Cognitive Modeling-GOMS family (Wed) - Cognitive Modeling-contextual (next Mon) • Interpretive (today) • User testing (later) Fall 2002 CS/PSY 6750 7

Approaches • Experimental (Lab studies, quantitative) - Typically in a closed, lab setting Manipulate

Approaches • Experimental (Lab studies, quantitative) - Typically in a closed, lab setting Manipulate independent variables to see effect on dependent variables • Naturalistic (Field studies, qualitative) - Observation occurs in “real life” setting Watch process over time “Ecologically valid” Fall 2002 CS/PSY 6750 8

Interpretive Evaluation • Experimental: Formal and objective • Interpretive: More subjective - Concerned with

Interpretive Evaluation • Experimental: Formal and objective • Interpretive: More subjective - Concerned with humans, so no objective reality - Sociological, anthropological approach • Users involved, as opposed to predictive approaches Fall 2002 CS/PSY 6750 9

Beliefs • Sees limitations in scientific hypothesis testing in closed environment - Lab is

Beliefs • Sees limitations in scientific hypothesis testing in closed environment - Lab is not real world - Can’t control all variables - Context is neglected - Artificial, short tasks Fall 2002 CS/PSY 6750 10

IE Methods • A number of different methods or techniques in this area exist

IE Methods • A number of different methods or techniques in this area exist - Ethnography Our main focus - Contextual inquiry More specific form of ethnography with a focus on asking questions - Field study Common notion, often equated with ethnography - Observational study Much more informal, just watching users -… Fall 2002 CS/PSY 6750 11

Ethnography • Deeply contextual study • Immerse oneself in situation you want to learn

Ethnography • Deeply contextual study • Immerse oneself in situation you want to learn about (anthropological and sociological roots) - Observing people in their cultural context • Interpretation of data is primary • Behavior is meaningful only in context Fall 2002 CS/PSY 6750 12

Philosophy • Argues that formal environment of controlled study is artificial --Experimenter wields “power”

Philosophy • Argues that formal environment of controlled study is artificial --Experimenter wields “power” over subject • So … get into working environment of user • -> Interpretation is primary, rather than data Fall 2002 CS/PSY 6750 13

Objectives • Understanding the user - Understand goals and values - Understand individual’s or

Objectives • Understanding the user - Understand goals and values - Understand individual’s or group’s interactions within a culture - Try to make tacit domain knowledge explicit - Do this in an unbiased fashion - For UI designers: Improve system by finding problems in way it is currently being used Fall 2002 CS/PSY 6750 14

Techniques • In person observation • Audio/video recording • Interviews • “Wallow in the

Techniques • In person observation • Audio/video recording • Interviews • “Wallow in the data” Fall 2002 CS/PSY 6750 15

Observation is Key • Carefully observe everything about users and their environment • Think

Observation is Key • Carefully observe everything about users and their environment • Think of describing it to someone who has never seen this activity before • What users say is important, but also nonverbal details Fall 2002 CS/PSY 6750 16

Observations • Things of interest to evaluator - Structure and language used in work

Observations • Things of interest to evaluator - Structure and language used in work Individual and group actions Culture affecting work Explicit and implicit aspects of work • Example: Office work environment - Business practices, rooms, artifacts, work standards, relationships between workers, managers, … Fall 2002 CS/PSY 6750 17

Interviews Important • Have a question plan, but keep interview open to different directions

Interviews Important • Have a question plan, but keep interview open to different directions • Be specific • Create interpretations together with users • At end, query “What should I have asked? ” • Record interviews Fall 2002 CS/PSY 6750 18

Steps • 1. Preparation - Understand organization policies and work culture Familiarize yourself with

Steps • 1. Preparation - Understand organization policies and work culture Familiarize yourself with system and its history Set initial goals and prepare questions Gain access and permission to observe & interview • 2. Field study - Establish rapport with users - Observe/interview users in workplace and collect all different forms of data - Follow any leads that emerge from visits - Record the visits Fall 2002 CS/PSY 6750 Rose et al ‘ 95 19

Steps • 3. Analysis - Compile collected data in numerical, textual and multimedia databases

Steps • 3. Analysis - Compile collected data in numerical, textual and multimedia databases - Quantify data and compile statistics - Reduce and interpret data - Refine goals and process used • 4. Reporting - Consider multiple audiences and goals - Prepare a report and present findings Fall 2002 CS/PSY 6750 20

One Technique • Affinity Diagram • Process - Write down each quote/observation on a

One Technique • Affinity Diagram • Process - Write down each quote/observation on a slip of paper - Put up on board - Coalesce items that have affinity If they are saying similar things about an issue - Give names to different groups (colors too) - Continue grouping subgroups - A hierarchy will be formed Fall 2002 CS/PSY 6750 21

Why Useful? • Can help designer gain a rich and true assessment of user

Why Useful? • Can help designer gain a rich and true assessment of user needs - Help to define requirements • Uncovers true nature of user’s job - Discovers things that are outside of job description or documentation • Allows you to play role of end-user - Can sit in when real users not available • Open-ended and unbiased nature promotes discovery - Empirical study and task analysis are more formal – ethnography may yield more unexpected revelations Fall 2002 CS/PSY 6750 22

Types of Findings • Can be both - Qualitative Observe trends, habits, patterns, …

Types of Findings • Can be both - Qualitative Observe trends, habits, patterns, … - Quantitative How often was something done, what per cent of the time did something occur, how many different … Fall 2002 CS/PSY 6750 23

Drawbacks • Time required - Can take weeks or months • Scale - Most

Drawbacks • Time required - Can take weeks or months • Scale - Most use small numbers of participants just to keep somewhat manageable • Type of results - Highly qualitative, may be difficult to present and use • Acquired skill - Identifying and extracting “interesting” things is challenging Fall 2002 CS/PSY 6750 24

Ethnomethodology • Concurrent/informed ethnography - Study is being done in conjunction with a system

Ethnomethodology • Concurrent/informed ethnography - Study is being done in conjunction with a system being developed - + Helps keep focus on user throughout design - - Requires lots of time and coordination Fall 2002 CS/PSY 6750 25

Cooperative Evaluation • User is viewed as collaborator in evaluation, not a subject -

Cooperative Evaluation • User is viewed as collaborator in evaluation, not a subject - “Friendly approach” • Relaxed version of think-aloud - Evaluator and participant can ask each other questions Fall 2002 CS/PSY 6750 26

CE Methods • Seeks to detect errors early in a prototype • Experimenter uses

CE Methods • Seeks to detect errors early in a prototype • Experimenter uses tasks, also talks to participant throughout, asks questions… • Have debriefing session at end Fall 2002 CS/PSY 6750 27

Low Level Models • Fitt’s Law - Models movement times for selection tasks •

Low Level Models • Fitt’s Law - Models movement times for selection tasks • Basic idea: Movement time for a wellrehearsed selection task - Increases as the distance to the target increases - Decreases as the size of the target increases Fall 2002 CS/PSY 6750 28

Components • ID - Index of difficulty ID = log 2 ( 2 A/W

Components • ID - Index of difficulty ID = log 2 ( 2 A/W ) bits result distance to move width tolerance of target Both quantities are distances so unitless result Fall 2002 CS/PSY 6750 29

Components • MT - Movement time MT = b * ID or MT =

Components • MT - Movement time MT = b * ID or MT = a + b*ID MT is a linear function of ID Fall 2002 CS/PSY 6750 30

Original Experiment • 1 -D A Fall 2002 CS/PSY 6750 W 31

Original Experiment • 1 -D A Fall 2002 CS/PSY 6750 W 31

Exact Equation • Run empirical tests to determine a and b in MT =

Exact Equation • Run empirical tests to determine a and b in MT = a + b*ID • Will get different ones for different input devices and ways the device is used MT Fall 2002 ID CS/PSY 6750 32

Common Equation • MT = a + b log 2 (A/W + 1) •

Common Equation • MT = a + b log 2 (A/W + 1) • Provides useful numbers Fall 2002 CS/PSY 6750 33

Questions • What do you do in 2 D? • Where can this be

Questions • What do you do in 2 D? • Where can this be applied in user interface design? Fall 2002 CS/PSY 6750 34