Research Methods in HumanComputer Interaction Second Edition Chapter

Research Methods in Human-Computer Interaction Second Edition Chapter 13: Measuring the human Lazar, Feng, and Hochheiser Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Participant as data source • • Physical and emotional measurements can give significant into how we use computers Possibly data that would be hard to get in other ways Many approaches Range of cost, complexity, and invasiveness Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Examples • • Mouse/keyboard Eye-tracking Measurements of physical and emotional responses Galvanic skin response, blood volume, heart rate measurement EKG/EMG Brain scan? FMRI Cost-benefit tradeoffs Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Eye-Tracking/Eye-gaze • • • Where does the user look? Where does she expect to find data? When is she lost/confused? Cameras or other sensors track position or orientation of eyes or other parts of body Transform raw data into detailed descriptions of “paths” of visual focus Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Eye-tracking challenges Eyes constantly in motion • • • Rapid eye motions – saccades – help us focus Larger motions indicate change of focus “Dwell” - relatively little motion indicating focus on a target Thresholds for identifying a “dwell” Interpretation Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Challenges Expense of equipment • • • Hard to interpret • • Although commercial products now available for < $200 Low-cost alternatives based on USB webcams may be possible Where you are looking is useful But it must be coordinated with what you are looking at Overlay trail on screen shot indicating path of user gaze Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Web Page with eye-tracking trails Card, et al, 2001 Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Uses for eye-tracking Pointing and selecting Assistive technology Study of design implications. . • • • Placement of target in list of link Length of text summaries for search results Eye movement during menu selection • • Fixation vs. movements in specific directions - “sweeps” Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Types of eye-tracking studies Hypothesis-driven experiments • • • Well-defined tasks, measures, etc Do users look at security indicators on web browsers? (Whalen & Inkpen, 2005) Exploratory, open-ended research • • “Let's see what they look at”? Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Physiological tools Bodies change behavior with stimuli Measurable differences when we are • • • Excited Frustrated Aroused Measurements from bodies can be used to understand these response Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Psychophysiology Use physiological signals to measure stress and emotional responses Assess responses • • • As they happen, rather than later (via surveys) Replace recall with measurement Surveyed with quantitative measurement Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Physiological data types Current flow • • • Every cell is an electrical system Blood flow Heart Rate Respiration Rate Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Sensors Electrodes • • • Direct measurements of electrical signals Galvanic skin response Transducers • • • Convert mechanical or physical measurements to electrical form Pressure sensors to measure posture Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Challenges No single notion of arousal Stimulus-Response specificity • • Certain stimuli may lead to increases in some measures alongside decreases in others Need to understand relationships between stimuli used in your tasks and potential responses Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Types of measurements • • Electrodermal Activity/Galvanic Skin Response – fingers/toes Blood volume pressure – finger Electrocardiography (EKG) – chest/abdomen Respiration – Thorax Muscular/Skeletal Positioning – Varied Electromyography (EMG) – Jaw/Face Electroencephalography (EKG)/Evoked responses - head Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Electrodermal Activity/ Galvanic Skin Response (GSR) • • Flow of electricity through the skin Pair of electrodes on fingers Conductance levels linked to Fear Sadness Arousal Cognitive activity Frustration Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Galvanic Skin Response (GSR) http: //www. thoughttechnology. com/sciencedivision/pages/products/skinconduct. html Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Cardiovascular Stimuli may cause complex changes in heart behavior • Variability in heart rate, blood pressure, and bloodvolume pressure • • • Fit. Bits, smart watches, etc – See Chapter 14 Measure stress, mental effort, fear, happiness, and anger Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Blood volume pressure (BVP) • • Light-absorption properties of capillaries in finger Changes in pressure lead to changes in reflected light Correlate with stimuli that provoke anxiety Also infer changes in heart rate Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Electrocardiography (EKG) Electrical current causes heart to beat Measure • • • Heart rate Interval between beats Heart-rate variability Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Respiration • • Arousal -> faster breathing Some emotions can cause irregular breathing • Sensors attached to thorax • • Or integrated into clothing Expansion and contraction of chest Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Measuring workload • How “hard” is a task? • Hard to measure • Surveys rely on post-task perception • NASA Task Load Index • Physiological measurements? • Eye-gaze -> pupil diameter increases with stress/frustration • Heart flux and variable of EKG • Brain imaging (f. NIRS) Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Muscular and Skeletal Position Sensing • • • How do we sit? How do we move? Measure movement and positioning Fiber optics Accelerometers Computer vision Foam sensors in clothing Pressure Sensors Nintendo Wii. . Microsoft Kinect – cameras Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Head tracking for large displays and virtual environments • Ball, North, & Bowman, 2007 Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Electromyography (EMG) Jaw measurements • • Eyebrows or cheeks • • Muscle tensions associated with clenching Frowns or smiles Mild positive emotions lead to lower readings over the eyebrow and higher over cheek Also sadness, fear, happiness Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Brain activity Electroencephalography (EEG) • • Helmet with electrodes on scalp Functional MRI (f. MRI) • • identify regions ctivated in response to stimuli Functional near-infrared spectroscopy (f. NIRS) – measure reflective characteristics • • Measure cognitive load in tasks? Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Data collection challenges Electrodes and sensors can be difficult to use correctly • • • Get proper training Work with an experienced professional Potential discomfort and unease • • • Electrodes should only be attached by someone of the same gender? Be explicit in informed consent (Chapter 15) Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Other challenges Electromyography needles placed in skin • • • Safe, but “Don't try this at home” Use electrodes placed on skin instead Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Analysis challenges • Noisy signals • Distortion • Interference • Comparison to “baseline” • Potentially hard to establish • Influences magnitude of response • Small increase in heart beat if heart is already beating quickly Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

More analysis challenges Habituation – response to stimulus decreases after repeated presentation Signal processing techniques to filter and “clean” data • • • Needs expertise Data granularity • • • Averages are OK for overall impressions Specific responses require synchronizing measurement stream with user actions Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Still more analysis challenges • • • Modeling to classify types of responses? Machine learning? What emotional state is associated with a response? – Happiness? Sadness? Disgust? Fear? • • Data mapping from measured signals to emotional states is inconclusive Mixed or incomplete signals? Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Synchronization challenges • • • Multiple data streams Actions on the computer (log files) Timing of physiological measurements – Often collected on a second computer. . • Modified mouse with signals to two computers? Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Example setup (Scheirer, et al. , 2002) • • • Two computers Stimulus presentation Recording physiological measurements Mouse with two outputs Additional display for clock Video camera Sensors Blood Volume Pressure Galvanic Skin Response sensors Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Example setup (Scheirer, et al. , 2002) Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Design questions • Costs vs. benefits of physiological measurement? • When do easier methods provide data that is almost as good, at a lower cost? • • Post-task questionnaires Observations Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Example: Multi-modal Input (Shi, et al. 2007) • • • Galvanic Skin Response (GSR) Test emotional response to speech, gesture, and multimodal interfaces Response levels lowest for multimodal, then speech, gestures Total response increases with task complexity GSR peaks correlated with stressful or frustrating events Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Example: Video Games (Mandryk & Inkpen, 2004) GSR, EKG, cardiovascular rate, respiration, facial EMG Responses to computer games played • • Against computer Against a friend Playing against a friend was more exciting • • • Most had higher GSR and facial EMG No differences on cardiovascular and respiratory measures Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Example: LAB-IN-A-BOX (Weibel, et al. 2014) • Detailed measurement of system use in context • Not lab • Electronic medical record (EMR) use during clinic visits • Understand use of computer and impact on physicianpatient interaction • Audio, video, eye-tracking, keystroke logging, screencapture, mouse events, Kinect • Analyze synchronized data streams with Chrono. Viz (Fouse, et al. 2011) Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License

Example: LAB-IN-A-BOX (Weibel, et al. 2014) Components Analysis with Chrono. Viz (Fouse, et al. 2011) Slides © 2017 Lazar, Feng, and Hochheiser, Creative Commons, Attribution-Non. Commercial-Share. Alike 4. 0 License