Utilizing Multiple Process Tracing Methods in DecisionMaking Research

Utilizing Multiple Process Tracing Methods in Decision-Making Research: Benefits to Cognitive Modeling and Practical Considerations Dr. Mary E. Frame Dr. Joseph G. Johnson 12 November 2020

Process-tracing methods Applications to decision making Verbal protocols » Information acquisition patterns » Response path tracking » Physiological measures » Most common use of such data involves predicting or interpreting meaningful differences in process variables across experimental conditions

Process-tracing methods • Eye Tracking – visual attention • Mouse Tracking – competitive pull, preference vacillation/reversals • Physiological metrics - (respiration rate, heart rate, etc. ) • EEG – ongoing neuronal activity as it influences attention, perceptual processing, motor preparation, etc. • f. MRI/f. NIRS • Electrodermal activity - stress

Key Benefits of Process Tracing Methods • Provide information beyond what can be understood from RT and outcome measures alone • Why and how a behavior occurred rather than a description of behavior • Modeling the process allows us to make more prescriptive changes or perturb patterns of behavior • More useful to applied domains as well as basic research • Joint modeling of PT methods can be used to bolster CCMs (bidirectional benefit) • Hierarchical modeling • Understanding the individual subject more thoroughly • Understanding cognition in the aggregate

Why Measure Multiple Simultaneous Methods? • Certain models in some fields require multiple methods for inference • Workload – cardiac metrics, eye tracking, EEG (Christensen et al. , 2012) • Ability to conduct joint modeling • Methods may overlap temporally or provide convergent evidence • Better able to model the full cognitive process in complex, higherorder cognition (Johnson & Frame, 2019) (e. g. decision making, Koop & Johnson, 2013) • Eye tracking revealing where attention is allocated, measuring mouse movement to determine how this correlates with preferences indicated in motoric activations • Possible to model all methods on a common timeline • Real-time feedback and visualizations (Blaha, et al. , 2013)

Example with Decision-Making Task Option A Option B $70 $30 30% 50% Start Mouse Location

Richer Data on a Common Timeline Chose Option A RT = 1500 Response 1500 ms Stimulus 0 ms Fixation duration for A and B Outcome A Prob. A Number of Transitions/Recursive Prob. B Outcome A Blink Rate/Workload Correlates Prob. A Outcome A Stimulus 0 ms Response 1500 ms Competitive Pull Toward B Outcome A Stimulus 0 ms Middle Prob. A Vacillation/Uncertainty Toward B Prob. B Outcome B Toward A Outcome A Velocity/Acceleration Toward B Prob. A Toward A Outcome A Response 1500 ms

Benefit: More Cohesive Understanding of Unfolding Cognition at the Trial Level • As eye is focusing on information, movement indicates preference based on relative dimension • Potentially greater joint benefit than collecting both in isolation • Eye tracking – which information is being attended to • Mouse tracking – unfolding preference over time • Both – Yolk attention with movement • Does movement correlate with attending to dominant features of each option?

Benefits to Setup and Analysis • Increasingly, hardware/software developers are creating process tracing technologies that work together seamlessly • Eprime experimental software collects mouse tracking data and works with numerous commercial eye trackers, f. MRI, EEG, f. NIRS • Psychopy can be programmed to collect data from a variety of process tracing measures • Open. Sesame & lab. js can be used to collect mouse tracking data and used with inmonitor or glasses eye tracking • Troubleshooting can be handled by the manufacturer remotely or in-person • There are many guides online that walk through common problems

Benefit of Multiple Methods: Visualizations • Simple Fitts’ Law finger dragging task between radially placed dots • Varied dot size (4) and path length (4) • Eye and hand movements captured • Aggregate Measures • Heat map (eye fixations) • Raw Data Playback • Eye trace (with field of view) • Finger trace

Benefit of Multiple Methods: Visualizations

Benefit of Multiple Methods: Joint Modeling • Findings from one PT method can inform another collected in different studies • LRP & mouse tracking (Frame, et al. , 2018) • Stronger connection if both are collected simultaneously • Drift Diffusion Model and other sequential sampling models can be bolstered by various types of process tracing methods, including EEG (Van Vugt, et al. , 2015; Van Vugt, et al. , 2017, Frame, 2019)

Computational cognitive models (CCMs) Applications to preferential choice Preference value Specify cognitive processes assumed to underlie decision outcomes, rather than simply predicting such outcomes using algebraic (utility) models or linear regression. time Simple heuristics (e. g. Thorngate, 1980; Gigerenzer, Todd, & ABC Research Group, 1999) » Production rule systems (e. g. Payne, Bettman, & Johnson, 1988) » Evidence accumulation models (e. g. Busemeyer & Townsend, 1993; Diederich, 1997) » Neural network models (e. g. Usher & Mc. Clelland, 2004; Glöckner & Betsch, 2008) »

Evaluating CCMs Common approach Finding parameters that best fit model to choice probabilities and reaction time distributions » Identifying qualitative data patterns that may (not) be produced by CCMs with specific assumptions » These are the outputs of the experimental task, although CCMs make specific claims about the nature of the process » A more direct and compelling test involves measuring aspects of the process itself, and evaluating whether these are in line with the processes proposed by CCMs »

Evaluating CCMs using process data The best of both worlds Comparing trends across conditions to qualitative predictions generated from theoretical CCM descriptions (e. g. Fiedler & Glöckner, 2012) » Fitting model parameters to behavioral data, and examining how these fit values relate to process data (e. g. van Vugt, et al, 2012) » Using process data to determine/constrain model properties or parameters (e. g. Glöckner, Heinan, Johnson, & Raab, 2012; Krajbich & Rangel, 2011) » Process data as model inputs, which in turn produce predictions further tested with process data (e. g. Koop & Johnson, 2013) »

Incorporating process-tracing data Multiple measures to define inputs and test outputs Eye fixation determines momentary attention in accumulation model » Relative advantage produces associated change in preference » Predicted preference (path) compared to empirical response tracking data » θ Mean (r 2) = 0. 62 Start θ Koop & Johnson (2013)

Advantages of Multiple Process-Tracing Measures: Summary » » » Provide additional dependent variables for testing model predictions Allows better discrimination among various process proposals that offer similar predictions for (or equally good fits to) outcome data Can reduce/eliminate free parameters to constrain models; can be applied to many model types to identify best candidate explanations “Hard wires” processes rather than relying on ad hoc (albeit defensible) assumptions Verifying model assumptions by focusing on very specific aspects of data (e. g. immediately prior to making choice)

Why Not Collect Multiple PT Methods? • More potential sources of error • More things can go wrong during data collection – equipment or user errors • May need specialized setup training • Particularly with EEG • Training RAs is more intense – 1 to 2 semesters of coursework • More difficult analyses • Goes beyond Research Methods 101 • Time series analysis methods • More sophisticated CCMs • Choosing appropriate devices can be overwhelming • Experiments determine which hardware is most appropriate

Considerations for Buying a Device Where are you conducting your study? In my lab! Desk mounted display (in-monitor or external) In the real world! Glasses or portable cameras Do you want to study smooth pursuit? Yes! Get an eye tracker with at least 500 Hz resolution! Nope. Get whatever resolution Eye Tracker you want!

Choosing Devices Can be Daunting • Mouse Tracking: • Can collect movements on multiple devices (mouse, Wiimote, trackpad, etc. ) • Incorporated as part of experimental software vs collected separately • Do you want to collect keylogging as well, timestamps, screen capture, visualizations? • EEG: • EGI System with 32, 64, 128, or 256 channels: $125, 000 -250, 000 (115. 000 -229. 000 €) • Emotiv Epoc+ with 5, 14, or 32 channels: $300 -1, 500 (275 -1. 375 €) • Bio. Semi, Neuroscan, ANT Neuro, etc. : $25, 000 -80, 000 (23. 000 -73. 000 €) • Eye Tracking: • Mobile Eye Trackers (glasses) • Stationary Eye Trackers • In-monitor – Expensive, higher resolution, useful for smooth pursuit • External – Bar style tend to have low resolution/high portability; External cameras have excellent resolution and can be put in more applied environments

Drawbacks & Difficulties: Setup & Analysis • Delay between experiment and device “start time” • May need to operate from separate computers • Lag from input devices (mouse, keyboard) may have variable impact, depending on device • May not matter much for eye tracking, but highly impactful for mouse tracking & EEG • Matters most when you are collecting multiple devices with different lags • Setting up a Network Time Protocol (NTP) solves this problem • Expect partial overlaps in time • Takes approximately 50 ms for visual stimuli to reach visual cortex (and thus get a cortical response) so there may be an offset between eye tracking and perceptual EEG processes

Drawbacks & Difficulties: Setup & Analysis • Physical interference between devices • Wearable eye trackers, ambulatory EEG, body sensors • Compounded if an individual is wearing glasses or conducting a physical activity • Signal interference between devices • Stress and discomfort to participants • May need to add breaks to protocol

Drawbacks & Difficulties: Data Cleaning • Each software/hardware has separate outputs – aggregating issue • Time Alignment – each collected at a different rate/resolution • Eye Tracking: 60 Hz – 1000 Hz • Mouse Tracking: 60 Hz – 100 Hz • EEG: 250 Hz – 1000 Hz • May need to down-sample or interpolate to model data together Time (ms) EEG (F 3) EEG (F 4) 1 . 0284 -. 0242 2 . 0838 -. 0348 3 . 3482 -. 1223 4 . 7395 -. 2385 5 . 5961 . 0241 6 . 1948 -. 2845 7 . 2301 -. 2312 8 . 2387 . 0184 9 . 2034 . 0313 10 . 5703 . 1284 11 . 4932 . 1204 12 . 4878 -. 0185 13 . 4878 -. 1204 14 . 9786 -. 1938 15 . 9411 16 1. 4964 17 1. 204 18 1. 2483 -1. 3854 19 1. 111 -. 8531 20 1. 1284 . 0483 21 2. 2392 . 0421 22 3. 3953 . 0184 23 3. 2020 . 1253 24 2. 3912 1. 2941 25 2. 3941 . 3041 26 2. 3842 . 2048 27 2. 3491 . 2421 28 2. 6357 . 9853 29 2. 3293 . 2532 30 2. 4677 . 8786 Eye X Eye Y 600 200 600 210 615 210 638 235 638 236 EEG – 1000 Hz -. 3021 Eye tracking – 500 Hz -. 4967 640 236 -. 2543 Mouse Tracking – 60 Hz 712 222 712 222 Mouse X Mouse Y 945 1100

Difficulty of Data Interpretation • Easy to analyze each process tracing metric on their own – should have hypotheses for each one anyway • When modeling together, may need to create new metrics for measuring joint effects • How to interpret contradictions in the data • Eyes concentrating on weakest factor on least preferred option, movement toward preferred option • Helps to not get lost in the individual participant / trial level • Many of these oddities disappear in the aggregate

Drawbacks & Difficulties: Time Series Analysis • Full exploitation of data requires time series analysis • Really this is a benefit and a drawback depending on expertise – allows for much richer analyses • Certain types of data (EEG) have oscillatory patterns over time that can be themselves modeled at different levels

Conclusions • Recommended to collect multiple PT methods, different studies or concurrently • Benefits are fantastic and advance fundamental research • Drawbacks can be mitigated – need to be mindful, not afraid

Questions?