Developing a Multimodal Affect Assessment for Aviation Training

Developing a Multimodal Affect Assessment for Aviation Training Tianshu Li, Imène Jraidi, Alejandra Ruiz Segura, Leo Holton, Susanne Lajoie Mc. Gill University Tianshu. li@mail. mcgill. ca S

Study Overview S Explore multimodal methodology of affect measurement S Address the lack of theoretical frameworks on affect and its assessment methodology in aviation training S Assess convergence between objective observation and subjective reports by correlational analysis S Preliminary step of a bigger project

Theoretical Background: Cyclic Process of Situation Appraisal and Affect Response • Dimensions: • Arousal • Valence • Control • Value Affect Respons e Appraisal Situation • Multiple Manifestations: • Behavioral • Physiological • Subjective experience • Favorable vs unfavorable • Success vs failure

Affect-related Research in Aviation S Affect is currently assessed by instructors subjectively during training • Arousal • Valence Affect S Instructors are over-tasked during training Response • Judgement and decision making • Situation awareness • Psychomotor function S Subjective questionnaires are time-consuming and disruptive Appraisal Situation S Need for automated concurrent affect assessment • Control • Value • Aviation task performance

Biometrics Assessment of Affect in Aviation S Literature review in Scopus S Lack of research measuring affective states other than stress responses S Lack of studies using behavioral measures S Lack of multimodal protocols Diagram of Comparison of the Number of Peer-reviewed Articles on Biometrics Measures in Aviation Context

Informed Research Design: Multimodal Protocol S Integration of subjective experience and objective observation a) Physiological arousal: Electrodermal activity b) Behavioral: facial expression (basic emotions: universality) Subjective: questionnaires c) S Cross-validation among different manifestations of affect

Research Questions and Hypotheses S RQ 1: relationship between EDA features and self-reported appraisal, work-load, fatigue and effort? Hypothesis: Arousal will correlate positively with workload, fatigue and effort. S RQ 2: relationship between emotions inferred from facial expression analysis and self -reported variables? Hypothesis: Negative, activating emotions will correlate positively with workload, fatigue and effort, negatively with perceived control and value; Opposite trends for positive emotions. S RQ 3: individual differences on how biometric measures (EDA and facial expression) relates to self-reported variables? Hypothesis: Variation among individuals on these relationships between biometric measures of affect and self-reported affect correlates

Experiment Design S Cognitive interview with subjective-matter expert: a) identify experimental environment b) design tasks for selected participants S 14 participants (9 females): graduate and undergraduate students in Mc. Gill University S Multimodal affect assessment Dr. Susanne Lajoie conducting Cognitive Interview with Subject-matter Expert Alain Bourgon from CAE Inc.

Experimental Tasks: X-Plane Maneuver Materials: Aviation simulation software and joystick. Tasks: changing altitude, speed, heading. 5 difficulty levels. Procedure: S Instruction from experimenter: i. e. ‘turn right at 30 -degree bank angle to heading 0, maintain speed and altitude’ S Participant complete the task independently

Experimental Setup Procedure per Task X-Plane Tasks Instruction EDA Facial expression video recording Retrospective questionnaire

Experiment Procedure S Training before experiment S 10 tasks in total S 5 difficulty levels: level 1 is baseline maneuver, minimal difficulty S 2 tasks per difficulty level S Concurrent recording of EDA and facial expression, questionnaires after paired tasks of each difficulty level

Multimodal Affect Measurement: Physiological arousal S Physiological activation: measured by electrodermal activity, Bio. Nomadix EDA module (1000 Hz). S SCR features extraction with Makowski’s algorithm implemented in Neurokit. S SCR features: phasic EDA, SCR peak count, SCR peak amplitude.

Multimodal Affect Measurement: Facial expression of emotions S Face. Reader 6. 0 S Microsoft Lifecam HD 5000 (30 Hz) S Intensity of basic emotions: happy, surprised, neutral, sad, angry, scared and disgusted

Multimodal Affect Measurement: Self-report Questionnaires ‘Grounded truth’ to assess data convergence among other measurements. Questionnaires administrated on a laptop during breaks between tasks: S Demographics (once before experiment). S Retrospective subjective experience. S Motivation and appraisal: perceived control and perceived value scales (usefulness, importance and interest). S Workload assessed by NASA Taskload Index (mental workload, physical workload, effort, fatigue).

Results: Physiological Inference RQ 1: Is there a relationship between EDA features and self-reported appraisal, work-load, fatigue and effort? S Significant positive correlations are found between SCR peak count and self- report variables, namely physical workload (r =. 355, p <. 01), effort (r =. 220, p <. 05) and fatigue (r =. 222, p <. 05). S At a general level (i. e. across all participants), peak frequency of physiological arousal increased as participants’ experienced physical workload, effort and fatigue increased.

Interpretation: Physiological Inference S Physiological arousal indicated by EDA correlates positively with workload moderately, with fatigue and effort weakly. S Prolonged activation and stress will lead to physical workload and fatigue. Elevated stress and activation associate with higher level of effort. S Our results suggest that the arousal dimension of affect in aviation training could be accounted for by EDA as a physiological measure.

Results: Physiological Inference Cont’d RQ 3: Are there any individual differences on how biometric measures (EDA) relate to self-reported variables? S Bivariate correlations between SCR peak count, fatigue, effort and physical workload for all valid tasks for each individual (N = 14) S Wide ranges of correlation coefficients on the individual level indicate variation of physiological manifestation of affect among participants S The individual variation may be due to the various motivational and behavioral outcomes of physiological arousal: Participants may be more motivated and driven to work hard under elevated stress, but they may lose hope under prolonged stress and disengage from the task. Bicorrelations between SCR peak count and subjective effort, fatigue and physical load.

Results: Behavioral Inference S Anger is positively correlated with mental workload, physical workload, effort and fatigue, value and control. S Surprise: negatively correlated with fatigue S Scared: negative correlated with control Bivariate Correlations between the Intensities of Emotions in Facial Expression Analysis and Self-report Variables Measures Mental Workload Physical Workload Effort Fatigue Perceived Value Perceived Control Angry . 267* . 268* . 316** . 369** . 222* . 283* Surprised -. 022 -. 095 . 043 -. 242* . 139 . 036 Scared . 071 -. 059 . 076 . 126 -. 064 -. 218* *p<. 05, **p<. 01

Interpretation: Behavioral Inference S Consistent with the second hypothesis: anger as a negative activating emotion, could be the result and the cause of high workload, effort and fatigue S Positive correlation between anger and control, value: The face of intense focus is similar to facial expressions of anger. Previous research shows positive correlation between focus and workload. Focus could indicate higher value and lead to better control over the situation.

Interpretation: Behavioral Inference Cont’d S Surprise: neutral emotion, unlikely to correlate with control and value; context dependent In our context: surprise could be a reaction of accomplishing a task faster or better than expected, correlating with less fatigue S The convergence between behavioral inference of affect (facial expression) and ‘grounded truth’ supports the validity of facial expression analysis as an affect measure.

Results: Behavioral Inference Cont’d RQ 3: Are there any individual differences on how biometric measures (facial expression) relates to self-reported variables? S Different patterns of the relationships between facial expression analysis features (angry, surprised, scared) and self-report variables are found across participants. Bivariate Correlations between the Facial Expression Variables and Selfreport Variables.

Conclusion S Our findings support the validity of EDA and facial expression analysis as measurements of affect in aviation training context. S The protocol developed could be used for future aviation training. S A comprehensive assessment of affect could help aviation training instructors to allocate appropriate scaffolding to pilots where needed, thereby improving the overall training experience and effectiveness.

Next steps The current study is a pilot study for a larger project: In. Look Project. Next steps include: S Replicate the current experiment with pilot trainees, high fidelity simulation and larger sample size. S Assess potential mediators in the relationship between affect and workload S Integrate other potential measurements of affect and workload (EEG, ECG, eye-tracking) through collaborating with other project partners

Thank you S Project In. Look: Biometric approaches to inferring pilot trainee's affective and cognitive states From CAE Inc. : David Bowness, Alain Bourgon, Hugh Grenier, Dr. Maher Chaouachi, Ilia Gourevitch, Aurelian Constantinescu, Jack Russ From National Research Council: Sion Jennings, Andrew Law From Concordia University: Dr. Yong Zeng, Mengting Zhao, Wen Jun Jia, Vlad House From University of Montreal: Dr. Jocelyn Faubert, Yannick Roy, Khashayar Misaghian From Marinvent Corporation: Marie-Helene La. Rose Glob. Vision Inc.

References Gross, J. J. , Emotion regulation: Affective, cognitive, and social consequences. Psychophysiology, 2002. 39(3): p. 281 -291. Pekrun, R. and L. Linnenbrink-Garcia, Introduction to emotions in education, in International handbook of emotions in education. 2014, Routledge. p. 11 -20. Kaempf, G. L. and G. Klein, 11 Aeronautical Decision Making: The next generation. Aviation psychology in practice, 2017: p. 223. Jensen, R. S. , Pilot judgment and crew resource management. 2017: Routledge. Causse, M. , et al. , The effects of emotion on pilot decision-making: A neuroergonomic approach to aviation safety. Transp. Res. Part C Emerg. Technol. , 2013. 33: p. 272 -281. Harley, J. M. , Measuring emotions: a survey of cutting edge methodologies used in computer-based learning environment research, in Emotions, technology, design, and learning. 2016, Elsevier. p. 89 -114. Regula, M. , et al. Study of heart rate as the main stress indicator in aircraft pilots. in Proc. ME 2014. IEEE. Pekrun, R. and L. Linnenbrink-Garcia, International Handbook of Emotions in Education. 2014, London, UNITED KINGDOM: Routledge. Pekrun, R. and R. P. Perry, Control-value theory of achievement emotions, in International handbook of emotions in education. 2014, Routledge. p. 130 -151. Murray, P. S. and W. L. Martin. Beyond situational awareness: A skill set analysis for situational control. in AAv. PA Symposium, Sydney, Australia. 2012. Flin, R. , et al. , Human factors in the development of complications of airway management: preliminary evaluation of an interview tool. Anaesthesia, 2013. 68(8): p. 817 -825. Hart, S. G. NASA-task load index (NASA-TLX); 20 years later. in Proc Hum Factors Ergon Soc Annu Meet. 2006. Sage Publications Sage CA: Los Angeles, CA. Burnham, J. F. , Scopus database: a review. Biomed. Digit. Libr. , 2006. 3(1): p. 1. Perry, R. P. , et al. , Perceived academic control and failure in college students: A three-year study of scholastic attainment. Res High Educ, 2005. 46(5): p. 535 -569. Wigfield, A. and J. S. Eccles, Expectancy-Value Theory of Achievement Motivation. Contemp. Educ. Psychol. , 2000. 25(1): p. 68 -81. Nittala, S. K. , et al. Pilot skill level and workload prediction for sliding-scale autonomy. in 2018 17 th IEEE (ICMLA). 2018. IEEE. Shiomi, K. Itano, and A. Suzuki. Development and evaluation of the fatigue and drowsiness predictor. in Archives of 28 th ICAS. 2012. D’Mello, S. and A. Graesser, Dynamics of affective states during complex learning. Learn Instr, 2012. 22(2): p. 145 -157. Duffy, E. , Activation and behavior. 1962. 2020, X. -P. X-Plane 11. 2020; Available from: https: //www. x-plane. com/. Storbeck, J. and G. L. Clore, Affective arousal as information: How affective arousal influences judgments, learning, and memory. SOC PERSONAL PSYCHOL COMPASS, 2008. 2(5): p. 1824 -1843. Braithwaite, J. J. , et al. , A guide for analysing electrodermal activity (EDA) & skin conductance responses (SCRs) for psychological experiments. Psychophysiology, 2013. 49(1): p. 1017 -1034. Dettmers, S. , et al. , Students’ emotions during homework in mathematics: Testing a theoretical model of antecedents and achievement outcomes. Contemporary Educational Psychology, 2011. 36(1): p. 25 -35. Makowski, D. , Neuro. Kit. 2016. D’Mello, S. , et al. , Confusion can be beneficial for learning. Learn Instr, 2014. 29: p. 153 -170. Den Uyl, M. and H. Van Kuilenburg. The Face. Reader: Online facial expression recognition. in Proceedings of measuring behavior. 2005. Citeseer. Mauss, I. B. and M. D. Robinson, Measures of emotion: A review. Cogn. Emot. , 2009. 23(2): p. 209 -237. Muis, K. R. , et al. , The role of epistemic emotions in mathematics problem solving. Contemp. Educ. Psychol. , 2015. 42: p. 172185.