U S ARMY COMBAT CAPABILITIES DEVELOPMENT COMMAND SOLDIER

U. S. ARMY COMBAT CAPABILITIES DEVELOPMENT COMMAND – SOLDIER CENTER Intelligent Exercise Control: Integrating GIFT and Battle Space Visualization Benjamin Goldberg, Ph. D. and Michael Hoffman Senior Scientist/GIFTsym Chair CCDC-SC, Simulation and Training Technology Center DISTRIBUTION A: APPROVED FOR PUBLIC RELEASE 16 May 2019 – GIFTSym 7 – Day 1 12 MAR 2019 UNCLASSIFIED

UNCLASSIFIED 2

AGENDA • Training Management Technology Overview – Battle Space Visualization via the Augmented Reality Sandtable (ARES) – GIFT Learning Effect Chain • Game. Master Concept for Intelligent Exercise Control – Training Facilitators and Observer Controllers (OC) – Adaptive Instructional Supports via GIFT to Support/Supplement OC Workload • GIFT Modifications – Human on the Learning-Effect-Chain Loop – Game Master Interface • Link to Persistent Modeling and Recommendation Engines – Integration with the Total Learning Architecture UNCLASSIFIED 3

Battle Space Visualization via ARES UNCLASSIFIED 4

UNCLASSIFIED//FOR OFFICIAL USE ONLY//DRAFT//PRE-DECISIONAL ARES ARCHITECTURE OVERVIEW Sample Interoperability 1 Data Inputs • Terrain ‒ Imagery (Google Maps, JPEG, TIFF, etc. ) ‒ Topographic (CADRG, Geo. PDF, etc. ) ‒ Elevation (DTED, DEM, Geo. TIFF, etc. ) • Services ‒ Intelligent Tutors (GIFT, etc. ) ‒ Models (Chem-Bio Plume, Route, Weapon Range, Cyber, etc. ) Head Mounted Projected • Augmented Reality • Virtual Reality • Mixed Reality • Pico • Table • Floor ‒ Live Unit Location (JCB-P, etc. ) ‒ Training Simulator (VBS, One. SAF, etc. ) 1 Distributed Interactive Simulation (DIS) v 7 compliant ARE S Personal Computer Exports ‒ Terrain Files (Geo. TIFF, OTF, Unity, etc. ) • Mobile • Desktop • Laptop ‒ Scenario Data 2 (Tactical Graphics & Symbols) 2 Military Scenario Definition Language (MSDL) compliant Sample Interaction Methods: Tangible User Interface, Game Controller, Tokens, Gesture, Voice, Mouse & Keyboard, Stylus UNCLASSIFIED//FOR OFFICIAL USE ONLY//DRAFT//PRE-DECISIONAL 5

UNCLASSIFIED//FOR OFFICIAL USE ONLY//DRAFT//PRE-DECISIONAL 6

UNCLASSIFIED//FOR OFFICIAL USE ONLY//DRAFT//PRE-DECISIONAL MULTIPLE MODALITIES Left: Floor projection @ Ft Benning Bottom Left: Sand table @ Ft Leonard Wood Bottom Right: Mixed reality @ West Point Supports distributed collaboration amongst all modes UNCLASSIFIED//FOR OFFICIAL USE ONLY//DRAFT//PRE-DECISIONAL 7

UNCLASSIFIED//FOR OFFICIAL USE ONLY//DRAFT//PRE-DECISIONAL Adaptive Training and Intelligent Tutoring via GIFT UNCLASSIFIED//FOR OFFICIAL USE ONLY//DRAFT//PRE-DECISIONAL 8

UNCLASSIFIED//FOR OFFICIAL USE ONLY//DRAFT//PRE-DECISIONAL (Sottilare et al. , 2013) UNCLASSIFIED//FOR OFFICIAL USE ONLY//DRAFT//PRE-DECISIONAL 9

UNCLASSIFIED//FOR OFFICIAL USE ONLY//DRAFT//PRE-DECISIONAL ADAPTIVE LEARNING EFFECT CHAIN… FOR TEAMS (Sottilare et al. , 2018) UNCLASSIFIED//FOR OFFICIAL USE ONLY//DRAFT//PRE-DECISIONAL 10

UNCLASSIFIED//FOR OFFICIAL USE ONLY//DRAFT//PRE-DECISIONAL Game Master Concept for Intelligent Exercise Control UNCLASSIFIED//FOR OFFICIAL USE ONLY//DRAFT//PRE-DECISIONAL 11

UNCLASSIFIED//FOR OFFICIAL USE ONLY//DRAFT//PRE-DECISIONAL 12

UNCLASSIFIED//FOR OFFICIAL USE ONLY//DRAFT//PRE-DECISIONAL 13

UNCLASSIFIED//FOR OFFICIAL USE ONLY//DRAFT//PRE-DECISIONAL OBSERVER CONTROLLER/TRAINER • Subject Matter Experts on doctrine and in their specific warfighting function – Mission Command – Movement and Maneuver – Fires – Sustainment – Protection – Intelligence • Play a critical role in providing feedback to the unit, informally, through everyday interactions and, formally, through mid and final after action reviews (AARs) – These events give the training audience actions to consider for sustainment and improvement UNCLASSIFIED//FOR OFFICIAL USE ONLY//DRAFT//PRE-DECISIONAL 14

UNCLASSIFIED//FOR OFFICIAL USE ONLY//DRAFT//PRE-DECISIONAL Adaptive Learning Engine via GIFT EKG, GSR, Breathing, Eye Tracking etc. (Real-time Assessment ) DIS/HLA/GPS/COMMS (Assessment) Workload, Affect, Cognitive State Gateway Module DIS/HLA/GPS/COMMS (Visualization) Human-Centered Measures (Spotlite? ) Stealth Assessments (TTPs) Team States (Task Work/Team Work) Strategy Options Battlefield Visualization UI Human Observer Controller (Interop Configurations) GIFT Widget (Assessment) GIFT Widget (Strategy Handler) Strategy Injects (Feedback/Adaptations) Game Master DIS/HLA/GPS/COMMS Training Environment (STE, Live Instrumented, VBS) Trainee A Trainee B Trainee C Trainee D UNCLASSIFIED//FOR OFFICIAL USE ONLY//DRAFT//PRE-DECISIONAL B i o s e n s o r s 15

Total Learning Architecture (TLA) Ca. SS Universal Learner Record Recommender Engine Adaptive Learning Engine (GIFT) Domain Modeling Assessment Coaching/Feedback Reinforcement Learning Real-time AAR Individual/Team Differences Enabling Tech Comms Capture/Parsing (NLP) Individual Task Work Team Cohesion Team Task Work Team Coord Team Comms Affect/ Workload Data Gateway TALK ON (Tank PLT) CGSC (Com Gen Staff School) USMA Convoy/Cyber (PLT) Competency (SPM) Camp Shelby (SQD) Team Marksmanship FO Smart (Indirect Fires) JRTC/CTC-IS (Mission Command) Others? (ASU, Army U, MCo. E) Synthetic Training Environment (IVAS, S/SVT, RCVT, OWT, CTC-IS, etc. ) Data Parser/Manager Unit B Unit C Unit D B io s e n s o rs Battle Space Visualization (ARES) Multi-Modal Interaction (Constructive vs. 3 D Interfacing) SMASH Library (Sniper at x, y, z)(NLP induced) Adaptations/MSELs (Manage Complexity) Cyber Injects Scenario Selection (Cob. Webs) (Targeted Objs) VBS 3 (Squad) Unit A Sensors (State Classifiers) (MASTR-E? ) Gesture Control Intelligent Adaptive Training SAM T (Fire Team) AAR Interfacing/Replay (CTC-IS Use Case) Physio Model Visualization? ? ALE Assessment Widget ALE Strategy Handler Cyber Threat Attribution Models (COBWEBS) UNCLASSIFIED Scenario Run-Time Visualization (Sand, Floor, AR, VR, etc) Data Layering (Echelon, Fog of War, etc. ) 16

GIFT Modifications UNCLASSIFIED 17

ADDING A HUMAN (OC) TO THE LEARNING EFFECT CHAIN UNCLASSIFIED 18

ADDING A HUMAN (OC) TO THE COLLECTIVE LEARNING EFFECT CHAIN (Sottilare et al. , 2018) UNCLASSIFIED 19

GAME MASTER INTERFACE (ALPHA) UNCLASSIFIED 20

GAME MASTER INTERFACE (ALPHA) UNCLASSIFIED 21

GAME MASTER INTERFACE (ALPHA) UNCLASSIFIED 22

This currently exists… Demonstration in the domains: (1) Cyber Bullets in Convoy Operation and (2) Clear Building Procedures UNCLASSIFIED 23

Link to Persistent Modeling and Recommendation Engines UNCLASSIFIED 24

Squad Performance Model: Currently Under Development UNCLASSIFIED 25

CASS Mapping (BTL Level) Squad Model Training Managers create these (METL tracking? ) CASS Mapping (Unit Level) Soldier 1 Movement Competency Soldier 1 Shooting Competency Soldier 2 Movement Competency Soldier 2 Shooting Competency CASS Mapping x. API Data Lake x. API Scenarios create these via GIFT x. API specifications (authored by domain experts… Represented in GIFT task model) x. API ROE Presence Patrol Weapon Control UNCLASSIFIED x. API Reporting 26

UNCLASSIFIED//FOR OFFICIAL USE ONLY//DRAFT//PRE-DECISIONAL PROPOSED ARCHITECTURE Feedback/Remediation x. API SAM T Shooter Weapon: Aim (x, y) Trigger Squeeze LMS STE TMT (GIFT) Coaching Model Bio. Harness Sensor Shooter Performance Bio. Harness: Breathing ECG Posture Performance : Hit/Miss Casualties ? ? Eye-Tracking (Nov. 19? ) Total Learning Architecture Competency Framework: SQD Performance Model Team/Trainee Model Recommender Engine Assessment Team Performance Model Back-Up Behavior Models Expert Marksmen Models Team Coordination Model Raw Data Eye Tracking: Gaze/Fixation Communication: ? ? ? OC Input Learner Record Store UNCLASSIFIED//FOR OFFICIAL USE ONLY//DRAFT//PRE-DECISIONAL Universal Trainee Record 27 8

UNCLASSIFIED//FOR OFFICIAL USE ONLY//DRAFT//PRE-DECISIONAL THANK YOU ? ? Questions? ? 3/8/2021 28 UNCLASSIFIED//FOR OFFICIAL USE ONLY//DRAFT//PRE-DECISIONAL 28