SAMI Situational Awareness from Multimodal Input Naveen Ashish

SAMI: Situational Awareness from Multi-modal Input Naveen Ashish

Talk Organization § § § Why are we at RESCUE interested ? Situational Awareness (SA) – Introduction System architecture Research challenges Expected outcomes and artifacts Extraction system demonstration

Team Naveen Ashish Sharad Mehrotra Nalini Venkatasubramanian Utz Westermann Dmitry Kalashnikov Stella Chen Vibhav Gogate Priya Govindarajan Ram Hariharan John Hutchinson Yiming Ma Dawit Seid Jay Lickfett Chris Davision Quent Cassen Bhaskar Rao Mohan Trivedi Rajesh Hegde Sangho Park Shankar Shivappa Ron Eguchi Mike Mio Jacob Green

Information from Various Sources News, video, audio footage Pushing “Human-as-sensor” Emergency responders People/Victims at disaster GIS, satellite imagery, maps

More Data ≠ More Information Where are the fire What Have all areas medical should supplies we personnel reached start evacuating ? ? first ? SA

Situational Awareness § § Wide variety of fields – Beginning in mid-80 s, accelerating thru 90 s – Fighter aircraft, ATM, Power plants, Manufacturing Definitions – "the perception of elements in the environment along with a comprehension of their meaning and along with a projection of their status in the near future" – "the combining of new information with existing knowledge in working memory and the development of a composite picture of the situation along with projections of future status and subsequent decisions as to appropriate courses of action to take" Knowing what is going on § § Situational awareness and decision making Areas – Cognitive science – Information processing – Human factors

Abstraction of Information Awareness Events Multimodal Input: Text, Audio, Video

First-cut Architecture Text Audio EVENT EXTRACTION VISUALIZATION and USER INTERFACES Video Internet KNOWLEDGE: ONTOLOGIES REFINEMENT Disambiguation Location Graph View Spatial Indexing PDF Histogram Querying and Analysis RAW DATA EVENT BASE Centered around EVENTS as fundamental abstractions

Research Areas Event Modeling Event Extraction Disambiguation GIS Querying Location Uncertainty Graph Analysis

Event Modeling § § What is an event ? Event Representation TIME LOCATION TYPE PEOPLE REPORT EVACUATION RELIABILIT Y AGENCY NAME LOCATION OPERATION FROM TO NUMBER

Domain Knowledge THAILAND EVACUATION IS-A ……. IS-A ROAD EVACUATION SOUTHERN REGION AIR EVACUATION PHUKET, CHANGWAT § Captured as Ontologies PHUKET

Event Extraction § § § Long history of information extraction – IR (MUC efforts) – Web data extraction DARPA ACE – Entities, Relations, Events – Events in 2004 Event extraction accuracy is still low SA Domain – Stream of information – Duplicated, ambiguous – Reliability – Conversations Modalities – Text

Semantics Driven Approach § § Semantics Driven Challenges – Framework – Ontologies § § What semantics required for event extraction ? Application § With NLP, ML techniques § Performance – SA specific § Duplicates, reconciliation, temporal, conversations …. .

Disambiguation

Disambiguation

Uncertainty is a Challenge Report 1: “. . . a massive accident involving a hazmat truck on I 5 -N between Sand Canyon and Alton Pkwy. . . ” Report 2: “. . . a strange chemical smell on Rt 133 between I 405 u uncertain, not (x, y) – reasoning on such data u support all types of queries Re terms of landmarks t 1 por u in Re – point-location po rt 2 and Irvine Blvd. . . ”

Implications of Uncertainty in Text How to model uncertainty? – probabilistic model – P(location | report) u e. g. report says “near building A” Queries – cannot be answered exactly. . . u use probabilistic queries u all events: P(location R | report) > 0 – SA requirements u triaging capabilities u fast response – top-k – threshold: P(location R | report) > – -RQ, k -RQ How to map text to probabilities? – use spatial ontologies A R B

Graph Analysis § § § GAAL Inherent spatio-temporal properties Graphs are powerful for querying and analysis

GIS Search Current FGDC Search

GIS Search Progressive Refinement of Data

Deliverables, Outcomes, Artifacts § § “Vertical” thrusts – Event extraction system (TEXT) – Disambiguation system – GIS search system Overall system demonstration ? “By-products” – Ontologies Computer science research areas Databases Semantic-Web Information Retrieval Intelligent Agents (AI)

http: //sami. ics. uci. edu Thank you !