Automated LowLevel Analysis and Description of Diverse INtelligence

Automated Low-Level Analysis and Description of Diverse INtelligence (ALADDIN) Video Intelligence Advanced Research Projects Activity (IARPA) John Garofolo, ALADDIN Program Manager INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

The Challenge • Massive numbers of video clips are generated daily on many types of consumer electronics. – Such clips can include any possible scene and events, and generally have limited quality control. – The "unconstrained" video clips produced by anyone who has a digital camera present a significant challenge for manual as well as automated analysis. • Analysts who must work with such data are overwhelmed by its volume and lack of tools to probe it. – No technology currently exists to provide an analyst with the information necessary to fully understand the content or temporal processes within these data. INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

The Goal • Develop advanced technology to empower analysts to create sophisticated event-oriented queries against diverse collections of open-source video clips – Video clip collections are processed once to create a general representation of event-related content that is useful for many potential queries. – Analysts create specific event search queries that are modeled after observational reports of events. – Information in the content representation is quickly analyzed for relevance to the queries and the most relevant clips are highlighted and textually summarized in an analyst-friendly form. INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

ALADDIN Event Queries • An event: – is a complex activity occurring at a specific place and time; – involves people interacting with other people and/or objects; – consists of a number of human actions, processes, and activities that are loosely or tightly organized and that have significant temporal and semantic relationships to the overarching activity; – is directly observable. • Is searched via an Event Query incorporating – – – a descriptive name of the event to be searched textbook-style definition for the event, explication of event domain-specific knowledge a list of evidence that an analyst would use to identify the event a small set of video clips containing examples of the event. INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

ALADDIN Approach Sorted Listing of Clips with Confidences and Recountings Event Query Event Agent Generator Open Source Video Clip Collection Content Description Representation Generator CDR Data and Knowledge Sources INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

Example ALADDIN Event Query Courtesy of NIST MED 11 Evaluation Event Name: Working on a woodworking project Exemplars: Definition: One or more people fashion an object out of wood. Event Explication: Woodworking is a popular hobby that involves crafting an object out of wood. Typical woodworking projects may range from creating large pieces of furniture to small decorative items or toys. The process for making objects out of wood can include cutting wood into smaller pieces …. (continues) Evidential Description: scene: Often indoors in a workshop, garage, artificial lighting. Occasionally outdoors objects/people: Woodworking tools (automatic or non-automatic saws, sander, knife), paint, stains, sawhorses, toolbox, safety goggles activities: Cutting and shaping wood, attaching pieces of wood together, smoothing/sanding wood sounds: sawing, chopping, planning, hammering, Target User: An Internet information analyst or experienced Internet searcher with event-specialized knowledge. INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

ALADDIN Event Search Agents Generates – a confidence probability for each clip in the collection about its relevance to the event query. – a hard decision about the relevance of each clip using the confidence threshold and confidence probability (Multimedia Event Detection) – a detailed recounting (summary transcript) of the event-salient information about the clips (Multimedia Event Recounting) – describing the spatial and temporal details of the occurrence – including the key observations regarding the scene, people, objects, sound, speech, text, and activities pertaining to the event occurrence – that is useful for users to perform analyses of the detected evidence and system decision – that is clear, concise, user-oriented, and is useful for downstream processing – that can be rendered at various levels of detail INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

Example Recounting – Fighting a Vehicle Fire Fighting a vehicle fire (99%) <fire> = fire (99%), explosion (98%), slow motion video of fire and explosion (85%) <vehicle> = a big rig (90%), truck (97%), possible fuel truck (50%) <fighting fire> = spraying water on fire (88%), fire truck (98%) <geographic location> = LA Freeway (99%), Northbound on 101 near Sunset Blvd (1/4 mile), Hollywood Blvd (3/4 mile), & Gower Street (1 ½ mile) (92%) <auxiliary> = helicopter (97%), newsworthy, important (98%), traffic stopped both ways (89%), daytime (92%), small fire on hill nearby (90%), five people in line for picture (80%), parked cars and people standing (89%) One perspective: 1. Not a script – listing most important items first 2. Hierarchical (only showing 2 levels) 3. Analyst can view several types of “evidence, ” e. g. , – corresponding audio/video clips – extracted components, including multiple instances – a timeline of events 4. Items grouped and summarized INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

ALADDIN Program Phases and Metrics Final Goals INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)

Training Events Process-Observed Events Testing Events The TRECVID MED ‘ 11 Events Process-Observed Events Attempting a board trick Feeding an animal Landing a fish Working on a woodworking project Changing a vehicle tire Getting a vehicle unstuck Grooming an animal Making a sandwich Parkour Repairing an appliance Working on a sewing project Life Events Wedding ceremony Life Events Birthday party Flash mob gathering Parade

Program Manager Contact Info John Garofolo IARPA, Incisive Analysis Office of the Director of National Intelligence Advanced Research Projects Activity Washington, DC 20511 Fax: 301 -851 -7672 Phone: 301 -851 -7438 Electronic mail: john. s. garofolo@ugov. gov INTELLIGENCE ADVANCED RESEARCH PROJECTS ACTIVITY (IARPA)