Video Understanding Tasks Overview Action localization is the

Video Understanding • Tasks Overview • Action localization is the cornerstone of high-level video understanding. • Action localization plays the “object detection” role in video understanding. • Video Action Understanding • High-level Video Understanding Polishing shoes Task: Temporal Action Localization Left: Sit, Talk to, Watch; Right: Crouch/Kneel, Listen to, Watch Task: Spatio-temporal Action Localization Task: video referring expression Task: Spatio-temporal Scene Graphs Task: Dense-Captioning Events in Videos

Video Understanding • Akin to Object Detection. • Top-down Methods (similar to Faster R-CNN) • Proposals Generation (video->temporal proposals). (RPN) • Proposals Classification and Refine. (R-CNN) • Bottom-Up Methods (similar to Anchor Free Detectors) • Directly localize the boundary of target video: • start time (left top corner), end time (right bottom corner), matching (associative embedding).

Video Understanding Similar to Semantic Segmentation Similar to Proposal-Based Detector SOTA Similar to Anchor-free Detector

BSN: Boundary-Sensitive Network for Temporal Action Proposal Generation ECCV 2018 Activity. Net 2017 Champion

Motivation • Drawbacks of Top-Down Methods • The main drawback of these methods is that generated proposals are usually not temporally precise or not flexible enough to cover ground-truth action instances of varies duration.

Approach • Overview

Approach • Framework

Approach • Details of proposal generation module.

DEBUG: A Dense Bottom-Up Grounding Approach for Natural Language Video Localization ACL 2019

Motivation • Drawbacks of top-down methods • The performance is sensitive to the heuristic rules. • In order to achieve a high recall, they are required to densely place candidates, which significantly increase the amount of computation and localization time. • Weaknesses of the existing bottom-up methods • Two boundary predictions are independent, i. e. , the model ignores the content consistency between two predictions. • The positive and negative training samples are extremely imbalanced. • Detecting temporal action boundary from frames is still a challenging task.

Contributions • This paper proposes a dense bottom-up framework for NLVL. • Regard all frames falling in the ground truth segment as positive samples. • For each positive frame, DEBUG has a classification subnet to predict its relatedness with the query, and a boundary regression subnet to regress the unique distances from its location to bi-directional ground truth boundaries.

Approach • Overview

Approach • Temporal Pooling

Experiments

Experiments