MultiObject Detection and Tracking from a Moving Platform

Multi-Object Detection and Tracking from a Moving Platform

Tracking from a Moving Platform 1 -Analysis and detection: • Registration across video group of frames (VGo. F) • Detection and segmentation of motion blobs (background models, shadow) 2 -Representation and tracking: • Video object representation (shape, color descriptors, geometric models) • Object tracking (prediction, correspondence, occlusion resolution etc. ) 3 -Access and event modeling: • Efficient data structures for video queries in high-dimensional feature space • High-level event representation

Multi-Object Tracking 1. Detect moving objects in stabilized frames. 2. Predict locations of the current set of objects. 3. Match predictions to actual measurements. 4. Update object trajectories. 5. Update image stabilized ref coord system. Multi-object Detection and Tracking Unit Tracking VGo. F Registration Into Common Coordinate System Moving Object Detection & Feature Extraction Context Data Association (Correspondence) Update Trajectories Prediction Update Coord System Object States

Dynamic State Estimation for Tracking System state Dynamic System noise System Errors • Agile motion • Distraction/clutter • Occlusion • Changes in lighting • Changes in pose • Shadow (Object or background models are often inadequate or inaccurate) State estimate Measurements Measurement System State Estimator Measurement noise Measurement Errors • Camera noise • Framegrabber noise • Compression artifacts • Perspective projection State uncertainties State Error • Position • Appearance • Color • Shape • Texture etc. • Support map

Motion Detection- 3 D Spatiotemporal Volume Spatio-temporal volume of hall monitor sequence: (a) Left entire volume, (b) Middle: cut taken at vertical position y 0, (c) Right: Cut taken at vertical Position y 1. Gerald Kuhne, “Motion-based segmentation and classification of Video Objects” Dissertation Univ. of Mannheim, 2002

Motion Detection - Structure and Flux Tensor Approach Typical Approach: threshold trace(J) Problem: trace(J) fails to capture the nature of gradient changes and results in ambiguities between stationary versus moving features Alternative Approach: Analyze the eigenvalues and the associated eigenvectors of J Problem: Eigen-decompositions at every pixel is computationally expensive for real time performance Proposed Solution: Flux tensor time derivative of J

Motion Detection Flux Tensor vs Gaussian Mixture

Multi-object Tracking Stages Probabilistic Bayesian framework Features Used in Data Association: Proximity and Appearance-based Data Association Strategy: Multi-hypothesis testing Gating Strategies: Absolute and Relative Discontinuity Resolution: Prediction (Kalman filter), or Appearance models Filtering: Temporal consistency check and Spatiotemporal cluster check

Association Strategy • Multi-hypothesis testing with delayed decision - Many matches are kept with evidence-based pruning • Support for multiple interactions - one-to-one object matches, many-to-one, one-to-many, many-to-many, oneto-none, or none-to-one matches • Corresponding low-level object tracking events • Segmentation errors • Group interactions (merge/split) • Occlusion • Fragmentation • Entering object • Exiting object Object. Match. Graph

Match Confidence Computation Match confidence quantifies correspondence goodness-of-fit Confidence value has two components: • Similarity confidence (Confsim) • Separation confidence(Confsep) 1, j* is the closest competitor in terms of distance Conf(i, j) Link Nodej -bounding box - support map -centroid -area etc. Nodei -bounding box - support map -centroid -area etc.

Trajectory Segment Generation • Trace links in the Object. Match. Graph to generate possible trajectory segments • Segment. List - Linked list of inner nodes (objects/cells) • Trajectory labeling - Extracted trajectory segments are labeled using a modified connected components labeling • Trajectory linking - Trajectories are formed by linking unfiltered segments sharing the same label. Trajectory Source Split Inner Segment Merge Sink Source-Split Object. Match. Graph Single Split-Merge Sink-Merge

Data Hierarchy Node (Object-Region) Segment Macro segment Trajectory Nod e Typ e Cent roid Bou ndin g Box Area Sup port Map Im. R GB Pare nts Chil dren Segment Type Label Consistency Start_ frame, object, child, nodetype End_ frame, object, nodetype Objects Centers Trajectory_type Trajectory_displacement Trajectory_length Trajectory_boundingbox Parents Children Trajectory Type Label Start_frame Start_position End_frame End_position Length Displacement Diagonal Segments

Need for Local Registration

Exp Results: DARPA ET 01 Video Frame #50 Registered Frame Foreground Mask Motion Detection Results Tracking Results

Exp Results - NGA Crystal View HD Video Frame #787 in Coord. #740 c) Predictions UPS d) After occlusion handling

Future Work - Trajectory Matching and Filtering • Establishing trajectory continuity (object ID matching) across moving coordinate systems • Customizing trajectory analysis for airborne video tracking with misregistration error, large platform motion, zooming, etc • Maintaining temporal consistency of trajectories • Removing periodic clustered trajectories • Resolving discontinuous trajectories • Trajectory display and visualization: video vs mosaic

Future Work – Performance Optimization and Tuning • Moving object detector filters • Flux tensor fixed optimal threshold learning or continuous adaptive thresholding • Morphological post processing filters • Real-time versus offline MATLAB (approximate): • Flux tensor detection 4 sec/frame • Object tracking 2 sec/frame (for around 10 objects) • Excluding I/O time

Future Work - Near Term Performance Improvements • • • Frame-to-frame registration accuracy difficult to maintain across a hundred frames or more (few seconds of video) Reducing false motion trajectories due to registration errors due to scene structure Maintaining a common coordinate system for registering long airborne video sequence Tracking through large platform motion Dealing with large camera field-of-view changes Platform motion jitter

Future Work - Longer Term Performance Improvements • • Filtering periodic motions produced by clutter, etc. Shadows (e. g. false detections, shape distortions, merges) Sudden illumination changes (e. g. due to cloud movements) Glare from specular surfaces (e. g. windshields, water surfaces) Perspective distortion (e. g. object size, shape and position) Trajectory gaps and distortion due to occlusion Poor video quality (e. g. low resolution, low color saturation)