MLMI 2005 Edinburgh Projective Kalman Filter Multiocular Tracking

MLMI 2005, Edinburgh Projective Kalman Filter: Multiocular Tracking of 3 D Locations Towards Scene Understanding C. Canton 1, J. R. Casas 1, A. M. Tekalp 2, M. Pardàs 1 Technical University of Catalonia, Barcelona, Spain 1 Koç University, Istanbul, Turkey 2

Outline l l l Introduction Problem statement & Objective Projective Kalman Filter (PKF) – – l l l Data scenario and formulation Data association problem on P 3→P 2 Results & Performance Conclusions & Future Research Questions

Introduction l Tracking 3 D locations within the Smart. Room scenario towards scene understanding can provide useful information (tracking of persons, head, …)

Problem statement l Standard approaches to track 3 D locations from its 2 D projections on N calibrated cameras involve: 2 D feature selection over the N images 3 D location estimation Correspondence search among views Kalman tracking Drawbacks: • Two disjoint problems • Data from N cameras is regarded as one single observation Initialization • Occlusion is handled in the estimation process but not in the tracking

Objective l Define a filtering scheme to track a 3 D location from its N projections 2 D feature selection over the N images Correspondence search among views Joint 3 D location estimation and tracking Improvements: • Unified framework • Projective nature of N observations is taken into account Initialization • Joint 3 D/2 D occlusion detection scheme

Example

Kalman Filter (KF) Model l When estimating a state s R 3 of a discrete time process governed by the linear stochastic difference equation with a measuremement z R 2 x. N that is Kalman filter provides the optimal solution under the conditions: • Relations between hidden and observed data are linear Projection is non-linear when seen as a morphism : R 3→N 2 • w[t] and v[t] have normal distribution Occlusions make this hypothesis unfeasible

Projective Kalman Filter (I) l Motivation: – – – Track a 3 D location in Euclidean coordinates taking advantage of projective geometry Model non-linearity between the hidden state s[t] and the observed data z[t] tacking into consideration the projective nature of the observations Handle non-Gaussian impulsive noise: detect occlusion and disregard occluded data Kalman theory can be applied to track 3 D locations (with a Newtonian dynamic model) taking its projections as input data.

Projective Kalman Filter (II) Modelling non-linearity l Tackling projection non-linearity through observation matrix H: An During adaptive Kalman filter design evolution, of H[t] when based applyingon H to a compensation the state vector s[t] of coordinates the non-linearity might notfrom be in the prediction image planeof(z 1). the estimated state resolves the conflict (z=1).

Projective Kalman Filter (III) Noise model l Observation noise covariance matrix R[t] controls how reliable is an observation. An adaptive approach to handle Gaussian noise and occlusions would be: where: Criterium to set the parameter βk from the PKF scheme: DATA ASSOCIATION & OCCLUSION DETECTION

Data association on P 3→P 2 (I) l Twofold objective: – – Determine the spatial correspondence of two projections generated by the same 3 D feature at two consecutive time instants in the same image Detect an occlusion in a given view and modify R[t+1] accordingly

Data association on P 3→P 2 (II) State Estimation Extrapolation Data Bounding Projection & Data Association Occlusion Detection

Results l Two types of data: – – l Synthetic: Exact algorithm evaluation and performance purposes Real: Practial usage of this technique within a Smart. Room scenario to track the head of present people Data specifications: – – 4 Calibrated cameras 768 x 576 pixels, 25 fps

Results on Synthetic Data (I) l First scenario: Gaussian noise • PKF outperforms KF by ~35%. Interest Region

Results on Synthetic Data (II) l Second scenario: Gaussian and impulsive noise (occlusions) • PKF outperforms KF when occlusions are present • Influence of occlusions is reduced by the data association process Interest Region

Results on Real Data (I) l l l Applied to track 2 people inside the Smart. Room at UPC towards scene understanding applications Input 2 D data is the top of non-overlapped foreground regions When the 2 people are close, KF loses track but PKF keep it properly

Results on Real Data (II)

Conclusions & Future Work l Conclusions: – – – l New scheme to track 3 D locations from multiple views embeding Kalman theory and projective geometry Model multiple projections of a 3 D location into a tracking loop Occlusion detection combining 2 D/3 D data Comparable computational complexity between PKF and KF Real-time performance Future Work: – – Comparison with Particle Filtering tracking schemes Apply this technique to body tracking into a Smart. Room

The End Thank you!!!!