Journes Thmatiques Filtrage Particulaire Image Sequence Based Particle

Journées Thématiques Filtrage Particulaire Image Sequence Based Particle Filter for Point Tracking Elise Arnaud - IRISA Etienne Mémin - IRISA Bruno Cernushi-Frias - Universitad de Buenos Aires

Introduction • Objective : Point tracking in computer vision : Reconstruction of a point trajectory along a given image sequence Particular framework with no a priori knowledge • Problem on which many high level tasks depend : motion estimation, 3 D reconstruction, dynamic vision, etc. • Applications : robotics, medical imaging, meteorological imaging, surveillance, etc.

Presentation 1. Introduction 2. Related works on point tracking 3. Why an image sequence based filter ? 4. Image sequence based particle filter 5. Application to point tracking 6. Results and comparison 7. Conclusion 8. Perspectives

Related works on point tracking 2. 1 Assumptions 2. 2 Matching approaches 2. 3 Differential approaches 2. 4 Use of filtering methods

Assumptions Related works on point tracking (1) Motion hypotheses : drifting point, constant velocity, constant acceleration, periodic motion • too difficult to model without any a priori knowledge • too restrictive in the case of abrupt changes of the trajectory (2) Relative position conservation within a rigid geometric structure of the scene • Problem in the case of points moving independently of the scene (3) Luminance pattern conservation along the trajectory

Matching Approaches Related works on point tracking Luminance pattern conservation • Maximization of a similarity criterion between the target point and the candidate point • Similarity criteria based on a description of the luminance pattern • Necessity of exhaustive research time consuming • Most similarity criteria are not invariant to affine changes • Comparative study of the most used criteria [Aschwanden 92]

Differential approaches Related works on point tracking Luminance pattern conservation • Differential formulation of a similarity criterion • Point intensity conservation optical flow constraint • Sum of square difference Shi-Tomasi-Kanade tracker [Shi 94]

Use of filtering methods Related work on point tracking • To design a tracker more robust to outliers and occlusions • State of the filter : feature position (+ intensity + velocity ) • Kalman filter for tracking in an image sequence : [Nguyen 01] [Meyer 94] [Ricquebourg 00]

Why an image sequence based filter ? 3. 1 Notations 3. 2 Which type of available measures ? 3. 3 Which type of dynamic ? 3. 4 Objectives and assumptions

Notations Why an image sequence based filter ? state of the system at time k trajectory from time 0 to time k measure at time k measures from time 1 to time k random vector corresponding to an image at time k image sequence from time 0 to time k

Which type of available measures ? Why an image sequence based filter ? • All available information contained in the image sequence Ideal case : measure = image • Often impossible to specify the relationship between state and image ( too complex structure and large size ) Use of a condensed information obtained from the sequence • Highly nonlinear form with respect to the images • Simple form with respect to the state Measure equation :

Which type of dynamic ? Why an image sequence based filter ? • Dynamic a priori (constant velocity, periodic motion etc. ) • Dynamic captured from learning [Blake 98] • If no a priori knowledge : Use of a dynamic obtained from the image sequence Dynamic equation :

Objectives and assumptions Why an image sequence based filter ? • Objective : estimation of the trajectory given all available data, i. e. measures and image sequence • Assumptions :

Objectives and assumptions Why an image sequence based filter ? • Dynamic equation and measures depend on the sequence • Such a dependency has to be taken into account • Conditioning with respect to the image sequence in the equations of the filter Definition of an image sequence based filter • Case of nonlinear dynamic or nonlinear measure equations Definition of an image sequence based particle filter

Image sequence based particle filter • Objective : approximation of • Knowledge of - N samples according to the importance function - N associated normalized weights • Non-normalized weights given by :

Image sequence based particle filter • Recursive equation of the importance function assumed • Recursive formulation of the weights • Increase over time of the weights’ variance • Optimal importance function minimizes the weights’ variance conditioned upon • New recursive formulation of the weights

Image sequence based particle filter • Initialisation for k = 1 … n • Measures Evaluate the measure and the dynamic from the image sequence • Prediction • Weights update • Resampling • Trajectory estimation

Application to point tracking Proposed tracker combines a dynamic and some measures all depending on the image data State of the filter : point position 4. 1 Conditional observation equation 4. 2 Conditional dynamic equation 4. 3 Image sequence based particle filter for point tracking

Conditional observation equation Application to point tracking • Restriction : linear observation equation • : result of an estimation process on the image sequence • : Gaussian white noise conditionally to is a Gaussian function

Conditional observation equation Application to point tracking • : most similar point to in image • Several matching criteria can be used to quantify the similarity • Criterion used invariant to affine transformations, originally defined for image matching applications [Schmid 97] • Measure carries enough information to write :

Conditional observation equation Application to point tracking initial point initial vector of characteristics - characterization of the luminance pattern in the neighborhood - invariant to affine transformations selected measure associated to the vector the most similar to the initial vector . . .

Conditional dynamic equation Application to point tracking • Use of an instantaneous motion measure from image data • • • : Gaussian white noise conditionally to : motion vector of : motion parameters vector, result of an estimation process between images and

Conditional dynamic equation Application to point tracking • Point belonging to the background : - motion parameters vector corresponds to a unique global linear motion - linear dynamic equation Image sequence based Kalman filter • Point with a motion different from the global motion : - motion parameters vector corresponds to a local linear motion - nonlinear dynamic equation Image sequence based particle filter

Image sequence based particle filter for point tracking Application to point tracking and Gaussians • Knowledge of the optimal importance function, which is Gaussian. • Knowledge of the distribution involved in the weights recursion, which is also Gaussian

Image sequence based particle filter for point tracking Application to point tracking New image at time k Position in image k Evaluate the observation by a matching method New particles and associated weights For each particle, evaluate the motion vector by a differential method Resampling

Results and comparisons Sequence Caltra - 40 frames (190 180 pixels) Image sequence based particle filter for tracking Shi-Tomasi-Kanade tracker

Results and comparisons Sequence Meteo - 14 frames (256 512 pixels) Image sequence based particle filter for tracking

Conclusion • Definition of an image sequence based particle filter • Application to point tracking • Proposed tracker combines a dynamic and some measures all depending on the image data • No a priori knowledge • Trajectories undergoing abrupt changes • Sequence with a cluttered background

Perspectives • Consider a confidence measure of the observation • Include occlusion rules in the tracker • Test other dynamic calculated from image sequence : use of dense motion field • Application to fluid imagery ( meteorological sequence )