An Object Tacking Paradigm with Active Appearance Models

An Object Tacking Paradigm with Active Appearance Models for Augmented Reality Presented by Pat Chan Pik Wah 28/04/2005 Qualifying Examination

Outline n n Research Objective Introduction ¡ ¡ ¡ n Proposed Object Tracking Paradigm ¡ ¡ n n Augmented Reality Object Tracking Active Appearance Models (AAMs) Paradigm Architecture Experiments Research Issues Conclusion

Research Objective n n n Object tracking is an essential component for Augmented Reality. There is a lack of good object tracking paradigm. Active Appearance Models is promising. Propose a new object tracking paradigm with AAMs in order to provide a real-time and accurate registration for Augmented Reality. Nature of the paradigm: ¡ ¡ ¡ Effective Accurate Robust

Augmented Reality n n An Augmented Reality system supplements the real world with virtual objects that appear to coexist in the same space as the real world Properties : ¡ ¡ ¡ Combine real and virtual objects in a real environment Runs interactively, and in real time Registers (aligns) real and virtual objects with each other

Augmented Reality n Projects related to AR

Augmented Reality n Display ¡ n Tracking ¡ n Following user’s and virtual object’s movements by means of a special device or techniques 3 D Modeling ¡ n Presenting virtual objects on real environment Forming virtual object Registration ¡ Blending real and virtual objects

Object Tracking n n Visual content can be modeled as a hierarchy of abstractions. At the first level are the raw pixels with color or brightness information. Further processing yields features such as edges, corners, lines, curves, and color regions. A higher abstraction layer may combine and interpret these features as objects and their attributes. Object edges, corners, lines, curves, and color regions Pixels

Object Tracking n n n Accurately tracking the user’s position is crucial for AR registration The objective is to obtain an accurate estimate of the position (x, y) of the object tracked Tracking = correspondence + constraints + estimation Based on reference image of the object, or properties of the objects. Two main stages for tracking object in video: ¡ ¡ Isolation of objects from background in each frames Association of objects in successive frames in order to trace them

Object Tracking n Object Tracking can be briefly divides into following stages: Input (object and camera) ¡ Detecting the Objects ¡ Motion Estimation ¡ Corrective Feedback ¡ Occlusion Detection ¡

Object Tracking n Expectation Maximization ¡ ¡ n Kalman Filter ¡ n Find the local maximum likelihood solution Some variables are hidden or incomplete Optimal linear predict the state of a model Condensation ¡ ¡ Combines factored sampling with learned dynamical models propagate an entire probability of object position and shape

Object Tracking n Pervious Work : Marker-based Tracking ¡ Feature-based Tracking ¡ Template-based object tracking ¡ Correlation-based tracking ¡ Change-based tracking ¡ 2 D layer tracking ¡ tracking of articulated objects ¡

Pervious Work n n n Marker-based Tracking Marker-less based Tracking Feature-based Tracking ¡ ¡ Shape-based approaches Color-based approaches

Pervious Work n Template-based object tracking ¡ Fixed template matching n n ¡ Image subtraction Correlation Deformable template matching

Pervious Work n Object tracking using motion information ¡ ¡ ¡ Motion-based approaches Model-based approaches Boundary-based approaches n n ¡ Snakes Geodesic active contour models Region-based approaches

Active Appearance Models n n n The Active Appearance Model (AAM) algorithm is a powerful tool for modeling images of deformable objects. AAM combines a subspace-based deformable model of an object’s appearance Fit the model to a previously unseen image.

Timeline for development of AAMs and ASMs

Active Appearance Models (AAMs) n 2 D linear shape is defined by 2 D triangulated mesh and in particular the vertex locations of the mesh. n Shape s can be expressed as a base shape s 0. n pi are the shape parameter. s 0 is the mean shape and the matrices si are the eigenvectors corresponding to the m largest eigenvalues n

Active Appearance Models (AAMs) n The appearance of an independent AAM is defined within the base mesh s 0. A(u) defined over the pixels u ∈ s 0 A 0(u) A 1(u) A 2(u) A 3(u) n A(u) can be expressed as a base appearance A 0(u) plus a linear combination of l appearance n Coefficients λi are the appearance parameters.

Active Appearance Models (AAMs) n The AAM model instance with shape parameters p and appearance parameters λ is then created by warping the appearance A from the base mesh s 0 to the model shape s. M(W(u; p)) Piecewise affine warp W(u; p): (1) for any pixel u in s 0 find out which triangle it lies in, (2) warp u with the affine warp for that triangle.

Fitting AAMs n n n Minimize the error between I (u) and M(W(u; p)) = A(u). If u is a pixel in s 0, then the corresponding pixel in the input image I is W(u; p). At pixel u the AAM has the appearance At pixel W(u; p), the input image has the intensity I (W(u; p)). Minimize the sum of squares of the difference between these two quantities: u u

DEMO Video – 2 D AAMs

Recent Work for Improving AAMs n Combine 2 D+3 D AAMs

Combined 2 D + 3 D AAMs n At time t, we have 2 D AAM shape vector in all N images into a matrix: n Represent as a 3 D linear shape modes W = MB = n

Compute the 3 D Model AAM shapes AAM appearance First three 3 D shapes modes

Constraining an AAM with 3 D Shape n Constraints on the 2 D AAM shape parameters p = (p 1, … , pm) that force the AAM to only move in a way that is consistent with the 3 D shape modes: n and the 2 D shape variation of the 3 D shape modes over all imaging condition is: n Legitimate values of P and p such that the 2 D projected 3 D shape equals the 2 D shape of AAM. The constraint is written as:

An Object Tacking Paradigm with Active Appearance Models

Proposed Object Tracking Paradigm Architecture Occlusion Detection Training Images Training Active Appearance Model 1. 2. Shape Model Appearance Model Video Initialization Modeling Kalman Filter

Steps in Object Tracking Paradigm n Preporcessing ¡ ¡ n Initialization ¡ ¡ n Locating the object position in the video. In our scheme, we make use of AAMs. Motion Modeling ¡ ¡ n Training the Active Appearance Model. Get the shape model and the appearance model for the object to be tracked. Estimate the motion of the object Modeling the AAMs as a problem in the Kalman filter to perform the prediction. Occlusion Detection ¡ Preventing the lost of position of the object by occluding of other objects.

Enhancing Active Appearance Models n Shape n Appearance n Combine the shape and the appearance parameters for optimization n In video, shape and appearance may not enough, there are many characteristics and features, such as lightering, brightness, etc… L=[L 1, L 2, ……, Lm]T

Iterative Search for Fitting Active Appearance Model

Iterative Search for Fitting Active Appearance Model Can be improved by: 1. Prediction matrix 2. Searching space

Initialization for AAMs

Motion Modeling n n Initial estimate in a frame should be better predicted than just the adaptation from the previous frame. Can be achieved by motion estimation AAMs can do the modeling part Kalman filter can do the prediction part

Kalman Filter n n Adaptive filter Model the state of a discrete dynamic system. Originally developed in 1960 Filter out noise in electronic signals.

Kalman Filter n Formally, we have the model n For our tracking system,

Kalman Filter

Occlusion Detection n WHY? ¡ ¡ ¡ n Positioning of objects To perform cropping When a real object overlays a virtual one, the virtual object should be cropped before the overlay HOW? ¡ ¡ ¡ High resolution and sharp object boundaries Right occluding boundaries of objects Camera matrix for video capturing

Proposed Object Tracking Paradigm Architecture Training Images Training Active Appearance Model 1. 2. Shape Model Appearance Model Occlusion Detection Video Initialization Active Appearance Model Fitting Kalman Filter

Experimental Setup n n n AAM-api from DTU Open. CV Pentium 4 CPU 2. 00 GHz and 512 MB RAM

Experiment on AAMs (1) n Training Image

Experiment on AAMs (1) Shape Texture

Experiment on AAMs (1) Initialization After optimized

Demo Video

Experiment on AAMs (2) n Training Images

Experiment on AAMs Shape Texture

Experiment on AAMs n Trapped in local minimum Initialization After optimized

Experiment on AAMs

Experiment on AAMs n Fit to the face Initialization After optimized

Experiment on AAMs

Object Tracking with AAMs

Experiment on Kalman Filter

Demo Video

Experiment on Kalman Filter

Demo Video

Research Issues n AAMs tracking is accurate ¡ ¡ n Kalman filter help is to increase the speed in prediction ¡ n n Very slow Cannot perform real-time tracking Modeling the problem from AAMs to Kalman Filter Improving the fitting algorithm in the AAMs Occlusion detection ¡ ¡ Important to object tracking Preventing the lost of the position

Conclusion n We have done a survey on object tracking and active appearance model is done We proposed a paradigm on video object tracking with active appearance models Goal: ¡ ¡ ¡ n Robust Real-time Good performance We have done some initial experiments: ¡ ¡ Experiments on AAMs Experiments on Kalman filter for object tracking

Q&A