TRACKING Particle Filter Computer Vision Seminar Hagit HelOr

TRACKING Particle Filter Computer Vision Seminar – Hagit Hel-Or Itzik Berrebi Rotem Klorin

Tracking o o Tracking - the process of locating a moving object (or multiple objects) over time using a camera. Variety of uses: human-computer interaction, security and surveillance, video communication, traffic control…

Tracking - Goal o Finding the objects. o Giving them an identity. o Follow the object as they move around while maintaining their identity – even if the objects collide, disappear for a short time, etc.

Particle Filter

Main differences with Kalman filter: o o o Assumes uni-modal (Gaussian) distribution. Predicts single new state for each object tracked. Updates state based on error between predicted state and observed data.

Main differences with Kalman filter: Particle filter: o o Can work for multi-modal distribution. Predicts multiple possible states for each object tracked. Each possible state has a different probability. Estimates probabilities of predicted states based on observed data.

Particle Vs Kalman

Particle Filter o Particle filter or Sequential Monte Carlo (SMC) methods are a set of algorithms that estimate the posterior density of the state variables given the observation variables.

Particle Filter Predicts multiple possible next states: o Achieved using sampling or drawing samples from the probability density functions.

Recursive Filter o The particle filter is a Bayesian sequential importance sampling technique, which recursively approximates the posterior distribution using a finite set of weighted samples. o Each state is estimated by the information from the last state.

Players o Observations and hidden states. o Particles and weights. o Probability

Observations and Hidden States o The particle filter is designed for a Hidden Markov Model, where the system consists of hidden and observable variables.

Observations and Hidden States Example - hand tracking Observations: o Images (R, G, B matrices)

Observations: o Binary contour images (0/1 matrices)

Observations: o Edge detection images

Observations: o o Harris corners (lists of all detected Harris corners) Other visual features collected from an image

Hidden States: o Hand contour images

Hidden States: o List of splines

Hidden States: o List of phalanx o List of angles between phalanxes

Hidden States: o One of the following: { paper, rock, scissors, lizard, spock}

Assumptions �

Assumptions �

Example Robot plays rock-paper-scissors DEMO

Players o Observations and hidden states. o Particles and weights. o Probability

Particles and Weights o o o Particles - Samples of hypotheses. The hypotheses are estimated positions of the object we are tracking. The more particles we sample, the faster we find the object.

Particles and Weights o o Each particle is given a weight according to its probability of being the next state. Particles with a bigger chance to be the next state will get higher weights.

Particle and Weights o o High probability samples should be drawn more frequently. Low probability samples should be drawn less frequently.

Particle Filter - Example DEMO

Players o Observations and hidden states. o Particles and weights. o Probability

HMM – Hidden Markov Models � S H 0. 2 0. 8 t-TV s-sleep c-cry l-learn H 0. 6 0. 2 0. 1 S 0. 2 0. 3 0. 5 0 H S H 0. 8 0. 2 S 0. 3 0. 7

HMM – Hidden Markov Models S H 0. 2 0. 8 t-TV s-sleep c-cry l-learn H 0. 6 0. 2 0. 1 S 0. 2 0. 3 0. 5 0 �

HMM – Hidden Markov Models �

Particle Filter �

Goal – Posterior Density �

Particle Filter Algorithm

Particle Filter - Algorithm o Sample the particles using a model. o Compute weights according to the model. o Resampling with respect to the weights: replace unlikely samples by more likely ones.

Particle Filter - Example DEMO

Particle Filter o Particle filter consists of essentially two steps: Prediction and update.

Prediction �

Update �

Particles and weights �

Particles and weights �

Numerical Example �

Numerical Example � A B 0. 5

Numerical Example � A A B B

Numerical Example �

Numerical Example o We create four particles o Each particle is equally likely to be in A or B

Numerical Example �

Numerical Example �

Numerical Example �

Numerical Example �

Numerical Example o Let’s say that we end up with 3 particles in A, and 1 particle in B. o We repeat step 2 and 3 until we have the robot localized. o (All particles more or less track the robot, within error).

Example: Real time tracking

Disadvantages o o High computational complexity. It is difficult to determine optimal number of particles.

Matlab Example

MSEPF Mean shift embedded particle filter 60

MSEPF Motivation � https: //www. youtube. com/watch? v=7 SFe. C goep 1 c 61

MSEPF � As computers become spreading in our daily life, people are in great need of natural and efficient human–computer interaction. 62

MSEPF � As a universal body language of mankind, hand gesture is one of the most natural and effective means for humans to communicate non-verbally with others. 63

MSEPF � Hand tracking is a challenging but essential step for hand gesture recognition. � Hand motion is performed in the 3 D space, and captured in 2 D image sequences. 64

MSEPF – in context � In 2 D methods, a hand is represented by its geometric features such as contours, finger tips… 65

MSEPF – in context � or non geometric features such as color skin. 66

MSEPF – in context � Many methods have been developed for tracking after hand motion, but it each of the algorithm was dealing with a specific problem. � All was based on assumptions like: ◦ Constrained viewpoint ◦ Static background ◦ Clutter free environments 67

MSEPF – in context 68

MSEPF – in context � Two successful approaches: � Particle � Mean � one filter shift is stochastic and model-driven while the other is deterministic and data-driven 69

Particle Filter – recall � https: //www. youtube. com/watch? v=O 1 FZy Wz_yj 4 70

Particle Filter – recall Ø The Ø High main problem: computation cost caused by a large number of particles usually makes PFs infeasible for real-time applications. Ø Particle filtering is relatively slow 71

Mean Shift – recall � https: //www. youtube. com/watch? v=kma. QA sot. T 9 s � As an efficient gradient-based optimization method, Mean Shift has fast convergence speed and low computation cost. 72

Mean Shift – recall � The main problem: Mean Shift trackers easily fail in tracking rapid moving objects, and cannot recover from failures. 73

BUT… 74

MSEPF - advantage � More efficient sampling by using mean shift. � Fewer particles needed, hence low computational cost. � Dealing great with hand’s rapid movement. and various kinds of distractors. 75

SO WHAT IS MSEPF? 76

MSEPF � Combine mean shift and particle filter - unite of both approaches for improved tracking. � Blending the MS optimization into particle filtering. Particle Filter Mean Shift 77

MSEPF � Particles are “herded” to the modes of observation after mean shift search, so it concentrates on particles with large weights. 78

MSEPF � Compared to the conventional particle filtering, the number of particles required is reduced greatly. 79

MSEPF Dynamic Model 80

MSEPF � Reminder Dynamic model from hmm: �P(xt|xt-1) 81

MSEPF � Msepf Dynamic model represents hands by rectangles (the size of the rectangle is constant). � The � (x, hand state is defined as x = {x, y}. y) is the coordinate of the rectangle center 82

MSEPF � The Dynamic model dynamic model can be learned from a set of pre-labeled training sequences. � The dynamics of hand motion is modeled as xt − xt− 1 = xt− 1 − xt− 2 + wt− 1, where wt− 1 is a zero-mean Gaussian stochastic component. 83

MSEPF y Dynamic model Xt Xt-1 x=30, y=30 y x x=30. 1, y=29. 9 x 84

MSEPF Observation Model 85

MSEPF � When Observation model we perform natural hand gestures, hand motion may be rapid and the shape of hands in image sequences varies greatly, so it is difficult to track hand reliably based on geometric features such as hand contours. 86

MSEPF � MSEPF Observation model using non geometric features such as color and motion to describe hands. 87

MSEPF � Following Observation model Bradski, the researchers learned a histogram based skin color model using HSV. 88

89

MSEPF - Observation model The skin in channel H is characterized by values between 0 and 50. For Asian people the channel S is characterized by values between 0. 23 to 0. 68. 91

MSEPF - Observation model After some tests it was found that we can use only the channel H with values ranging between 6 and 38 92

MSEPF - Observation model Convert the RGB image to HSV. 93

MSEPF - Observation model all pixels that were classified as skin (using the range in channel H) were set to value 255, and non-skin pixels was fixed to 0. 94

MSEPF - Observation model First, we run dilatation filter that expands the areas in the skin regions. After that run eroding filter to reduce all the imperfections that the dilatation created. 95

MSEPF - Observation model Then, we run a 3 x 3 median filter in order to soften more the results achieved by the dilatation and erosion. 96

MSEPF - Observation model Finally, only skin regions are represented as white pixels. 97

MSEPF - Observation model � https: //www. youtube. com/watch? v=i. D 2 Tm updf 5 w 98

MSEPF � In Observation model reality, there are many skin-colored objects in background such as faces. To deal with the distraction caused by skin-colored objects, the algorithm include motion cue in the observation. � The assumption is that other skin-colored objects like faces in background move more slowly than hands. 99

MSEPF - Observation model � Background � As Substraction? background changes all the time due to the wheelchair’s movement, not background substraction but image differencing is adopted here to detect motion. 10 0

MSEPF � Image Observation model differencing: 10 1

MSEPF - Observation model color probability distribution image motion–color probability distribution image 10 2

MSEPF � Lets define 2 definition (will help us perform MS) 10 3

MSEPF � 10 4

MSEPF � 10 5

Q: when do you think we should use motion cue and when color cue 10 6

MSEPF � It was found that the motion color probability distribution image contains reliable motion and color cues when hand moves rapidly. � On the other hand, when a hand moves slowly, particles can be mainly weighted by the color cue in the color probability distribution image. 10 7

MSEPF � 10 8

MSEPF � 10 9

MSEPF MEAN SHIFT OPTIMIZATI ON 11 0

MSEPF - MS optimization � 11 1

MSEPF � The MS optimization Algorithm: ◦ 1. Compute the observed cues of the rectangle window that the particle represents: 11 2

MSEPF - MS optimization � M 00 is the zeroth moment � M 10 is the first moment for x � M 01 is the first moment for y 11 3

MSEPF - MS optimization � 11 4

MSEPF - MS optimization � 11 5

MSEPF - MS optimization 11 6

MSEPF - MS optimization Move the particle to the center of mass 11 7

MSEPF - MS optimization 11 8

MSEPF Likelihood Function – “the weight” 11 9

MSEPF - Likelihood function – “the weight” � 12 0

MSEPF - Likelihood function � 12 1

MSEPF - Likelihood function � 12 2

MSEPF short summary 12 3

MSEPF – short summary � 12 4

MSEPF – short summary � 12 5

12 6

MSEPF – short summary � 12 7

MSEPF – short summary � 12 8

MSEPF � Experimental results demonstrate the superior performance of the MSEPF over the mean shift tracker and the conventional particle filter 12 9

MSEPF – experiments � Mean shift tracker & conventional particle filter VS. MSEPF � Particle filter: ◦ 160 samples ◦ 63 ms � MSEPF: ◦ 20 samples ◦ 28 ms 13 0

MSEPF – experiments 13 1

MSEPF – � Color experiments cue Vs. Color cue & Motion cue: 13 2

MSEPF – experiments � Demos: 13 3

MSEPF – experiments � https: //www. youtube. com/watch? v=s 9 pd 05 Me 3 es 13 4

Summary �Particle filter algorithm - stochastic and model-driven. �MSEPF – combine MS and PF to get better tracking results. 13 5

ref o Michael Isard and Andrew Blake, CONDENSATION -- conditional density propagation for visual tracking Int. J. Computer Vision, 29, 1, 5 --28, (1998) http: //www. stat. ucla. edu/~sczhu/Courses/UCLA/Stat_202 C/lecture_note/Particle_filt ering_Isard_1998. pdf o Particle filtering: http: //www. comp. nus. edu. sg/~cs 6240/lecture/particle. pdf o o o Fast Multiple Object Tracking via a Hierarchical Particle Filter by Changjiang Yang, Ramani Duraiswami and Larry Davis: http: //www. umiacs. umd. edu/~ramani/pubs/YDD_ICCV 05_HPF. pdf http: //citeseerx. ist. psu. edu/viewdoc/download? doi=10. 1. 1. 123. 3867&rep=rep 1&type =pdf o http: //citeseerx. ist. psu. edu/viewdoc/download? doi=10. 1. 1. 331. 6104&rep=rep 1&typ e=pdf o https: //www. youtube. com/watch? v=s 9 pd 05 Me 3 es o http: //www. matmidia. mat. puc-rio. br/sibgrapi/media/posters/59928. pdf