Computer Vision Motion Extraction Computer Vision Motion is

Computer Vision Motion Extraction

Computer Vision Motion is a basic cue Motion can be the only cue for segmentation Biologically favoured because of camouflage

Computer Vision Motion is a basic cue … which set in motion a constant, evolutionary race, encouraging animals to sit still, only moving in bursts

Computer Vision Motion is a basic cue Motion can be the only cue for segmentation

Computer Vision Motion is a basic cue Even impoverished motion data can elicit a strong percept http: //www. biomotionlab. ca/Demos/BMLwalker. html

Computer Vision Some applications of motion extraction o Change / shot cut detection o Surveillance / traffic monitoring o Autonomous driving o Analyzing game dynamics in sports o Motion capture / gesture analysis (HCI) o Image stabilisation o Motion compensation (e. g. medical robotics) o Feature tracking for 3 D reconstruction o Etc. !

Computer Vision Shot cut detection & Keyframes Shot cut

Computer Vision Human-Machine Interfacing

Computer Vision 3 D: Structure-from-Motion Tracked Points gives correspondences

Computer Vision 3 D: Structure-from-Motion Temple of the Masks, Edzna, Mexico

Computer Vision www. arc 3 d. b

Computer Vision in this lecture. . . Several techniques, but… this lecture is restricted to the 1. detection of the “optical flow” 2. tracking with the “Condensation filter”

Computer Vision optical flow

Computer Vision Definition of optical flow Ideally, the optical flow is the projection of the 3 D velocity vectors onto the image In practice OF = apparent motion of brightness patterns A motion vector is sought at every pixel of the image

Computer Vision OF may fail to capture the real motion ! Two examples : 1. Uniform, rotating sphere ⇓ O. F. = 0 2. No motion, but changing lighting ⇓ O. F. ≠ 0

Computer Vision Caution required !

Computer Vision Qualitative formulation Suppose a point of the scene projects to a certain pixel of the current video frame. Our task is to figure out to which pixel in the next frame it moves… That question needs answering for all pixels of the current image. In order to find these corresponding pixels, we need to come up with a reasonable assumption on how we can detect them among the many. We assume these corresponding pixels have the same intensities as the pixels the scene points came from in the previous frame. That will only hold approximately…

Computer Vision Mathematical formulation I (x, y, t) = brightness at (x, y) at time t Optical flow constraint equation : This equation states that if one were to track the image projections of a physical point through the video, it would not change its intensity… This tends to be true over short lapses of time. (Note the different types of time derivatives!)

Computer Vision Mathematical formulation We will use as shorthand notation for

Computer Vision The aperture problem Note that we can measure the 3 derivatives of I, but that u and v are unknown 1 equation in 2 unknowns

Computer Vision The aperture problem Aperture problem : only the component along the gradient can be retrieved

Computer Vision The aperture problem

Computer Vision Remarks

Computer Vision Remarks 1. The underdetermined nature could be solved using higher derivatives of intensity 2. for some intensity patterns, e. g. patches with a planar intensity profile, the aperture problem cannot be resolved anyway. For many images, large parts have planar intensity profiles… higher-order derivatives than 1 st order are typically not used (also because they are noisy)

Computer Vision Horn & Schunck algorithm Breaking the spell via an … additional smoothness constraint : to be minimized, besides the OF constraint equation term minimize es+λec (also reduces influence of noise)

Computer Vision The calculus of variations look for functions that extremize functionals (a functional is a function that takes a vector as its input argument, and returns a scalar) like for our functional: what are the optimal u(x, y) and v(x, y) ?

Computer Vision The calculus of variations look for functions that extremize functionals with and ,

Computer Vision Calculus of variations Suppose 1. f(x) is a solution 2. η(x) is a test function with η(x 1)= 0 and η(x 2) = 0 We then consider Rationale: supposed f is the solution, then any deviation should result in a worse I; when applying classical optimization over the values of the scalar e the optimum should be e = 0

Computer Vision Calculus of variations Suppose 1. f(x) is a solution 2. η(x) is a test function with η(x 1)= 0 and η(x 2) = 0 We then consider With this trick, we reformulate an optimization over a function into a classical optimization over a scalar… a problem we know how to solve

Computer Vision Calculus of variations Suppose 1. f(x) is a solution 2. η(x) is a test function with η(x 1)= 0 and η(x 2) = 0 for the optimum : Around the optimum, the derivative should be zero

Computer Vision Calculus of variations Suppose 1. f(x) is a solution 2. η(x) is a test function with η(x 1)= 0 and η(x 2) = 0 for the optimum : with

Computer Vision Calculus of variations Using integration by parts: Using integration by parts : where

Computer Vision Calculus of variations Using integration by parts :

Computer Vision Calculus of variations Using integration by parts Therefore regardless of η(x), then Euler-Lagrange equation :

Computer Vision Calculus of variations Generalizations n 1. Simultaneous Euler-Lagrange equations, i. c. one for u and one for v : n 2. 2 independent variables x and y

Computer Vision Hence Calculus of variations Now by Gauss’s integral theorem, such that =0

Computer Vision Calculus of variations

Computer Vision Calculus of variations Consequently, for all test functions η , thus is the Euler-Lagrange equation

Computer Vision Horn & Schunck The Euler-Lagrange equations : In our case , so the Euler-Lagrange equations are is the Laplacian operator

Computer Vision Horn & Schunck Remarks : 1. Coupled PDEs solved using iterative methods and finite differences (iteration i) i i 2. More than two frames allow for a better estimation of It 3. Information spreads from edge- and corner-type patterns

Computer Vision

Computer Vision Horn & Schunck, remarks 1. Errors at object boundaries 2. Example of regularisation (selection principle for the solution of ill-posed problems by imposing an extra generic constraint, like here smoothness)

Computer Vision condensation filter

Computer Vision condensation filter as an example of a `tracker’, shifting the emphasis from pixels to objects…

Computer Vision Condensation tracker state vector noise in system model observation vector noise in measurement model System. M. Measur. M.

Computer Vision Condensation tracker 1. Prediction , based on the system model ( f = system transition function) 2. Update , based on the measurement model ( h = measurement function) is the history of observations

Computer Vision Condensation tracker Example System model position velocity Measurement model

Computer Vision Condensation tracker Recursive Bayesian filter Object not as a single state but a prob. distribution

Computer Vision Condensation tracker Recursive Bayesian filter Object not as a single state but a prob. Distribution (p here means probability…) 1. Prediction 2. Update

Computer Vision Condensation tracker Recursive Bayesian filter Object not as a single state but a prob. distribution 1. Prediction 2. Update can be considered a normalization factor

Computer Vision Condensation tracker Recursive Bayesian filter Object not as a single state but a prob. distribution Bayes´ rule 2. Update can be considered a normalization factor

Computer Vision Condensation tracker Recursive Bayesian filter Object not as a single state but a prob. distribution Bayes´ rule 2. Update can be considered a normalization factor

Computer Vision Condensation tracker Recursive Bayesian filter Object not as a single state but a prob. distribution 1. Prediction 2. Update

Condensation tracker Computer Vision Example cont’d a posteriori prob. distr. at t-1 prediction a priori prob. distr. at t update

Computer Vision Condensation tracker Recursive Bayesian filter Calculating numerically is very time consuming, and the prob. distributions have to be known… Analytic solutions are only available for the simplest of cases, e. g. when distr. are Gaussian and the system and measurement models are linear… (Kalman filter, 1960 - Kalman was prof. at ETH, D-ITET)

Computer Vision Condensation tracker Recursive Bayesian filter Calculating numerically is very time consuming, and the prob. distributions have to be known… Analytic solutions are only available for the simplest of cases, e. g. when distr. are Gaussian and the system and measurement models are linear… That’s where CONDENSATION comes in, acronym for CONditional DENSity propag. ATION

Computer Vision In our example model is linear, distributions Gaussian K A L M model System A N F I L Measurement model T E R

Computer Vision K A L M A N F I L T E R

Computer Vision Condensation tracker The probability distribution is represented by a sample set S (set of selected states s) With a weight determining the sampling probability

Computer Vision Condensation tracker 1. prediction Start with , the sample set of the previous step, and apply the system model to each sample, yielding predicted samples 2. update Sample from the predicted set, where samples are drawn with replacement and with probability (i. e. using meas. model) In the limit (large N) equivalent to Bayesian tracker

Computer Vision Condensation tracker prediction weighing update

Computer Vision Condensation tracker NOTE Sample may be drawn multiple times, but different noise will yield different predictions under the system model, for samples corresponding to the same state after drawing. This diversification through noise is important, as otherwise fewer and fewer different samples would survive

Computer Vision Example cont’d prediction weighing with update C O N D E N S A T I O N

Computer Vision Condensation tracker Comparison with Kalman filter Condensation Kalman-Bucy Unrestricted system models Unrestricted noise models Multiple hypotheses Linear system models Gaussian noise Unimodal Discretisation error Postprocessing for interpret. Exact solution Direct interpretation

Computer Vision Condensation tracker

Computer Vision Condensation tracker

Computer Vision Condensation tracker

Computer Vision Condensation tracker

Computer Vision Condensation tracker

Computer Vision Condensation tracker

Computer Vision Condensation tracker

Computer Vision Condensation tracker

Computer Vision Condensation tracker

Computer Vision Condensation tracker

Computer Vision Condensation tracker Elliptical region with prescribed color histogram System model position velocity size chance

Computer Vision Condensation tracker Measurement model with where p and q are the color histograms of a sample and the target, resp.

Computer Vision Condensation tracker

Computer Vision Mean shift tracker

Computer Vision Mean shift tracker

Computer Vision Condensation tracker

Computer Vision Other approaches 1. Model-based tracking (application-specific) - active contours (discussed with segmentation) - analysis/synthesis schemes 2. Feature tracking (more generic) - corner tracking (shown when we discuss 3 D) - blob/contour tracking - intensity profile tracking - region tracking

Computer Vision Model-based tracker (EPFL)

Computer Vision Model-based tracker (EPFL)

Computer Vision Motion capture for special effects