OPTICAL FLOW PART 2 MOTION ESTIMATION AND OCCLUSION

OPTICAL FLOW – PART 2 #MOTION ESTIMATION AND OCCLUSION DETECTION #BLURRED VIDEO WITH LAYERS Liron Gruber

OUTLINE Local Layering for Joint Motion Estimation and Occlusion Detection (Sun 1 Liu 2 Pfister 1): • • • The problem Some layers models history Local layers model Probabilistic model Results Modeling Blurred Video with Layers (Jonas Wulff and Michael J. Black): • • The problem The two layered model Initialization and optimization Results

Joint Motion Estimation and Occlusion Detection • The problem: Most motion estimation algorithms (optical flow, layered models) cannot handle large amount of occlusion • Their solution: Local layering model where motion and occlusion relationships are inferred jointly

What we have today • Optical flow methods: -> X and T junctions problem • Layered models : 1. Number of layers 2. The layer ownership 3. Depth ordering 4. The motion for each layer

Layers ‘HISTORY’ Previous work on layers relies on either motion or color cues to initialize layer segmentation For example: 1. optical flow algorithm -> motion vectors clustering -> layer primitive (Limor’s Sun) 2. optical flow algorithms -> different features ‘random forest’ classifiers (Humayun) 3. ( estimate the motion of superpixels and the occlusion boundaries between superpixels for epipolar constrained optical flow estimation )

The Problem: [1] [2] Their method

Mid - summary • all layers methods separate motion estimation (Do not use the detected occlusion to improve optical flow) (exp 3 – does that , but not vise versa. . . ) Global layers : • limited in capturing mutual or self occlusions • often only contain a few number of layers (complexity explodes) o L – So … s r e y a L l a c

local layering model Jointly infer motion and occlusion: • superpixel representations (from over segmentation) • Each superpixel’s occlusion-relationships with its neighbors • In the inference - keeping the uncertainties of both motion and occlusion relationships motion is inferred by considering all the possibilities of local occlusion relationships and vice versa

In practice – given I 1, I 2 Unknowns: (after segmentation of I 1 to local layers) • m - movement of each layer • o - occlusion map for I 1 • R=1, 0, -1 - occlusion relationship between spatially close local layers

R

From R to Pseudo Depth (d) A: A(i , i) =| Ni | A(i, j) = -1 if j in Ni 0 otherwise b: b(i) = sum of Rij (over all j in Ni )

Probabilistic Model • Data term • Motion and occlusion • Motion prior

Probabilistic Model – Data term (intuition only…)

Probabilistic Model - Motion prior (intuition only…) • Similar to optical flow: Motion is smooth and slow Occasionally abrupt near object boundaries

Probabilistic Model - Motion and occlusion There are pixels in the set Only Make sure o is consistent with R, m : overlap No pixels in the set No overlap

Inference process

Classical /baseline optical flow methods (motion) (Average EPE) Results State-of-the-art learning based approach (occlusion)

Summary • Local layering model can handle motion and occlusion well for both challenging synthetic and real sequences (“two bars” sequence and the MPI Sintel dataset) • This method improves the baseline algorithms that provide the motion estimations and performs comparably with one learning-based occlusion detection algorithm

OUTLINE Local Layering for Joint Motion Estimation and Occlusion Detection (Sun 1 Liu 2 Pfister 1): • • • The problem Some layers models history Local layers model Probabilistic model Results Modeling Blurred Video with Layers (Jonas Wulff and Michael J. Black): • • The problem The two layered model Initialization and optimization Results

Modeling Blurred Video with Layers The problem: Videos contain complex spatially-varying motion blur due to finite shutter speeds. Existing methods (to estimate optical flow, deblur the images, and segment the scene) fail in such cases and fail specifically at object boundaries. Their solution: A novel 2 layered model of scenes in motion. Jointly estimate the layer segmentation and each layer's appearance and motion.

Example

Notation observed color image unblurred color “appearance" of layer l segmentation mask for l * assumed to be constant across the sequence * only consider opaque layers (is binary) the transformation (motion) parameters for layer l at frame t a blur matrix (s is the shutter speed)

A walk through • A Single Layer with Motion Blur • Two Layers without Motion Blur • Two Layers with Foreground Motion and Blur • ……. ->

The Two-Layer Model observed image blur matrix segmentation mask unblurred “appearance" transformation matrices (according to ) They minimized: + Regularization term

(Regularization term) Spatial smoothness: Background preference:

The generative model - example

Initialization • Good initialization is important (the choice of initial dense optical flow algorithm is not critical, they use MDP-Flow) [b] • 2 dominant motions are robustly estimated [c] Pyramids levels (large motions)

Optimization Iterative, alternating optimization method: 1. optimize one variable at a time (using gradient descent) 2. Terminate the optimization after 3 iterations (to avoid reaching local optima) and switch to the next variable • Relax the binary-valued • To deal with large motions - a Gaussian pyramid

“…the shape of the person and rims of the bicycle being evident”

Results accurate optical flow method (no layers, no blur) layered optical flow model (no blur) motion blur in an algorithm for optical flow (no layers)

Summary • They developed a principled formulation of motion blur in layers. • They jointly estimated parametric motion, deblurred appearance, and scene segmentation. • The layered model captures the blur at boundaries and by modeling the blur process one achieves better motion estimation, layer segmentation, and layer deblurring.

The end Thank you

EXTRA SLIDES (1)

Extra equations +explanations 1

Extra equations +explanations 2

Extra equations +explanations 3

Inference

EXTRA SLIDES (2)

Evaluation 2