Stereo Iterative GraphCuts Alex RavAcha Vision Course Hebrew

Stereo & Iterative Graph-Cuts Alex Rav-Acha Vision Course Hebrew University

The stereo problem

The stereo problem • Both images are very similar (like images that you see with your two eyes) • Most of the pixels in the left image are present in the right image (except for few occlusions)

The stereo problem After rectification: all correspondences are along the same horizontal scan lines (pixels in one image simply shift horizontally in the other image)

The stereo problem • The horizontal shifts between the images are sometimes called: “disparities” • The Disparities relates to depth: Closer objects have larger disparities.

The stereo problem: compute the disparity map between two images

Traditional Approaches • Matching rigid windows around each pixel • Each window is matched independently Modern approaches • Finding coherent correspondences for all pixels - “Graph cuts” - “Belief Propagation”

Window-Based Approach • Compute a cost for each location • Location with the lowest cost wins

General Problem : Ambiguity Left scanline Right

Window-Based Approach Small Window noisy in low texture areas Large Window blurred boundaries

Results with best window size (still not good enough) Window-based matching (best window size) Ground truth

Graph Cuts Graph cuts Ground truth

Maximum flow problem a flow F • Max flow problem: “source” “sink” S T A graph with two terminals – Each edge is a “pipe” – Find the largest flow F of “water” that can be sent from the “source” to the “sink” along the pipes – Edge weights give the pipe’s capacity

Minimum cut problem a cut C • Min cut problem: “source” “sink” S T A graph with two terminals – Find the cheapest way to cut the edges so that the “source” is completely separated from the “sink” – Edge weights now represent cutting “costs”

Max flow/Min cut theorem • Max Flow = Min Cut: “source” “sink” S T A graph with two terminals – Maximum flow saturates the edges along the minimum cut. – Ford and Fulkerson, 1962 – Problem reduction! • Ford and Fulkerson gave first polynomial time algorithm for globally optimal solution

Min-Cut: Important Rule No subset of the cut can also be a cut This is not a minimal cut

Energy Minimization Using Iterative Graph cuts Fast Approximate Energy Minimization via Graph Cuts Yuri Boykov, Olga Veksler and Ramin Zabih Pami 2001

To do better we need a better model of images • We can make reasonable assumptions about the surfaces in the world • Usually assume that the surfaces are smooth • Can pose the problem of finding the corresponding points as an energy (or cost) minimization: f-assignment how well the pixels match up for different disparities neighboring pixels have similar disparities

To do better we need a better model of images • We can make reasonable assumptions about the surfaces in the world • Usually assume that the surfaces are smooth • Can pose the problem of finding the corresponding points as an energy (or cost) minimization: f-assignment p, q-pixels Data term is calculated for each pixels Smoothness is calculated on neighbor pixels

Example for Smoothness terms • Quadratic • L 1 • Truncated L 1 • Potts model

Constructing a Graph to Solve the Stereo Problem

Constructing a Graph to Solve the Stereo Problem The labels of each pixel are the possible disparity values

Relation between the Energy and the Graph labeling problem {fp=10} Data term Smoothness term 10 1 p q {fq=2}

Relation between the Energy and the Graph labeling problem Dp(10) Data term Smoothness term 10 V(p, q)(1, 10) 1 p q

Iterative graph-cuts • Use an iterative scheme to find a “good” local optimum of the energy function. • In each iteration: convert the original multi-label problem to a binary one, and solve it by finding a minimal graph-cut (max-flow). • The most popular scheme is the expansion move. • -expansion: set the label of each pixel to be either or the current label.

Types of Moves A Single Pixel Move Problem: A lot of local minima

Types of Moves Expansion Move Any pixel can change its label to alpha

Types of Moves Expansion Move Claim (without proof): The difference between the optimal solution and the solution from the iterative expansion moves is bounded

Energy Minimization Algorithm 1. Start with arbitrary labeling f 2. Set success = 0 3. For each label – Find – If set 4. If success =1 goto (2) 5. Return f and success =1

Conditions on the Smoothness for using expansion moves: In other words: V should be a metric Note : The Quadratic smoothness is not a metric

For each pair of vertices such that we add a ‘dummy’ vertex (together with the respective edges as shown in the table).

The Relation between the cut and the Energy • Given a cut C, we define a labelling fc by: If the cut C separates p and • The cost of a cut C is |C| = E(f. C) (plus a constant)

The Relation between the cut and the Energy

Conditions on the Smoothness for using expansion moves: In other words: V should be a metric