Alon Rubin Shira Kritchman Present Realtime Background Cut

Alon Rubin Shira Kritchman Present: Real-time Background Cut 7. 5. 2006, Weizmann Institute of Science

The Challenge Real-time bilayer segmentation of video

The Challenge • Real-time bilayer segmentation of video – Fully automatic – High quality – Robustness to changing background – And yet – efficient!

Application Background substitution in video conferencing • Privacy • Coolness

Example

Overview • General method • 2 binocular (stereo) algorithms • 2 monocular algorithms

How to Segment?

How to Segment? • Information – Colour – Contrast – Disparity – Motion

How to Segment? • Information • Prior assumptions – Spatial coherence – Temporal coherence

Notation How to Segment? • Information - point in a 3 D colour space • • Prior assumptions - segmentation label • - set of neighbours F/B

Combining Cues Need: mathematical formulation • Prior assumptions – Spatial coherence – Temporal coherence – Priors over disparity • Informative cues – Colour – Contrast – Stereo – Motion ?

x - labels I - data Combining Cues • Probabilistic framework: – Maximizing • Bayes’ law: • Gibbs energy Prior Constant. Likelihood – Maximizing probability Minimizing – Dynamic programming / Graph cut energy

General energy: Spatial coherence + Contrast Colour

Modeling Colour • Global VS Local • Initial VS Dynamic Lots of blue! This is white!

Modeling Colour – Globaly – Histograms • Overlearning – GMM’s (Gaussian Mixture Models) • Number of components • Learning: EM (iterations) – Initialization parameters – Stopping condition – Time consuming

General energy: Spatial coherence + Contrast Colour

Modeling Contrast and Spatial Coherence • Spatial coherence • Contrast – inhibits the penalty 4 22 pixels, 72 maps penalty Segmentation Black: Foreground White: Background 3 22 pixels, 21 penalty

Algorithms Review Stereo Bilayer Segmentation "Probabilistic fusion of stereo with color and contrast for bi-layer segmentation", V. Kolmogorov et al. , CVPR 2005. Represents two stereo algorithms: – LDP Layered Dynamic Programming – LGC Layered Graph Cut Background Cut "Background Cut", J. Sun et al. , ECCV 2006, to appear Temporal Bilayer Segmentation "Bilayer Segmentation of Live Video", A. Criminisi et al. , CVPR 2006, to appear

Stereo Bilayer Segmentation Information: – Colour – Contrast – Stereo Prior: – Spatial coherence – Disparity-labeling relations Stereo Bilayer Segmentation

Notations F – foreground B – background O – Occluded – disparity vector Stereo Bilayer Segmentation

Stereo Bilayer Segmentation • Want to find Stereo Bilayer Segmentation

Stereo Bilayer Segmentation • Want to find This is intractable! Stereo Bilayer Segmentation

Dealing with Intractability • LDP – Layered Dynamic Programming – Defining a similar problem – Separating to scanlines – Solving with dynamic Stereo Bilayer Segmentation programming

Dealing with Intractability • LGC – Layered Graph Cut – Relaxing some dependencies on disparity – Marginalizing over – Solving with graph Stereo Bilayer Segmentation cut

Energy Function • Define a Gibbs energy • Model it as Prior: Spatial Coherence + contrast Likelihood: Matching Stereo Bilayer Segmentation Likelihood: Colour

The Prior V • Sum of binary and unary potentials: • F: – Spatial coherence – Contrast dependency Stereo Bilayer Segmentation

The Prior V Recall: • F: – F, B, O – Sophisticated switch Stereo Bilayer Segmentation

The Prior V Recall: • V*: – e=0 same equation, – e=1 dilution, – e=0 no use of contrast, Stereo Bilayer Segmentation

The Prior V • Sum of unary and binary potentials: • G: – Higher disparities in foreground – Based on a threshold – Uniform penalty Stereo Bilayer Segmentation

Likelihood for Matching • Distinguish Matched (F, B) from Occluded (O) • Determine disparity • Model as • - balance between occlusion and bad matches Stereo Bilayer Segmentation

Likelihood for Matching N – measures quality of match between patches – Classical SSD: Additive + Multiplicative normalization Robustness – NSSD: Stereo Bilayer Segmentation

Likelihood for Matching Balance between occlusion and bad matches: Preference for occlusion Stereo Bilayer Segmentation

Likelihood for Colour • GMM’s for Foreground and Background – 20 mixture componenets • Learn from previous frames • Learn using EM – 10 iterations Stereo Bilayer Segmentation

Likelihood for Colour • Model as: • Too strong Balancing factor Stereo Bilayer Segmentation

Fusion of Cues Colour Stereo Bilayer Segmentation Stereo Fusion

LDP Layered Dynamic Programming Want: separation to scanlines Recall: Prior: Spatial Coherence + contrast • V – Sum on neighbouring pixels • Use only horizontal cliques Work on scanlines Stereo Bilayer Segmentation - LDP

LDP • Classical DP Stereo Bilayer Segmentation - LDP Diagonal: matched Vertical: occluded Horizontal: occluded

LDP • Layered DP Stereo Bilayer Segmentation - LDP Alternates between matched and occluded!

LDP • Layered DP Stereo Bilayer Segmentation - LDP The whole line is matched!

LDP • Layered DP No diagonal moves Vertical: matched or occluded Horizontal: matched or occluded Stereo Bilayer Segmentation - LDP

LDP • 4 -State Space • Many parameters: • Learn parameters from labeled data a –a 0 viewing , c=- mean normalization geometry width of considerations matched region = mean width of occluded region Stereo Bilayer Segmentation - LDP

LDP 6 -State Space Solve with dynamic programming! Stereo Bilayer Segmentation - LDP

LGC – Layered Graph Cut • Does not solve for disparity • Minimizes • Marginalize over disparities: X Stereo Bilayer Segmentation - LGC

LGC • Expansion move algorithm Stereo Bilayer Segmentation - LGC

LGC • Expansion move algorithm with savings: – Only 3 Labels – Only 2 iterations: • Initialize with B for all pixels • Run F-expansion • Run O-expansion on a constrained Stereo Bilayer Segmentation - LGC region

Results LDP Stereo Bilayer Segmentation LGC

Results (LGC) Stereo Bilayer Segmentation

Results – Errors Stereo Bilayer Segmentation

Quantitative Results • Hand labeled ground truth (any 5 th/10 th frame) • Percentage of misclassified Stereo Bilayer Segmentation

Quantitative Results Stereo Bilayer Segmentation

Quantitative Results Computation times: Around 10 fps at 320 X 240 resolution On a conventional 3 GHz processor Stereo Bilayer Segmentation

Results Stereo Bilayer Segmentation

Stereo Segmentation – Summary • 2 algorithms: LGC and LDP • Require binocular configuration • Temporal relations are implicit • Stereo cues are very strong Stereo Bilayer Segmentation

Algorithms Review Stereo Bilayer Segmentation "Probabilistic fusion of stereo with color and contrast for bi-layer segmentation", V. Kolmogorov et al. , CVPR 2005. Represents two stereo algorithms: – LDP Layered Dynamic Programming – LGC Layered Graph Cut Background Cut "Background Cut", J. Sun et al. , ECCV 2006, to appear Temporal Bilayer Segmentation "Bilayer Segmentation of Live Video", A. Criminisi et al. , CVPR 2006, to appear

Background Cut Information: – Colour – Contrast – Initialization phase Prior: – Spatial coherence Background Cut

Most Efficient Approach: Background Subtraction - Background Cut =

Most Efficient Approach: Background Subtraction Problems: Foreground-Background similarity Sensitive threshold Background Cut

Background Cut (Minimize by min-cut) • Spatial coherence • Colour model Ø Background maintenance Background Cut

Basic Model – Colour Term Background: global and local • Global: GMM model Background Cut

Basic Model – Colour Term Background: global and local • Global: GMM model • Local: single Gaussian t Background Cut

Basic Model – Colour Term Background: global and local • Global: GMM model • Local: single gaussian • Combination: Background Cut

Basic Model – Colour Term Foreground colour model 5 components GMM Background Cut

Basic Model – Colour Term Background Cut

Colour Term Adaptive mixture global-local colour model Background Cut

Colour Term Adaptive mixture global-local colour model How can we quantify the difference? Kullback-Liebler divergence Background Cut

Colour Term Kullback-Liebler divergence quantify the difference between two GMM’s Background Cut

Colour Term Only Global Equally Local and Global Background Cut

Colour Term – Summary Background Cut

Basic Model – Contrast Term Penalty term + Penalty inhibition Background Cut

Contrast Term Background contrast attenuation Background Cut

Contrast Term Foreground boundaries Background contrast Clues: • Comparison to original background contrast Background Cut

Contrast Term Over attenuation of boundaries! Background Cut

Contrast Term Clues: • Comparison to original background contrast • Difference from original background Background Cut

Contrast Term – Summary +Background contrast Background Cut +Background contrast +Background colour

Contrast Term – Summary Background Cut

Background Maintenance Sudden illuminance change • Auto gain control • Fluorescent lamps • Light switching Background Cut

Background Maintenance Minor change • Histogram transformation function Major change • Colour model rebuilding Background Cut

Background Maintenance Colour model rebuilding • Foreground threshold increasing • Background uncertainty map initialization • Mixture model modification • Dynamic updating of Background Cut and

Background Maintenance • Movement in the background - Relying on global model • Sleeping and waking objects - Keeping biggest connected component • Casual camera shaking - Background maintenance - Appling Gaussian blurring - Using less local colour model Background Cut

Background Maintenance Background Cut

Quantitative Results Background Cut

Quantitative Results Computation times: Around 12 -15 fps at 320 X 240 resolution On a conventional 3. 2 GHz processor Background Cut

Background Cut – Summary • Adaptive mixture global-local colour model • Background contrast attenuation • Background Maintenance Background Cut

Algorithms Review Stereo Bilayer Segmentation "Probabilistic fusion of stereo with color and contrast for bi-layer segmentation", V. Kolmogorov et al. , CVPR 2005. Represents two stereo algorithms: – LDP Layered Dynamic Programming – LGC Layered Graph Cut Background Cut "Background Cut", J. Sun et al. , ECCV 2006, to appear Temporal Bilayer Segmentation "Bilayer Segmentation of Live Video", A. Criminisi et al. , CVPR 2006, to appear

Temporal Bilayer Segmentation Information: – Colour – Contrast – Initialization phase – Motion Prior: – Spatial coherence – Temporal coherence Temporal Bilayer Segmentation

Motion – Notations Basic image features (YUV) Temporal Bilayer Segmentation

Temporal Bilayer Segmentation • Motion Temporal continuity Colour likelihood Spatial continuity Temporal Bilayer Segmentation

Temporal Coherence 4 pixel types: Not likely: BF B, FB F Temporal Bilayer Segmentation

Temporal Coherence 4 pixel types: Temporal Bilayer Segmentation

Spatial Coherence The usual term: Temporal Bilayer Segmentation

Likelihood for Colour GMM X – Number of mixture components – Learning: • Initialization • Convergence • Time consuming Histograms – Nonparametric – Smoothed to avoid overlearning Temporal Bilayer Segmentation

Likelihood for Colour Foreground – Learned adaptively from previous frames Background – Learned from initialization phase – Static over time – Only global (Claim: local doesn’t improve much) Temporal Bilayer Segmentation

Likelihood for Motion X Optical flow – Inaccuracies along boundaries – The aperture problem – Expensive Motion/Non-motion classifier – Adaptive – Efficient Temporal Bilayer Segmentation

Motion Classifier Basic features: ? Temporal Bilayer Segmentation

Motion Classifier X-axis: Grad(I) Y-axis: . I Temporal Bilayer Segmentation !

Testing the Motion Classifier Not sufficient: Must fill in the gaps Temporal Bilayer Segmentation

Minimizing the Energy Want: Instead: Temporal Bilayer Segmentation Allowing for changes in t-1

Minimizing the Energy Instead: Temporal Bilayer Segmentation

Minimizing the Energy Now minimize using Graph Cut Instead: Temporal Bilayer Segmentation

Results Temporal Bilayer Segmentation

Quantitative Results • Hand labeled ground truth (any 5 th/10 th frame) • Percentage of misclassified Temporal Bilayer Segmentation

Limitations High illuminance changes Failure (2/6 seq’s) Stereo V Monocular X Recommend: switch off Auto Gain Control Temporal Bilayer Segmentation

Summary LDP LGC Background Cut Temporal Bilayer Seg. V V Colour Model GMM’s Histograms Background maintenance V V V – Explicit Implicit Background Attenuation – – – V – – Motion – – – V V Colour/ Contrast Disparities Temporal Coherence

Another Approach to Background Substituition

Thank You! A special thank to Dr. Vladimir Kolmogorov and to Eli Shechtman for their assitatnce 7. 5. 2006, Weizmann Institute of Science

Alon Rubin Shira Kritchman F. A. Q. 7. 5. 2006, Weizmann Institute of Science

Likelihood for Matching Empirical test – Is N discriminative? – Take labeled data – Compute and discretize N – Count matched pixels for each N – Count occluded pixels for each N – Divide Get: Likelihood ratio of matching as a function of N Stereo Bilayer Segmentation

Likelihood for Matching Example: Take N=0. 1 – 15% of matched pixels have N=0. 1 – 5% of occluded pixels have N=0. 1 => likelihood ratio for N=0. 1 Get: Likelihood ratio of matching as a function of N Stereo Bilayer Segmentation