Spectral Matting Anat Levin 1 2 Alex RavAcha

Spectral Matting Anat Levin 1, 2 Alex Rav-Acha 1 Dani Lischinski 1 1 School of CS&Eng The Hebrew University 2 CSAIL MIT

Hard segmentation and matting Hard segmentation compositing Source image matte compositing

Previous approaches to segmentation and matting Input Hard output Matte output

Previous approaches to segmentation and matting Input Hard output Unsupervised Spectral segmentation: Shi and Malik 97 Yu and Shi 03 Weiss 99 Ng et al 01 Zelnik and Perona 05 Tolliver and Miller 06 Matte output

Previous approaches to segmentation and matting Input Hard output Unsupervised Supervised July and Boykov 01 Rother et al 04 Li et al 04 Matte output

Previous approaches to segmentation and matting Input Hard output Matte output Unsupervised Supervised Trimap interface: interface Bayesian Matting (Chuang et al 01) Poisson Matting (Sun et al 04) Random Walk (Grady et al 05) Scribbles interface: Wang&Cohen 05 Levin et al 06 Easy matting (Guan et al 06)

Previous approaches to segmentation and matting Input Unsupervised Supervised Hard output Matte output ?

Unsupervised matting Input Automatically computed hard segments (Yu and Shi 03) Automatically computed matting components

Using components Building foreground object by simple components addition + + =

Generalized compositing equation 2 layers compositing = x + x

Generalized compositing equation 2 layers compositing x + x = x + x + x = K layers compositing Matting components

Generalized compositing equation K layers compositing = x + x + x Matting components: “Sparse” layers- 0/1 for most image pixels

Goals: • Automatically extract matting components from an image • Derive analogy between hard spectral segmentation and matting, and use similar tools. • Use matting components to automate matte extraction process and suggest new modes of user interaction

Spectral segmentation: Analyzing smallest eigenvectors of a graph Laplacian L E. g. : Shi and Malik 97 Yu and Shi 03 Weiss 99 Ng et al 01 Maila and shi 01 Zelnik and Perona 05 Tolliver and Miller 06

Spectral segmentation Fully separated classes: class indicator vectors belong to Laplacian nullspace General case: class indicators approximated as linear combinations of smallest eigenvectors Null Binary indicating vectors Laplacian matrix

Spectral segmentation Fully separated classes: class indicator vectors belong to Laplacian nullspace General case: class indicators approximated as linear combinations of smallest eigenvectors Smallest eigenvectors- class indicators only up to linear transformation Zero eigenvectors Laplacian matrix Smallest eigenvectors Binary indicating vectors Linear transformation

The matting Laplacian (Levin, Lischinski and Weiss CVPR 06) • • semidefinite sparse matrix local function of the image:

The matting Laplacian and user constrains Levin et al CVPR 06 Input: Image+ user scribbles

The matting Laplacian and user constrains Levin et al CVPR 06 Input: Image+ user scribbles Our goal: • Matting components from matting Laplacian- without user input • Build on hard spectral segmentation ideas

Matting components and the matting Laplacian Claim: • For an image consisting of “well separated” layers, the matting components belong to the matting Laplacian nullspace • In the general case, matting components are reasonably approximated as linear combinations of smallest eigenvectors Null Matting components Matting Laplacian

From eigenvectors to matting components linear transformation

Hard segmentation- matting analogy Traditional Laplacian Matting Laplacian Smallest eigenvectors Linear transformation Binary class indicators Continuous matting components

From eigenvectors to matting components 1) Initialization: projection of hard segments Smallest eigenvectors K-means Projection into eigs space 2) Non linear optimization for sparse components

Components with the scribble interface Components Levin et al cvpr 06 (our approach) Wang&Cohen 05 Random Walk Poisson

Components with the scribble interface Components Levin et al cvpr 06 (our approach) Wang&Cohen 05 Random Walk Poisson

Direct component picking interface Building foreground object by simple components addition + + =

Limitations Need to set number of components: Too few - may not contain desired matte Too many - complicates computation and user interaction Cluttered images require a large number of components Input Ground truth matte 70 eigs approximation

Conclusions • Derived analogy between hard spectral segmentation to image matting • Automatically extract matting components from eigenvectors • Automate matte extraction process and suggest new modes of user interaction Ground truth data and code available online: vision. huji. ac. il/Spectral. Matting + + =