Segmentation and Boundary Detection Using Multiscale Intensity Measurements

Segmentation and Boundary Detection Using Multiscale Intensity Measurements Eitan Sharon, Meirav Galun, Ronen Basri, Achi Brandt Dept. of Computer Science and Applied Mathematics The Weizmann Institute of Science Eitan Sharon - Weizmann Institute

Image Segmentation Eitan Sharon - Weizmann Institute

Local Uncertainty Eitan Sharon - Weizmann Institute

Global Certainty Eitan Sharon - Weizmann Institute

Local Uncertainty Eitan Sharon - Weizmann Institute

Global Certainty Eitan Sharon - Weizmann Institute

Coarse Measurements for Texture Eitan Sharon - Weizmann Institute

A Chicken and Egg Problem: Coarse measurements mix neighboring statistics Solution: support of measurements is determined as the segmentation process proceeds Eitan Sharon - Weizmann Institute

Segmentation by Weighted Aggregation q Normalized-cuts measure in graphs q Complete hierarchy in linear time q Use multiscale measures of intensity, texture, shape, and boundary integrity Eitan Sharon - Weizmann Institute

Segmentation by Weighted Aggregation q Normalized-cuts measure in graphs q Complete hierarchy in linear time q Use multiscale measures of intensity, texture, shape and boundary integrity Eitan Sharon - Weizmann Institute

The Pixel Graph Couplings Reflect intensity similarity Low contrast – strong coupling High contrast – weak coupling Eitan Sharon - Weizmann Institute

Hierarchical Graph Eitan Sharon - Weizmann Institute

Hierarchy in SWA Eitan Sharon - Weizmann Institute

Normalized-Cut Measure Eitan Sharon - Weizmann Institute

Normalized-Cut Measure Minimize: Eitan Sharon - Weizmann Institute

Normalized-Cut Measure High-energy cut Minimize: Eitan Sharon - Weizmann Institute

Normalized-Cut Measure Low-energy cut Minimize: Eitan Sharon - Weizmann Institute

Recursive Coarsening Eitan Sharon - Weizmann Institute

Recursive Coarsening Representative subset Eitan Sharon - Weizmann Institute

Recursive Coarsening For a salient segment : , sparse interpolation matrix Eitan Sharon - Weizmann Institute

Weighted Aggregation aggregate Eitan Sharon - Weizmann Institute

Segment Detection Eitan Sharon - Weizmann Institute

SWA Detects the salient segments Hierarchical structure Linear in # of points (a few dozen operations per point) Eitan Sharon - Weizmann Institute

Coarse-Scale Measurements • Average intensities of aggregates • Multiscale intensity-variances of aggregates • Multiscale shape-moments of aggregates • Boundary alignment between aggregates Eitan Sharon - Weizmann Institute

Adaptive vs. Rigid Measurements Original Our algorithm - SWA Averaging Geometric Eitan Sharon - Weizmann Institute

Adaptive vs. Rigid Measurements Original Our algorithm - SWA Interpolation Geometric Eitan Sharon - Weizmann Institute

Recursive Measurements: Intensity aggregate intensity of pixel i average intensity of aggregate Eitan Sharon - Weizmann Institute

Use Averages to Modify the Graph Eitan Sharon - Weizmann Institute

Texture Examples Eitan Sharon - Weizmann Institute

Isotropic and Oriented Filters A brief tutorial Textons by K-Means Malik et al IJCV 2001 Eitan Sharon - Weizmann Institute

Isotropic Texture in SWA Intensity Variance Isotropic Texture of aggregate – average of variances in all scales Eitan Sharon - Weizmann Institute

Oriented Texture in SWA with Meirav Galun Shape Moments • center of mass • width • length • orientation Oriented Texture of aggregate – orientation, width and length in all scales Eitan Sharon - Weizmann Institute

Gestalt – Perceptual Grouping Shashua and Ullman ICCV 1988: A brief Tutorial Group curves by: • Proximity • Co-linearity Sharon, Brandt, Basri PAMI 2000: Eitan Sharon - Weizmann Institute

Boundary Integrity in SWA Eitan Sharon - Weizmann Institute

Sharpen the Aggregates Top-down Sharpening: • Expand core • Sharpen boundaries Eitan Sharon - Weizmann Institute

Hierarchy in SWA Eitan Sharon - Weizmann Institute

Experiments images on a pentium III 1000 MHz PC: • Our SWA algorithm (CVPR’ 00 + CVPR’ 01) run-time: 5 -10 seconds. • Normalized cuts (Shi and Malik, PAMI’ 00; Malik et al. , IJCV’ 01) run-time: about 10 -15 minutes. Software courtesy of Doron Tal, UC Berkeley. Eitan Sharon - Weizmann Institute

Isotropic Texture - Horse I Our Algorithm (SWA) Normalized Cuts Eitan Sharon - Weizmann Institute

Isotropic Texture - Horse II Our Algorithm (SWA) Normalized Cuts Eitan Sharon - Weizmann Institute

Isotropic Texture - Tiger Our Algorithm (SWA) Normalized Cuts Eitan Sharon - Weizmann Institute

Isotropic Texture - Butterfly Our Algorithm (SWA) Normalized Cuts Eitan Sharon - Weizmann Institute

Isotropic Texture - Leopard Our Algorithm (SWA) Eitan Sharon - Weizmann Institute

Isotropic Texture - Dalmatian Dog Our Algorithm (SWA) Eitan Sharon - Weizmann Institute

Isotropic Texture - Squirrel Our Algorithm (SWA) Normalized Cuts Eitan Sharon - Weizmann Institute

Full Texture - Squirrel Our Algorithm (SWA) Normalized Cuts with Meirav Galun Eitan Sharon - Weizmann Institute

Full Texture - Composition Our Algorithm (SWA) with Meirav Galun Eitan Sharon - Weizmann Institute

Full Texture – Lion Cub Our Algorithm (SWA) with Meirav Galun Eitan Sharon - Weizmann Institute

Full Texture – Polar Bear Our Algorithm (SWA) with Meirav Galun Eitan Sharon - Weizmann Institute

Full Texture – Penguin Our Algorithm (SWA) with Meirav Galun Eitan Sharon - Weizmann Institute

Full Texture –Leopard Our Algorithm (SWA) with Meirav Galun Eitan Sharon - Weizmann Institute

Full Texture - Zebra Our Algorithm (SWA) with Meirav Galun Eitan Sharon - Weizmann Institute

Segmentation by Weighted Aggregation • Efficient approximation to Ncut-like measures • Recursive computation of multiscale measurements • Novel adaptive pyramid representing the image Eitan Sharon - Weizmann Institute

Matching Experiment (Chen Brestel) Eitan Sharon - Weizmann Institute

Experiments – Clustering Silhouettes Data from: Gdalyahu, Weinshall, Werman – CVPR 99 (Also: Domany, Blatt, Gdalyahu, Weinshall) Eitan Sharon - Weizmann Institute

Descending order of prominence – objects are fully categorized most prominent less prominent Eitan Sharon - Weizmann Institute

Bottom-Up and Top-Down Eitan Sharon - Weizmann Institute