Multiscale Combinatorial Grouping for Image Segmentation and Object

Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation Student: YEH, HAO-WEI Advisor: Sheng-Jyh, Wang

References • P. Arbelaez, M. Maire, C. Fowlkes, J. Malik. “Contour Detection and Hierarchical image Segmentation, ” IEEE Trans. on PAMI , 2011. • Also slides from Hsin-Min, Cheng • X. Ren and L. Bo, “Discriminatively trained sparse code gradients for contour detection, ” in NIPS, 2012. • P. Doll´ar and C. Zitnick, “Structured forests for fast edge detection, ” ICCV, 2013. • Also Slides by Piotr Dollár • P. Arbelaez, J. Pont-Tuset, Jonathan T. Barron, F. Marques, and J. Malik. “Multiscale combinatorial grouping, ” In CVPR, 2014. • Jordi Pont-Tuset, Pablo Arbelaez, Jonathan T. Barron, Ferran Marques, Jitendra Malik, “Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation, ” ar. Xiv 2015

Outline • Introduction to Image Segmentation & Object Generation • Workflow of the paper • Single-scale Segmentation • UCM in PAMI 2010 • Modifications in CVPR 2014, ar. Xiv 2015 • Rescaling & Alignment • Combinatorial grouping • Experimental Results

Introduction

Image Segmentation vs. Contour Detection Contour Original Image Segmentation

Hierarchical Image Segmentation

Object Generation

Object Generation

Workflow

Workflow (CVPR 2014, ar. Xiv 2015)

Workflow (CVPR 2014, ar. Xiv 2015)

g. Pb-owt-ucm (PAMI 2011) Original Image Local cues Global cues Oriented Gradient of histograms Contour (g. Pb) • Oriented Watershed Transform (OWT) • Iteratively Merging Hierarchical Image Segmentation (UCM)

Local Pb (m. Pb) Example • Local Cues • Brightness, Color • L*a*b* color space • Texture • Textons with 17 filters Filters for creating textons Gradient magnitude G at location(x, y) • Method • Oriented gradient of histograms • Three scales of r

ure

Spectral Pb (s. Pb) • Global Cues • Weighted graph G=(V, E) from image • V: image pixels • E: connections between pairs of nearby pixels => • Method • Normalized Cuts (Spectral Clustering) U. v. Luxburg, “A tutorial on spectral clustering, ” Statistics and Computing, vol. 17, no. 4, pp. 395– 416, 2007. http: //www. informatik. uni-hamburg. de/ML/contents/people/luxburg/publications/Luxburg 07_tutorial. pdf

Spectral Pb (s. Pb)

Globalized Pb (g. Pb) Local Cues Global cues

g. Pb-owt-ucm (PAMI 2011) Original Image Local cues Global cues Oriented Gradient of histograms Contour (g. Pb) • Oriented Watershed Transform (OWT) • Iteratively Merging Hierarchical Image Segmentation (UCM)

Hierarchical Image Segmentation

Hierarchical Segmentation • From contour to segments (OWT) • Hierarchical Segmentation by iteratively merging the regions (UCM)

Hierarchical Segmentation • Oriented Watershed transform • Approximate the watershed arcs with line segments • Use the orientation of the arcs o(x, y) OWT WT

Hierarchical Segmentation • Hierarchical Segmentation by iteratively merging the regions (UCM)

g. Pb-owt-ucm (PAMI 2011) Original Image Local cues Global cues Oriented Gradient of histograms Contour (g. Pb) • Oriented Watershed Transform (OWT) • Iteratively Merging Hierarchical Image Segmentation (UCM)

UCM vs. Region-Tree

Experimental Results(PAMI 2011)

Experimental Results(PAMI 2011)

Single-scale Segmentation • Modifications • Different local cues • Brightness, color, texture differences as in PAMI 2010 • Sparse coding on patches • Structured forest contour • Faster eigenvector computation • Similarity of the affinity matrix in different scales

Sparse Coding Patches X. Ren and L. Bo, “Discriminatively trained sparse code gradients for contour detection, ” in NIPS, 2012.

Structured Forest Detector P. Doll´ar and C. Zitnick, “Structured forests for fast edge detection, ” ICCV, 2013.

Structured Forest Detector P. Doll´ar and C. Zitnick, “Structured forests for fast edge detection, ” ICCV, 2013.

Workflow (CVPR 2014, ar. Xiv 2015)

Rescaling & Alignment • Align every UCM to the original scale • Problem : How to upsample/downsample the contour? • Solution : Projection of the corresponding segmentations

Workflow (CVPR 2014, ar. Xiv 2015)

Object Generation

Combinatorial Grouping • • Choose object proposals(combination of regions) from UCM Method: Supervised Learning Problem: Exhaustive search is intractable!! Solution: • Pareto Optimization • Regressed Ranking

Experimental Results(ar. Xiv 2015)

Experimental Results(ar. Xiv 2015)

Experimental Results(ar. Xiv 2015)

Experimental Results(ar. Xiv 2015)

Experimental Results(ar. Xiv 2015)

Experimental Results(ar. Xiv 2015)

Experimental Results(ar. Xiv 2015)

Workflow (CVPR 2014, ar. Xiv 2015)

Thank you for your attentions!!