JIGSAWS JOINT APPEARANCE AND SHAPE CLUSTERING John Winn

  • Slides: 21
Download presentation
JIGSAWS: JOINT APPEARANCE AND SHAPE CLUSTERING John Winn with Anitha Kannan and Carsten Rother

JIGSAWS: JOINT APPEARANCE AND SHAPE CLUSTERING John Winn with Anitha Kannan and Carsten Rother Microsoft Research, Cambridge

Patch models Used for: Object recognition/detection Object segmentation But also: Stereo matching, photo stitching

Patch models Used for: Object recognition/detection Object segmentation But also: Stereo matching, photo stitching Texture synthesis Super-resolution Motion segmentation Image/video compression

Patch models Patch clustering/codebook (e. g. Leibe & Schiele) Epitome (Jojic et al. )

Patch models Patch clustering/codebook (e. g. Leibe & Schiele) Epitome (Jojic et al. ) parameter sharing + translation invariant

Issues with fixed patch size/shape Patch includes background patches containing the same object are

Issues with fixed patch size/shape Patch includes background patches containing the same object are not clustered together Patch excludes part of object patch is less discriminative Patch includes occlusion occluded and unoccluded objects are not clustered together

Patch size? More sharing More discriminative Less sharing Size Small Large (single pixel) (entire

Patch size? More sharing More discriminative Less sharing Size Small Large (single pixel) (entire image) Optimal size/shape? Depends on: • object size/shape • object variability • size of training set

Aims of jigsaw model Learn patches (jigsaw pieces) which are 1. Shared: each piece

Aims of jigsaw model Learn patches (jigsaw pieces) which are 1. Shared: each piece is similar in shape and appearance to many regions of the training images; 2. Discriminative: each piece is as large as possible; 3. Exhaustive: all parts of the training images can be reconstructed from the set of jigsaw pieces.

The Jigsaw model Jigsaw J . . . Image I 1 Offset map L

The Jigsaw model Jigsaw J . . . Image I 1 Offset map L 1 Image I 2 Offset map L 2 Image IN Offset map LN

The Jigsaw model Jigsaw J . . . Image I 1 Offset map L

The Jigsaw model Jigsaw J . . . Image I 1 Offset map L 1 Image I 2 Offset map L 2 Image IN Offset map LN

The Jigsaw model Jigsaw J . . . Image I 1 Offset map L

The Jigsaw model Jigsaw J . . . Image I 1 Offset map L 1 Potts model: Image I 2 Offset map L 2 Image IN Offset map LN

Toy example Training image Jigsaw Learned using EM + graph cuts

Toy example Training image Jigsaw Learned using EM + graph cuts

Dog example Training image 32 32 Jigsaw mean

Dog example Training image 32 32 Jigsaw mean

Dog example Reconstructed image 32 32 Jigsaw mean Learned segmentation Epitome reconstruction

Dog example Reconstructed image 32 32 Jigsaw mean Learned segmentation Epitome reconstruction

Faces example 128 Jigsaw mean 100 64 64 images Source: Olivetti face database

Faces example 128 Jigsaw mean 100 64 64 images Source: Olivetti face database

Learning the ‘pieces’ Jigsaw J . . . Image I 1 Offset map L

Learning the ‘pieces’ Jigsaw J . . . Image I 1 Offset map L 1 Image I 2 Offset map L 2 Image IN Offset map LN

Learning the ‘pieces’ Jigsaw J . . . Image I 1 Offset map L

Learning the ‘pieces’ Jigsaw J . . . Image I 1 Offset map L 1 Image I 2 Offset map L 2 Image IN Offset map LN

Faces example Results of shape clustering on the face images

Faces example Results of shape clustering on the face images

Object recognition (preliminary) Trained set: 20 street images 64 x 64 jigsaw Allow patches

Object recognition (preliminary) Trained set: 20 street images 64 x 64 jigsaw Allow patches to deform (as in Layout. CRF, CVPR 2006).

Object recognition (preliminary) Trained set: 20 street images (10 labelled) 64 x 64 jigsaw

Object recognition (preliminary) Trained set: 20 street images (10 labelled) 64 x 64 jigsaw Allow patches to deform (as in Layout. CRF, CVPR 2006). Accuracy improves (~1%) if you include an additional 10 unlabelled images when learning the jigsaw.

Work in progress… Training larger jigsaws on 100 s of images Incorporating shape clustering

Work in progress… Training larger jigsaws on 100 s of images Incorporating shape clustering into the probabilistic model Learning additional invariances e. g. to illumination Object recognition results on MSRC and other datasets

Conclusions Jigsaw model allows learning the shape and appearance of objects or object parts

Conclusions Jigsaw model allows learning the shape and appearance of objects or object parts in images. Can also handle occlusion. Clustering shape and appearance much more powerful for recognition than appearance alone. Can be used as a ‘plug-and-play’ replacement for fixed size patches in any existing patch-based system.

Thank you jwinn@microsoft. com http: //johnwinn. org

Thank you jwinn@microsoft. com http: //johnwinn. org

JIGSAWS JOINT APPEARANCE AND SHAPE CLUSTERING John WinnJIGSAWS JOINT APPEARANCE AND SHAPE CLUSTERING John Winn
LEARNING JIGSAWS FOR CLUSTERING APPEARANCE AND SHAPE JohnLEARNING JIGSAWS FOR CLUSTERING APPEARANCE AND SHAPE John
Modeling Facial Shape and Appearance Shape and ChangesModeling Facial Shape and Appearance Shape and Changes
John Ellis appearance disappearance and corrections appearance LesJohn Ellis appearance disappearance and corrections appearance Les
Appearance Models Shape models represent shape variation EigenmodelsAppearance Models Shape models represent shape variation Eigenmodels
Texton Boost Joint Appearance Shape and Context ModelingTexton Boost Joint Appearance Shape and Context Modeling
Lecture 12 Clustering What is Clustering Clustering isLecture 12 Clustering What is Clustering Clustering is
ENSEMBLE CLUSTERING ENSEMBLE CLUSTERING clustering algorithm 1 partitionENSEMBLE CLUSTERING ENSEMBLE CLUSTERING clustering algorithm 1 partition
FLAT CLUSTERING HIERARCHICAL CLUSTERING What is clustering GroupingFLAT CLUSTERING HIERARCHICAL CLUSTERING What is clustering Grouping
DATA MINING CLUSTERING Clustering 4 Clustering unsupervised classificationDATA MINING CLUSTERING Clustering 4 Clustering unsupervised classification
Clustering Kmeans Clustering 2 means Clustering Curve ShowsClustering Kmeans Clustering 2 means Clustering Curve Shows
Kmeans Clustering Kmeans Clustering What is clustering WhyKmeans Clustering Kmeans Clustering What is clustering Why
C 0 clustering Liang Zheng Clustering Clustering alongC 0 clustering Liang Zheng Clustering Clustering along
Variable Clustering 1 Variable Clustering Variable clustering findsVariable Clustering 1 Variable Clustering Variable clustering finds
CLUSTERING Overview Definition of Clustering Existing clustering methodsCLUSTERING Overview Definition of Clustering Existing clustering methods
Clustering Outline Introduction Kmeans clustering Hierarchical clustering COBWEBClustering Outline Introduction Kmeans clustering Hierarchical clustering COBWEB
Stable Clustering Vladyslav Kolbasin Clustering data Clustering isStable Clustering Vladyslav Kolbasin Clustering data Clustering is
Space Shape and Form Shape Shape is aSpace Shape and Form Shape Shape is a
Shape Form and Space Shape What is shapeShape Form and Space Shape What is shape
Whats my shape Element Shape What is shapeWhats my shape Element Shape What is shape