Transfer Learning of Object Classes From Cartoons to

Transfer Learning of Object Classes: From Cartoons to Photographs NIPS Workshop Inductive Transfer: 10 Years Later Geremy Heitz Gal Elidan Daphne Koller December 9 th, 2005

Localization vs. Recognition Traditional question: “Is there an object of type X in this image? ” Airplane? NO Human? YES Dog? YES Our question: “Where in this image is the object of type X? ” MAN DOG The man is walking the dog

Outline Landmark-based shape model Localization as inference Transfer learning from cartoon drawings Results

Shape Model tail nose Set of landmarks Piecewise-linear contour between neighbors Features of individual landmarks Features of pairs of landmarks

Outline Landmark-based shape model Localization as inference Transfer learning from cartoon drawings Results

“Registering” the Model to an Image Requires assigning each landmark to a pixel location ? ?

Localization Are local cues enough? Lnose Ltail Markov Random Field “Correct” pixel is often not the best match! • Potentials = Functions of local and global features Lcockpit Need to jointly Lunder consider all cues (features) Registration = Most Likely Assignment

Outline Landmark-based shape model Localization as inference Transfer learning from cartoon drawings Results

Learning Challenge Hand Label Hidden Variables ? Bootstrap from simple? instances ? where outlining is easy = cartoons / drawings ? ? Costly, and time-consuming no confusing background Where to start? Local optima problem outline (shape) is easily recovered using snake

Learning from Cartoon Drawings Shape Learning + Shape and Appearance Learning Registration

Phase I: Learning from Cartoons Registration Pyramid Final Shape Model Extract high resolution contour using snake Create shape-based model from training contours Pairwise merging of models Selection of landmarks

Phase II: Learning from Images Training Set Selection Cartoon Phase Model Natural Image Model Transfer high score low score Correspond initial model to training images Select best correspondences as training instances Learn final shape- and appearance-based model

Outline Landmark-based shape model Localization as inference Transfer learning from cartoon drawings Results

Localization Results sample training cartoons sample registration 0. 81 0. 84 0. 81 0. 75 0. 66 0. 84 0. 77 0. 72 0. 40 0. 18

Transfer of Object Shape Average overlap 0. 6 transfer 0. 5 Benefit of shape transfer 0. 4 no transfer 0. 3 0. 2 0. 1 0 0 2 4 6 8 10 # images in phase II Transfer of shape speeds up learning

Learning Appearance shape + appearance 0. 64 Average overlap 0. 62 0. 6 0. 58 Shape template 0. 56 No Appearance 0. 54 0. 52 0. 5 0. 48 0. 46 0 2 4 6 # images in phase II 8 10 FG/BG Appearance

Training Instance Selection AUTO HAND Average overlap 0. 7 0. 65 PICKED 0. 6 AUTO 0. 55 0. 45 0. 4 0. 35 0. 3 0 2 4 6 8 # images in phase II 10 12 PICKED

Summary and Future Work Ø Ø Ø Flexible probabilistic shape model Effective registration to images Transfer Ø Ø Shape from cartoons Appearance from real images Develop a better appearance model Investigate self-training issues Transfer from one class to another

Thanks!

Cartoon vs. Hand Segmentation Mean Overlap Score 0. 9 Human Inter-Observer Hand Constructed 0. 7 Learned from Drawings 0. 5 0. 3 cartoon 0. 1 0 1 2 3 4 5 hand segmented Number of Training Instances Learning shape from cartoons is competitive with hand segmentation!

Landmark Features Shape Template Patch Appearance (Foreground/Background) Location

Prediction True positive rate 1 0. 8 0. 6 0. 4 0. 2 0 0 0. 2 0. 4 0. 6 0. 8 False positive rate 1 object car side cougar airplane buddha bass rooster recognition 86% 86% 84% 76% 73% § Comparable to constellation w/ 5 instances (Fei et. Al) § Leading (discriminative) methods require many instances