Transfer Learning of Object Classes From Cartoons to
Transfer Learning of Object Classes: From Cartoons to Photographs NIPS Workshop Inductive Transfer: 10 Years Later Geremy Heitz Gal Elidan Daphne Koller December 9 th, 2005
Localization vs. Recognition Traditional question: “Is there an object of type X in this image? ” Airplane? NO Human? YES Dog? YES Our question: “Where in this image is the object of type X? ” MAN DOG The man is walking the dog
Outline Landmark-based shape model Localization as inference Transfer learning from cartoon drawings Results
Shape Model tail nose Set of landmarks Piecewise-linear contour between neighbors Features of individual landmarks Features of pairs of landmarks
Outline Landmark-based shape model Localization as inference Transfer learning from cartoon drawings Results
“Registering” the Model to an Image Requires assigning each landmark to a pixel location ? ?
Localization Are local cues enough? Lnose Ltail Markov Random Field “Correct” pixel is often not the best match! • Potentials = Functions of local and global features Lcockpit Need to jointly Lunder consider all cues (features) Registration = Most Likely Assignment
Outline Landmark-based shape model Localization as inference Transfer learning from cartoon drawings Results
Learning Challenge Hand Label Hidden Variables ? Bootstrap from simple? instances ? where outlining is easy = cartoons / drawings ? ? Costly, and time-consuming no confusing background Where to start? Local optima problem outline (shape) is easily recovered using snake
Learning from Cartoon Drawings Shape Learning + Shape and Appearance Learning Registration
Phase I: Learning from Cartoons Registration Pyramid Final Shape Model Extract high resolution contour using snake Create shape-based model from training contours Pairwise merging of models Selection of landmarks
Phase II: Learning from Images Training Set Selection Cartoon Phase Model Natural Image Model Transfer high score low score Correspond initial model to training images Select best correspondences as training instances Learn final shape- and appearance-based model
Outline Landmark-based shape model Localization as inference Transfer learning from cartoon drawings Results
Localization Results sample training cartoons sample registration 0. 81 0. 84 0. 81 0. 75 0. 66 0. 84 0. 77 0. 72 0. 40 0. 18
Transfer of Object Shape Average overlap 0. 6 transfer 0. 5 Benefit of shape transfer 0. 4 no transfer 0. 3 0. 2 0. 1 0 0 2 4 6 8 10 # images in phase II Transfer of shape speeds up learning
Learning Appearance shape + appearance 0. 64 Average overlap 0. 62 0. 6 0. 58 Shape template 0. 56 No Appearance 0. 54 0. 52 0. 5 0. 48 0. 46 0 2 4 6 # images in phase II 8 10 FG/BG Appearance
Training Instance Selection AUTO HAND Average overlap 0. 7 0. 65 PICKED 0. 6 AUTO 0. 55 0. 45 0. 4 0. 35 0. 3 0 2 4 6 8 # images in phase II 10 12 PICKED
Summary and Future Work Ø Ø Ø Flexible probabilistic shape model Effective registration to images Transfer Ø Ø Shape from cartoons Appearance from real images Develop a better appearance model Investigate self-training issues Transfer from one class to another
Thanks!
Cartoon vs. Hand Segmentation Mean Overlap Score 0. 9 Human Inter-Observer Hand Constructed 0. 7 Learned from Drawings 0. 5 0. 3 cartoon 0. 1 0 1 2 3 4 5 hand segmented Number of Training Instances Learning shape from cartoons is competitive with hand segmentation!
Landmark Features Shape Template Patch Appearance (Foreground/Background) Location
Prediction True positive rate 1 0. 8 0. 6 0. 4 0. 2 0 0 0. 2 0. 4 0. 6 0. 8 False positive rate 1 object car side cougar airplane buddha bass rooster recognition 86% 86% 84% 76% 73% § Comparable to constellation w/ 5 instances (Fei et. Al) § Leading (discriminative) methods require many instances
- Slides: 22