CONSTRAINED SEMISUPERVISED LEARNING USING ATTRIBUTES AND COMPARATIVE ATTRIBUTES

CONSTRAINED SEMI-SUPERVISED LEARNING USING ATTRIBUTES AND COMPARATIVE ATTRIBUTES Abhinav Shrivastava, Saurabh Singh, Abhinav Gupta The Robotics Institute Carnegie Mellon University

SUPERVISION BIG-DATA ACTIVE LEARNING SUPERVIS ED [Prakash and Parikh, ECCV 2012] [Vijayanarasimhan and Grauman, CVPR 2011] [Kapoor et al. , ICCV 2007] [Qi et al. , CVPR 2008] [Joshi et al. , CVPR 2009] [Siddiquie and Gupta, CVPR 2010] [von Ahn and Dabbish, 2004], [Russell et al. , IJCV 2009] 2

SUPERVISION UNSUPERVISE D [Russell et al. , CVPR 2006] [Kang et al. , ICCV 2011] ACTIVE LEARNING SUPERVIS ED 3

SUPERVISION UNSUPERVISE D SEMISUPERVISED [Zhu, TR, 2005], [Chunsheng Fang, Slides, 2009] ACTIVE LEARNING SUPERVIS ED 4

SUPERVISION UNSUPERVISE D SEMISUPERVISED ACTIVE LEARNING BOOTSTRAPPING Labeled Seed Examples Retrain Train Models Unlabeled Data Amphitheatre SUPERVIS ED Select Candidates Amphitheatre Add to Labeled Set 5

SUPERVISION UNSUPERVISE D SEMISUPERVISED ACTIVE LEARNING BOOTSTRAPPING Labeled Seed Examples Retrain Models Unlabeled Data Amphitheatre Select Candidates Amphitheatre + Auditorium Amphitheatre Add to Labeled Set [Curran et al. , PACL 2007] SUPERVIS ED Semantic Drift 25 th Iteration 6

SUPERVISION UNSUPERVISE D SEMISUPERVISED ACTIVE LEARNING SUPERVIS ED GRAPH-BASED METHODS [Ebert et al. , ECCV 2010] [Fergus et al. , NIPS 2009] 7

OUR APPROACH Amphitheatre Auditorium 8

OUR APPROACH Amphitheatre Joint Learning Auditorium Amphitheatre Auditorium Share Data [Carlson et al. , NAACL HLT Workshop on SSL for NLP 2009] 9

BINARY ATTRIBUTES (BA) Amphitheatre Conference Room Indoor Tables and Chairs Man-made Large Seating Capacity Banquet Hall Auditorium 10 [Farhadi et al. , CVPR 2009] [Lampert et al. , CVPR 2009]

BINARY ATTRIBUTES (BA) Tables and Chairs Large Seating Capacity Conference Room Amphitheatre Man-made Indoor Banquet Hall [Patterson and Hays, CVPR 2012] Auditorium

Auditorium Indoor Has Seat Rows ✗ 12

SHARING VIA DISSIMILARITY Amphitheatre Auditorium Has Larger Circular Structures [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] 13

Amphitheatre Has Larger Circular Structures Auditorium ? 14

Amphitheatre Has Larger Circular Structures Auditorium ✗ 15

DISSIMILARITY COMPARATIVE ATTRIBUTES Has Larger Circular Structures [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] 16

DISSIMILARITY COMPARATIVE ATTRIBUTES Features Has Larger Circular Structures • GIST • RGB (Tiny Image) • Line Histogram of: § Length § Orientation • LAB histogram ………… [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] [Hays and Efros, SIGGRAPH 2007] [Oliva and Torralba, 2006] [Torralba et al. , PAMI, 2008] 17

DISSIMILARITY COMPARATIVE ATTRIBUTES Classifier Boosted Decision Tree [Hoiem et al. , IJCV 2007] Has Larger Circular Structures ………… Has Larger Circular Structures [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] ✗ or 18

COMPARATIVE ATTRIBUTES Is More Open Amphitheatre > Barn Amphitheatre > Conference Room Desert > Barn Has Taller Structures Church (Outdoor) > Cemetery Barn > Cemetery [Parikh and Grauman, ICCV 2011] [Gupta and Davis, ECCV 2008] 19

Selected Candidates Labeled Seed Examples Bootstrapping Amphitheatre Auditorium 20

Labeled Seed Examples Selected Candidates Our Approach Bootstrapping (Constrained Bootstrapping ) Amphitheatre Attributes Indoor Has Seat Rows Auditorium Comparative Auditorium Attributes Auditorium Has Larger Circular Structures 21

banquet hall bedroom Banquet has grass Labeled Data indoor Bedroom Scene Classifiers Unlabeled Data has larger structures Training Pairwise Data more space has more space larger structures Attribute Classifiers Comparative Attribute Classifiers Conference Room [Gupta and Davis, ECCV 2008] Banquet Hall Promoted Instances 22

Iteration 40 Iteration 1 Seed Examples Conference Room Introspection 23

Our Approach BA Constraints Bootstrapping Seed Images Amphitheatre 24

EXPERIMENTAL EVALUATION 26

CONTROL EXPERIMENTS # Images (SUN Database) 15 Scene Classes Black-box Binary Attributes (BA) 2 (seed) 25 (separate) Black-box Comparative Attributes (CA) 25 (separate) Unlabeled Dataset (Distractors) Test Set SUN Database : [Xiao et al. , CVPR 2010] 18, 000 (9, 500) 50 27

CONTROL EXPERIMENTS 28

CONTROL EXPERIMENTS 29

CONTROL EXPERIMENTS 30

CONTROL EXPERIMENTS 31

\ CONTROL EXPERIMENTS 32

\ CONTROL EXPERIMENTS Eigen Functions: [Fergus et al. , NIPS 2009] 33

Iterations Seed Images Banquet Hall 1 10 40 34

CONTROL EXPERIMENTS # Images (SUN Database) 15 Scene Classes Black-box Binary Attributes (BA) 2 (seed) 25 (separate) Black-box Comparative Attributes (CA) 25 (separate) Unlabeled Dataset (Distractors) Test Set SUN Database : [Xiao et al. , CVPR 2010] 18, 000 (9, 500) 50 35

CO-TRAINING (SMALL SCALE) # Images (SUN Database) 15 Scene Classes Black-box Binary Attributes (BA) 2 (seed) 15 x 2 (seed) Black-box Comparative Attributes (CA) 15 x 2 (seed) Unlabeled Dataset (Distractors) Test Set SUN Database : [Xiao et al. , CVPR 2010] 18, 000 (9, 500) 50 36

Iteration-60 Iteration-1 Our Approach Iteration-60 Iteration-1 Bootstrapping Seed Images Bedroom 37

SCENE CLASSIFICATION Eigen Functions: [Fergus et al. , NIPS 2009] 38

CO-TRAINING (LARGE SCALE) • 15 Scene Categories § 25 Seed images / category • Unlabeled Set § 1 Million (SUN Database + Image. Net) § >95% distractors Improve 12 out of 15 scene classifiers SUN Database: [Xiao et al. , CVPR 2010] Image. Net: [Deng et al. , CVPR 2009] 39

CONCLUSION • Sharing via Dissimilarities Amphitheatre Auditorium Has Larger Circular Structures • Constrained Bootstrapping Labeled Seed Examples Bootstrapping Amphitheatre Auditorium Constrained Bootstrapping 40

banquet hall bedroom Banquet has grass Labeled Data indoor Bedroom Scene Classifiers Unlabeled Data has larger structures Training Pairwise Data more space has more space larger structures Attribute Classifiers Comparative Attribute Classifiers Conference Room Banquet Hall Promoted Instances 41

42

Unary Binary 43

EIGEN FUNCTIONS • 44

FEATURES • • • 960 D GIST 75 D RGB (image is resized to 5 x 5) 30 D histogram of line lengths 200 D histogram of orientation of lines 784 D 3 D-histogram Lab color space (14 x 4) • Total 2049 [Hays and Efros, SIGGRAPH 2007] [Oliva and Torralba, 2006] [Torralba et al. , PAMI, 2008] 45

CATEGORIES 46

BINARY ATTRIBUTE RELATIONSHIP 47

COMPARATIVE ATTRIBUTE RELATIONSHIPS 48

CLASSIFIERS • Boosted Decision Trees – From [Hoiem et al. , IJCV 2007] • Scene – 20 Trees, 8 Nodes • Binary Attribute – 40 Trees, 8 Nodes • Comparative Attribute • 20 Trees, 4 Nodes • Differential features as in [Gupta and Davis, ECCV 2008] 49

SUPERVISION ACTIVE LEARNING [Torralba et al. , PAMI 2008] SUPERVIS ED 50

SUPERVISION UNSUPERVISE D SEMISUPERVISED ACTIVE LEARNING SUPERVIS ED GRAPH-BASED METHODS Train Bus 51

MUTUAL EXCLUSION (ME) 52

EXPERIMENTAL EVALUATION 15 Scene Categories Experiments • Control • Small-scale • Large-scale Datasets • SUN Database • Image. Net Evaluation Metrics • Scene Classification – Mean AP • Purity of Labels 53

BASELINES • Bootstrapping (Self-learning) - Binary Classifiers - Multi-class Classifiers • Eigen Function based Graph Laplacian 54

SCENE CLASSIFICATION Eigen Functions: [Fergus et al. , NIPS 2009] 55

PURITY OF LABELING Eigen Functions: [Fergus et al. , NIPS 2009] 56

57

Seed Images Banquet Hall Iterations 1 10 40 90 58

59

PURITY OF LABELING Eigen Functions: [Fergus et al. , NIPS 2009] 60

SCENE CLASSIFICATION Mean 61