Using Group Prior to Identify People In Consumer

The Problem Consumer image collections are growing exponentially each year. Consumers want to search

Traditional Face Recognition Determines the assignment of each person independently Extract facial features Build

The Group Prior for Learning The Semantics of People in Images Determine the joint

System Diagram Hannah Jonah Holly Andy Jonah Holly Hannah Holly Andy Jonah Images (Faces)

Recognizing a Person When a single person is in the image: Posterior Probability Likelihood

Estimating the Group Prior For pairs of names, the group prior is estimated by

Ambiguous Labels Jonah Hannah Andy Hannah Holly Andy Hannah Jonah Hannah Holly Jonah Hannah

Classification with Group Prior From the joint pdf, inference questions can be answered: Most

Experiment The image collection: Images 1197 Images with multiple people 188 No. Faces in

Results Group Prior produces a large benefit. Note: All images were ambiguously labeled; no

Prior Work Many face recognition methods- most ignore the issue of prior probabilities. [Zhao

Conclusions The group prior models the social relationships between individuals. We learn feature distributions

Slides: 16

Download presentation

Using Group Prior to Identify People In Consumer Images Andrew C. Gallagher Tsuhan Chen Carnegie Mellon University Eastman Kodak Company June 18, 2007 CVPR SLAM 2007

The Problem Consumer image collections are growing exponentially each year. Consumers want to search for images based on whom the image contains. And they don’t like to label images! This is more than a face recognition problem. To best understand the semantics of who is in the images, we need to understand the people in the images. CVPR SLAM 2007

Traditional Face Recognition Determines the assignment of each person independently Extract facial features Build a classifier that finds the most likely name, given the features. But this method does not take full advantage of the available information! CVPR SLAM 2007

The Group Prior for Learning The Semantics of People in Images Determine the joint assignment of all people in the image to names, using the group prior. By the unique object constraint (UOC), an individual can appear only once in the image. The group prior characterizes the prior probability of certain groups of people appearing together in an image. CVPR SLAM 2007

System Diagram Hannah Jonah Holly Andy Jonah Holly Hannah Holly Andy Jonah Images (Faces) Ambiguous Label Resolution Ambiguous Labels Group Prior Labeled Faces Classifier Training Unlabeled Image Annotated Image Recognize People Hannah Holly CVPR SLAM 2007

Recognizing a Person When a single person is in the image: Posterior Probability Likelihood Individual Prior : the set of all unique names : a member of the set : the features from person image CVPR SLAM 2007

The Group Prior p 1 p. M p 2 … f 1 f 2 … f. M Recognizing Multiple People The graph model represents the features and people in an image. The graph encodes the independence assumptions of our model. E. g. given the identity of a person, their features are independent of others in the image. CVPR SLAM 2007

The Group Prior p 1 p. M p 2 … f 1 f 2 … Recognizing Multiple People The joint probability function: f. M The Group Prior. Likelihood : an index over the people in the image : the set of all features for all people : the set of people in the image : a subset of ; a particular assignment of a name to each person in. CVPR SLAM 2007

Estimating the Group Prior For pairs of names, the group prior is estimated by counting the number of images the pair appears, then normalizing. The Group Prior The group prior for 3 or more people is estimated according to the group prior pairwise graphical model. The Individual Prior CVPR SLAM 2007

Ambiguous Labels Jonah Hannah Andy Hannah Holly Andy Hannah Jonah Hannah Holly Jonah Hannah Andy Hannah Holly Jonah Andy Jonah Hannah Andy Hannah Jonah Holly Jonah Andy Hannah Andy Ambiguous labels indicate who is in the image, but not which person is which name. A constrained clustering algorithm is used to ‘resolve’ the labels. The resolved labels are used to learn the feature distribution for each name. CVPR SLAM 2007

Classification with Group Prior From the joint pdf, inference questions can be answered: Most Probable Explanation MAP- Most probable assignment of a particular person. CVPR SLAM 2007

Experiment The image collection: Images 1197 Images with multiple people 188 No. Faces in these images 420 Individuals 5 Facial Features: Active Shape Model [Cootes 95] based features, then PCA reduces to 5 D. • Ambiguously label a portion of the image collection, classify the identities of all the rest. • Compare 4 Priors: • Group Prior (GP) • UOC Prior A binary version of the GP that respects the UOC. • The individual prior. • No Prior • The performance is quantified for: • MAP • MPE CVPR SLAM 2007

Results Group Prior produces a large benefit. Note: All images were ambiguously labeled; no people were explicitly labeled. Example Classification (from 10 labeled images) Individual Prior / no Prior Ethan Unique Object Constraint Hannah Ethan Group Prior Holly Ethan CVPR SLAM 2007

Prior Work Many face recognition methods- most ignore the issue of prior probabilities. [Zhao 03] Face recognition methods have been used to assist the labeling of image collections. [Zhang 04] In news photos, names from captions have been assigned to faces. [Berg 04] The co-occurrence of people in images has been studied, but not combined with image features. [Naaman 05] CVPR SLAM 2007

Conclusions The group prior models the social relationships between individuals. We learn feature distributions and relationships between the labels (people). By using the group prior, recognition accuracy is significantly improved! CVPR SLAM 2007