Venus Classification Faces Different Faces Same Lighting affects

  • Slides: 52
Download presentation
Venus

Venus

Classification

Classification

Faces – Different

Faces – Different

Faces -- Same

Faces -- Same

Lighting affects appearance

Lighting affects appearance

Three-point alignment

Three-point alignment

Object Alignment Given three model points P 1, P 2, P 3, and three

Object Alignment Given three model points P 1, P 2, P 3, and three image points p 1, p 2, p 3, there is a unique transformation (rotation, translation, scale) that aligns the model with the image. (SR + d)Pi = pi

Alignment -- comments • The projection is orthographic projection (combined with scaling). • The

Alignment -- comments • The projection is orthographic projection (combined with scaling). • The 3 points are required to be non-collinear. • The transformation is determined up to a reflection of the points about the image plane and translation in depth.

 Proof of the 3 -point Alignment: The 3 3 -D points are P

Proof of the 3 -point Alignment: The 3 3 -D points are P 1, P 2, P 3. We can assume that they are initially in the image plane. In the 2 -D image we get q 1, q 2, q 3. The transformation P 1 > q 1, P 2 > q 2, P 3 > q 3, defines a unique linear transformation of the plane, L(x). We can easily recover this transformation. L is a 2*2 matrix. We fix the origin at P 1 = q 1. We have two more points that define 4 linear equations for the elements of L. We now choose two orthogonal vectors E 1 and E 2 in the original plane of P 1, P 2, P 3. We can compute E 1’ = L(E 1), E 2’ = L(E 2). We seek a scaling S, Rotation R, so that the projection of SR(E 1) = E 1’ and SR(E 2) = E 2’. Let SR(E 1) (without the projection) be V 1 and SR(E 2) = V 2. V 1 is E 1’ plus a depth component, that is, V 1 = E 1’ + c 1 z, where z is a unit vector in the z direction. Similarly, V 2 = E 2’ + c 2 z. We wish to recover c 1 and c 2. This will give the transformation between the points (show that it is unique, and it will be possible to recover the transformation). We know that the scalar product (V 1 V 2) = 0. (E 1’ + c 1 z) = 0 Therefore c 1 c 2 = -(E’ 1 E’ 2). The magnitude -(E’ 1 E’ 2) is measurable in the image, call it C 12, therefore c 1 c 2= c 12. Also |V 1| = |V 2|. Therefore (E 1’ + c 1 z) = (E 1’ + c 1 z). 2 2 This implies c 1 - c 2 = k 12, where k 12 is a measurable quantity in the image (it is |E’ 1 | - |E’ 2 |. The two equation of c 1 c 2 are: c 1 c 2 = c 12 2 2 c 1 - c 2 = k 12 2 2 and they have a unique solution. One way of seeing this is by setting a complex number Z = c 1 + ic 2. Then Z = k 12 + ic 12. Therefore, Z is measurable. We take the square root and get Z, therefore c 1, c 2. There are exactly two roots giving the two mirror reflection solutions.

Car Recognition

Car Recognition

Car Models

Car Models

Alignment: Cars

Alignment: Cars

Alignment: Unmatch

Alignment: Unmatch

Face Alignment

Face Alignment

Linear Combination of Views

Linear Combination of Views

Linear Combination of Views O is a set of object points. I 1, I

Linear Combination of Views O is a set of object points. I 1, I 2, I 3, are three images of O from different views. N is a novel view of O. Then O is the linear combination of I 1, I 2, I 3.

Car Recognition

Car Recognition

VW – SAAB

VW – SAAB

LC – Car Images

LC – Car Images

Linear Combination: Faces

Linear Combination: Faces

Classification

Classification

Structural descriptions

Structural descriptions

RBC

RBC

RBC

RBC

Structural Description G 1 Above G 2 Touch G 3 Right-of G 4 G

Structural Description G 1 Above G 2 Touch G 3 Right-of G 4 G 2 Above Left-of G 4

Fragment-based Representation

Fragment-based Representation

Mutual Information Mutual information Entropy Binary variable -H(C) = P(C=1)Log(P(C=1) + P(C=0)Log(P(C=0)

Mutual Information Mutual information Entropy Binary variable -H(C) = P(C=1)Log(P(C=1) + P(C=0)Log(P(C=0)

Mutual information H(C) F=1 H(C) when F=1 F=0 H(C) when F=0 I(C; F) =

Mutual information H(C) F=1 H(C) when F=1 F=0 H(C) when F=0 I(C; F) = H(C) – H(C/F)

Selecting Fragments

Selecting Fragments

Fragments Selection • For a set of training images: • Generate candidate fragments –

Fragments Selection • For a set of training images: • Generate candidate fragments – Measure p(F/C), p(F/NC) • Compute mutual information • Select optimal fragment • After k fragments: Maximizing the minimal addition in mutual information with respect to each of the first k fragments

Optimal Face Fragments

Optimal Face Fragments

Face Fragments by Type

Face Fragments by Type

Low-resolution Car Fragments Front – Middle - Back

Low-resolution Car Fragments Front – Middle - Back

10 5 0 0 1 2 3 Relative object size Relative mutual info. a.

10 5 0 0 1 2 3 Relative object size Relative mutual info. a. 100 x 100 Merit, weight x Merit 15 6 5 4 3 2 1 0 0 1 2 3 4 Relative object size b. 1. 5 1 0. 5 0 - 0. 5 0 1 2 Relative object size 3 100 x Merit, weight 100 x 100 Merit, weight Intermediate Complexity 1. 2 1 0. 8 0. 6 0. 4 0. 2 0 0 0. 5 1 Relative resolution 1. 5 2

Fragment ‘Weight’ Likelihood ratio: Weight of F:

Fragment ‘Weight’ Likelihood ratio: Weight of F:

Combining fragments w 1 D 1 wk w 2 Dk

Combining fragments w 1 D 1 wk w 2 Dk

Non-optimal Fragments Fragme nts Optimal size detecti on 11% 95 F/A 0 Small 3%

Non-optimal Fragments Fragme nts Optimal size detecti on 11% 95 F/A 0 Small 3% 97 30 Large 33% 39 0 Same total area covered (8*object), on regular grid

Training & Test Images • • • Frontal faces without distinctive features (K: 496,

Training & Test Images • • • Frontal faces without distinctive features (K: 496, W: 385) Minimize background by cropping Training images for extraction: 32 for each class Training images for evaluation: 100 for each class Test images: 253 for Western and 364 for Korean

Training – Fragment Extraction

Training – Fragment Extraction

Extracted Fragments Korean Fragment Score 0. 92 0. 82 0. 77 0. 76 0.

Extracted Fragments Korean Fragment Score 0. 92 0. 82 0. 77 0. 76 0. 75 0. 74 0. 72 0. 68 0. 67 0. 65 Weight 3. 42 2. 40 1. 99 2. 23 1. 90 2. 11 6. 58 4. 14 4. 12 6. 47 Western Fragment

Classifying novel images Compare Summed Weights Detect Fragments Decision k. F Westerner w. F

Classifying novel images Compare Summed Weights Detect Fragments Decision k. F Westerner w. F Unknown Korean

Effect of Number of Fragments • 7 fragments: 95%, 80 fragments: 100% • Inherent

Effect of Number of Fragments • 7 fragments: 95%, 80 fragments: 100% • Inherent redundancy of the features • Slight violation of independence assumption

Comparison with Humans Algorithm outperformed humans for low resolution images •

Comparison with Humans Algorithm outperformed humans for low resolution images •

Class examples

Class examples

Distinctive Features

Distinctive Features