Dimensionality Reduction and Embeddings SVD The mathematical formulation

Dimensionality Reduction and Embeddings

SVD: The mathematical formulation n n Normalize the dataset by moving the origin to the center of the dataset Find the eigenvectors of the data (or covariance) matrix These define the new space Sort the eigenvalues in “goodness” order f 2 e 1 f 1

Compute Approximate SVD efficiently Exact SVD is expensive: O(min{n 2 m, n m 2}) So, we try to compute it approximately. We exploit the fact that, if A = U L VT then: A AT = U L 2 UT and ATA = V L 2 VT n 1. Random projection + SVD. Cost O(m n logn) 2. Random sampling ( p rows) and then SVD on the samples. Cost O(max{m p 2+p 3}) or O(p 4)!! (caution: constants can be high!)

Approximate SVD n We can guarantee an approximation like the following: || A – P||2 F <= || A – Ak||2 F+ e ||A||2 F

A randomized SVD

SVD Cont’d n n Advantages: n Optimal dimensionality reduction (for linear projections) Disadvantages: n Computationally expensive… but can be improved with random sampling n Sensitive to outliers and non-linearities

Embeddings n n Given a metric distance matrix D, embed the objects in a k-dimensional vector space using a mapping F such that n D(i, j) is close to D’(F(i), F(j)) Isometric mapping: n exact preservation of distance Contractive mapping: n D’(F(i), F(j)) <= D(i, j) D’ is some Lp measure

Multi-Dimensional Scaling (MDS) n n Map the items in a k-dimensional space trying to minimize the stress Steepest Descent algorithm: n Start with an assignment n Minimize stress by moving points But the running time is O(N 2) and O(N) to add a new item Another method: stress iterative majorization

Fast. Map What if we have a finite metric space (X, d )? Faloutsos and Lin (1995) proposed Fast. Map as metric analogue to the PCA. Imagine that the points are in a Euclidean space. n Select two pivot points xa and xb that are far apart. n Compute a pseudo-projection of the remaining points along the “line” xaxb. n “Project” the points to an orthogonal subspace and recurse.

Fast. Map n We want to find e 1 first f 2 e 1 e 2 f 1

Selecting the Pivot Points The pivot points should lie along the principal axes, and hence should be far apart. x 2 Select any point x 0. Let x 1 be the furthest from x 0. Let x 2 be the furthest from x 1. Return (x 1, x 2). x 0 n n x 1

Pseudo-Projections Given pivots (xa , xb ), for any third point y, we use the law of cosines to determine the relation of y along xaxb. xb db, y da, b y cy da, y The pseudo-projection for y is xa

“Project to orthogonal plane” cz-cy Given distances along xaxb we dy, z can compute distances y within the “orthogonal hyperplane” using the Pythagorean theorem. y’ Using d ’(. , . ), recurse until k features chosen. xb z xa d’y’, z’ z’

Compute the next coordinate n n Now, we have projected all objects into a subspace orthogonal to first dimension (line xa, xb) We can apply recursively Fast. Map on the new projected dataset: Fast. Map(k-1, d’, D)

Embedding using ML n n n We can try to use some learning techniques to “learn” the best mapping. Works for general metric spaces, not only “Euclidean spaces” Vassilis Athitsos, Jonathan Alon, Stan Sclaroff, George Kollios: Boost. Map: An Embedding Method for Efficient Nearest Neighbor Retrieval. IEEE Trans. Pattern Anal. Mach. Intell. 30(1): 89 -104 (2008)

Boost. Map Embedding x 1 database x 2 x 3 xn 16

Embeddings x 1 x 2 x 3 database embedding F Rd x 1 x 2 xn x 3 x 4 xn 17

Embeddings x 1 x 2 x 3 database embedding F Rd x 1 xn x 3 xn x 2 x 4 query q 18

Embeddings database x 1 Rd x 2 x 3 embedding F x 1 xn x 3 xn q x 4 query q 19

Embeddings n Measure distances between vectors (typically much faster). database x 1 Rd x 2 x 3 embedding F x 1 xn x 3 xn q x 4 query q 20

Embeddings n n x 1 x 2 x 3 Measure distances between vectors (typically much faster). Caveat: the embedding must preserve similarity structure. database embedding F Rd x 1 xn x 3 xn x 2 q x 4 query q 21

Reference Object Embeddings original space X 22

Reference Object Embeddings r original space X r: reference object 23

Reference Object Embeddings r original space X r: reference object Embedding: F(x) = D(x, r) D: distance measure in X. 24

Reference Object Embeddings r original space X r: reference object F Real line Embedding: F(x) = D(x, r) D: distance measure in X. 25

Reference Object Embeddings n F(r) = D(r, r) = 0 r original space X r: reference object F Real line Embedding: F(x) = D(x, r) D: distance measure in X. 26

Reference Object Embeddings n n r a original space X r: reference object b F F(r) = D(r, r) = 0 If a and b are similar, their distances to r are also similar (usually). Real line Embedding: F(x) = D(x, r) D: distance measure in X. 27

Reference Object Embeddings n n r a original space X r: reference object b F F(r) = D(r, r) = 0 If a and b are similar, their distances to r are also similar (usually). Real line Embedding: F(x) = D(x, r) D: distance measure in X. 28

F(x) = D(x, Lincoln) F(Sacramento). . = F(Las Vegas). . . = F(Oklahoma City). = F(Washington DC). = F(Jacksonville). . = 1543 1232 437 1207 1344 29

F(x) = (D(x, LA), D(x, Lincoln), D(x, Orlando)) F(Sacramento). . = F(Las Vegas). . . = F(Oklahoma City). = F(Washington DC). = F(Jacksonville). . = ( 386, ( 262, (1345, (2657, (2422, 1543, 2920) 1232, 2405) 437, 1291) 1207, 853) 1344, 141) 30

F(x) = (D(x, LA), D(x, Lincoln), D(x, Orlando)) F(Sacramento). . = F(Las Vegas). . . = F(Oklahoma City). = F(Washington DC). = F(Jacksonville). . = ( 386, ( 262, (1345, (2657, (2422, 1543, 2920) 1232, 2405) 437, 1291) 1207, 853) 1344, 141) 31

Basic Questions What is a good way to optimize an embedding? 32

Basic Questions F(x) = (D(x, LA), D(x, Denver), D(x, Boston)) What is a good way to optimize an embedding? What are the best reference objects? What distance should we use in R 3? 33

Key Idea n n Embeddings can be seen as classifiers. Embedding construction can be seen as a machine learning problem. n n Formulation is natural. We optimize exactly what we want to optimize. 34

Ideal Embedding Behavior original space X q F Rd a Notation: NN(q) is the nearest neighbor of q. For any q: if a = NN(q), we want F(a) = NN(F(q)). 35

A Quantitative Measure b q a If b is not the nearest neighbor of q, F(q) should be closer to F(NN(q)) than to F(b). For how many triples (q, NN(q), b) does F fail? 36

A Quantitative Measure q a F fails on five triples. 37

Embeddings Seen As Classifiers b q a Classification task: is q closer to a or to b? 38

Embeddings Seen As Classifiers b q n a Classification task: is q closer to a or to b? Any embedding F defines a classifier F’(q, a, b). n F’ checks if F(q) is closer to F(a) or to F(b). 39

Classifier Definition b q n Given embedding F: X Rd: n n n a Classification task: is q closer to a or to b? F’(q, a, b) = ||F(q) – F(b)|| - ||F(q) – F(a)||. F’(q, a, b) > 0 means “q is closer to a. ” F’(q, a, b) < 0 means “q is closer to b. ” 40

Classifier Definition Goal: build an F such that F’ has low error rate on triples of type (q, NN(q), b). n Given embedding F: X Rd: n n n F’(q, a, b) = ||F(q) – F(b)|| - ||F(q) – F(a)||. F’(q, a, b) > 0 means “q is closer to a. ” F’(q, a, b) < 0 means “q is closer to b. ” 41

1 D Embeddings as Weak Classifiers n 1 D embeddings define weak classifiers. n n Better than a random classifier (50% error rate). We can define lots of different classifiers. n Every object in the database can be a reference object. Question: how do we combine many such classifiers into a single strong classifier? 42

1 D Embeddings as Weak Classifiers n 1 D embeddings define weak classifiers. n n Better than a random classifier (50% error rate). We can define lots of different classifiers. n Every object in the database can be a reference object. Question: how do we combine many such classifiers into a single strong classifier? Answer: use Ada. Boost (uses boosting) n Ada. Boost is a machine learning method designed for exactly this problem. 43

Using Ada. Boost original space X Real line F 1 F 2 Fn n Output: H = w 1 F’ 1 + w 2 F’ 2 + … + wd. F’d. n n n Ada. Boost chooses 1 D embeddings and weighs them. Goal: achieve low classification error. Ada. Boost trains on triples chosen from the database. 44

From Classifier to Embedding Ada. Boost output H = w 1 F’ 1 + w 2 F’ 2 + … + wd. F’d What embedding should we use? What distance measure should we use? 45

From Classifier to Embedding Ada. Boost output Boost. Map embedding H = w 1 F’ 1 + w 2 F’ 2 + … + wd. F’d F(x) = (F 1(x), …, Fd(x)). 46

From Classifier to Embedding Ada. Boost output Boost. Map embedding Distance measure H = w 1 F’ 1 + w 2 F’ 2 + … + wd. F’d F(x) = (F 1(x), …, Fd(x)). D((u 1, …, ud), (v 1, …, vd)) = d w |u – v | i i=1 47

From Classifier to Embedding Ada. Boost output Boost. Map embedding Distance measure H = w 1 F’ 1 + w 2 F’ 2 + … + wd. F’d F(x) = (F 1(x), …, Fd(x)). D((u 1, …, ud), (v 1, …, vd)) = d w |u – v | i i=1 Claim: Let q be closer to a than to b. H misclassifies triple (q, a, b) if and only if, under distance measure D, F maps q closer to b than to a. 48

Significance of Proof n n Ada. Boost optimizes a direct measure of embedding quality. We have converted a database indexing problem into a machine learning problem. 50

Recap of Boost. Map Algorithm n n Start with a large collection of 1 D embeddings. Each embedding defines a weak classifier on triples of objects. Ada. Boost combines many weak classifiers into a strong classifier. The strong classifier defines an embedding and a weighted L 1 distance. n Classifier equivalent to embedding and distance. 51

How Do We Use It? Filter-and-refine retrieval: n Offline step: compute embedding F of entire database. n Given a query object q: n Embedding step: n n Filter step: n n Compute distances from query to reference objects F(q). Find top p matches of F(q) in vector space. Refine step: n Measure exact distance from q to top p matches. 52

Evaluating Embedding Quality How often do we find the true nearest neighbor? n Embedding step: n n Filter step: n n Compute distances from query to reference objects F(q). Find top p matches of F(q) in vector space. Refine step: n Measure exact distance from q to top p matches. 53

Random Projections n n Based on the Johnson-Lindenstrauss lemma: For: n 0< e < 1/2, n any (sufficiently large) set S of M points in R n -2 n k = O(e ln. M) There exists a linear map f: S Rk, such that n (1 - e) D(S, T) < D(f(S), f(T)) < (1+ e)D(S, T) for S, T in S Random projection is good with constant probability

Random Projection: Application n n Set k = O(e-2 ln. M) Select k random n-dimensional vectors n n n (an approach is to select k gaussian distributed vectors with variance 1 and mean value 0: N(0, 1) ) Project the original points into the k vectors. The resulting k-dimensional space approximately preserves the distances with high probability

Database Friendly Random Projection n n For each point (vector) x in d-dimensions need to find the projection to point y in k-dimensions For n points, using the naive approach, I need to perform ndk operations. this can be large for large datasets and dimensionalities. A better approach is the following [Achlioptas 2003]: n Create a matrix A such that: 1 with prob 1/6 A[i, j] = 0 with prob 2/3 -1 with prob 1/6 Then, we can compute each y as: y = x A Why this is better?

Random Projection n A very useful technique, Especially when used in conjunction with another technique (for example SVD) Use Random projection to reduce the dimensionality from thousands to hundred, then apply SVD to reduce dimensionality farther References: [Achlioptas 2003] Dimitris Achlioptas: Database-friendly random projections: Johnson-Lindenstrauss with binary coins. J. Comput. Syst. Sci. 66(4): 671 -687 (2003)