Linear Discriminant Analysis Mark Stamp LDA 1 LDA

Linear Discriminant Analysis Mark Stamp LDA 1

LDA and QDA q Linear discriminant analysis (LDA) and quadratic discriminant analysis (QDA) q For both, the concept is simple q Based on labeled data, separate match from nomatch region o In LDA, using a (linear) hyperplane o In QDA, use a quadratic surface q Easy LDA to visualize in 2 -d 2

LDA vs QDA q In LDA 2 -d, separate with line vs parabola 3

LDA vs QDA q LDA is simpler, fewer parameters o Connections to both SVM and PCA o But, no kernel trick, and projection of data is simpler in LDA q QDA more complex, more parameters o Can separate some cases that LDA cannot o In practice, may not be too much better q We’ll LDA only consider LDA here 4

LDA and PCA and SVM (Oh My!) q Can view LDA training similar to PCA o Training set consists of m “experiments” with n “measurements” each o Form a covariance-like matrix q LDA training also related to SVM o We project/separate based on hyperplane o But, no kernel trick, so LDA is simpler q Lagrange LDA multipliers & eigenvectors? 5

Big Idea Behind LDA? q Scatter is closely related to variance q Mean and scatter of training data is not under our control o But, have some control in projection space q Project training data onto hyperplane. . . o Making distance between class means LARGE and the within class scatter small q But LDA how? 6

LDA and Clustering q Again, want projected data to have… o Between-class means that are far apart o Within-class scatter is small (both classes) q Recall that in clustering, we want. . . o Distance between clusters to be large o And each cluster should be compact o Replace “cluster” with “class” in LDA q So, LDA is related to clustering too! 7

Projection Examples q Projecting onto hyperplane q Why is projection (b) better than (a)? LDA 8

LDA Projection q Again, in LDA we maximize separation between means and minimize scatter o In the projection space, not input space q But, why worry about the scatter? q That is, why not just separate means? o Kind of like the opposite of K-means o Would be easier if we forget the scatter o What could possibly go wrong? LDA 9

LDA Projection q In (a), means more widely separated q But in (b), much smaller scatter LDA 10

LDA Training q Want to account for both within-class cohesion and between-class separation o Cohesion is scatter in projection space o Separation based on projected means q LDA uses a fairly simple approach o One expression that is a function of both cohesion and separation q But LDA first, we need some notation 11

Notation q Let X 1, X 2, …, Xm and Y 1, Y 2, . . . , Yn be training vectors of 2 classes q Each vector is of length l q (Scalar) projection of X is q Where LDA vector w is to be determined 12

Mean and Scatter q Class means are o Note μx and μy are vectors q And class scatter defined as o Note sx 2 and sy 2 are scalars, not vectors q Next, want to compute projected means and scatter LDA 13

Projected Mean and Scatter q Mean of X in projection space o And similar for projected mean of Y q Scatter of X in projection space o And similar for projected scatter of Y LDA 14

Projected Means q Consider the function o argmax M(w) is w that has largest distance between projected means q Consider the function o argmin C(w) is w with smallest scatter o argmax 1/C(w) is same as argmin C(w) LDA 15

Fisher Discriminant q In general… o argmax M(w) ≠ argmax 1/C(w) q So, how (or what) to optimize ? q In LDA, we’ll keep it simple q Then argmax J(w) is desired solution q Function J(w) is Fisher Discriminant LDA 16

Fisher Discriminant q Why maximize Fisher Discriminant? q Combines both large separation and small scatter in one simple formula q Have we seen anything similar before? q Reminiscent of silhouette coefficient LDA 17

Example q Which has larger Fisher Discriminant? q Why? LDA 18

Maximizing J(w) q Expanding, we have o Note J(w) defined in projection space q Game plan… o Write J(w) in matrix form o Then maximize resulting matrix function o Then relate result to other methods LDA 19

Matrix Form q Define within-class scatter matrices o Essentially, covariance matrices of X and Y o Total within-class scatter: SW = Sx + Sy q Between-class scatter: q These matrices all defined in input space, not in projection space LDA 20

In Projection Space q But J(w) defined in projection space… q All of matrices on previous slide have analog in projection space o As above, use “� ” for projection space q Define analog of SB as q Can be shown that q Define then LDA 21

Matrix Form q In matrix form, q Let’s maximize! o Note that if vector w = argmax J(w) then αw also works, for any scalar α o So, let’s require w to satisfy w. TSWw = 1 q The problem can now be stated as o Maximize: w. TSBw o Subject to: w. TSWw = 1 LDA 22

Lagrange Multipliers q The problem can now be stated as o Maximize: w. TSBw o Subject to: w. TSWw = 1 q Lagrangian is o L(w, λ) = w. TSBw + λ(w. TSWw – 1) q How to solve? q Partial derivatives, set equal to 0… LDA 23

Lagrange Multipliers q Equivalent version of the problem is o Minimize: - ½ w. TSBw o Subject to: ½ w. TSWw = ½ q In this form, Lagrangian is o L(w, λ) = - ½ w. TSBw + ½ λ(w. TSWw – 1) q Take derivatives wrt w, we find o - SBw + λSWw = 0 o Or SBw = λSWw LDA 24

Lagrange Multipliers q Solution w to LDA training problem satisfies SBw = λSWw q Now, suppose the inverse SW-1 exists, and define S = SW-1 SB q Then desired solution w satisfies o Sw = λw q We see that w is an eigenvector of S o And Lagrange multiplier λ is its eigenvalue LDA 25

LDA and PCA and SVM q Previous slide shows (deep) connection between LDA, PCA, and SVM o That is, Lagrange multipliers (SVM) and eigenvectors (PCA) arise in LDA training q And it gets even better… o Recall, Sw = λw where S = SW-1 SB o Want between-class scatter large (SB) and within-class scatter small (SW-1 large) o So, λ must be largest eigenvalue of S LDA 26

Comparison of LDA and PCA q In PCA, we determine score based on large variances q LDA distinguishes between classes based on means and variances q So, LDA does well if means separated o LDA will not do so well if means are too close, even if variances are distinguishing o Of course, underlying LDA and PCA problems are different LDA 27

Numerical Example q Given the labeled training data… q Want to train LDA o That is, find projection vector w o So that J(w) is max o We’ll solve based on Sw = λw LDA 28

Mean and Scatter Matrices q For given training data, means are q Scatter LDA matrices 29

Matrices q From previous slide, we find q And we find q So that LDA 30

Eigenvectors q We find q Eigenvectors and of S are o Eigenvalues λ 1=0. 8256 and λ 2=0. 0002 LDA 31

Solution? q We have q Just for fun, let’s project training data onto each of w 1 and w 2 q Slope of w 1 is o m 1 = 0. 7826/0. 6225 = 1. 2572 q Slope of w 2 is o m 2 = -0. 4961/0. 8683 = -0. 5713 LDA 32

Projecting Onto Eigenvectors q Comparing LDA eigenvectors w 1 and w 2 33

LDA q Project onto largest eigenvector of S gives best possible result for J(w) q Easy to generalize LDA to more than 2 classes o Vector w replaced by a matrix W, each column of which determines a hyperplane o These hyperplanes partition the space for classification LDA 34

Bottom Line q LDA is useful in its own right q Also interesting because of many connections to other ML techniques o PCA, SVM, clustering, other? o Relates Lagrangian to eigenvectors q LDA generalizes to more classes o QDA is another a generalization of LDA q Nice! LDA 35