CISC 667 Intro to Bioinformatics Fall 2005 Support

CISC 667 Intro to Bioinformatics (Fall 2005) Support Vector Machines I CISC 667, F 05, Lec 22, Liao 1

Terminologies • An object x is represented by a set of m attributes xi, 1 i m. • A set of n training examples S = { (x 1, y 1), …, (xn, yn)}, where yi is the classification (or label) of instance xi. – For binary classification, yi ={ 1, +1}, and for k-classification, yi ={1, 2, …, k}. – Without loss of generality, we focus on binary classification. • The task is to learn the mapping: xi yi • A machine is a learned function/mapping/hypothesis h: xi h(xi , ) where stands for parameters to be fixed during training. • Performance is measured as E = (1/2 n) i=1 to n |yi- h(xi , )| CISC 667, F 05, Lec 22, Liao 2

Linear SVMs: find a hyperplane (specified by normal vector w and perpendicular distance b to the origin) that separates the positive and negative examples with the largest margin. Margin w · xi + b > 0 if yi = +1 + w · xi + b < 0 if yi = 1 An unknown x is classified as sign(w · x + b) w b Separating hyperplane (w, b) Origin CISC 667, F 05, Lec 22, Liao 3

Rosenblatt’s Algorithm (1956) ; // is the learning rate w 0 = 0; b 0 = 0; k = 0 R = max 1 i n || xi || error = 1; // flag for misclassification/mistake while (error) { // as long as modification is made in the for-loop error = 0; for (i = 1 to n) { if (yi ( <wk · xi> + bk ) 0 ){ // misclassification wk+1 = wk + yi xi // update the weight bk+1 = bk + yi R 2 // update the bias k = k +1 error = 1; } } } return (wk, bk) // hyperplane that separates the data, where k is the number of // modifications. CISC 667, F 05, Lec 22, Liao 4

Questions w. r. t. Rosenblatt’s algorithm – Is the algorithm guaranteed to converge? – How quickly does it converge? Novikoff Theorem: Let S be a training set of size n and R = max 1 i n || xi ||. If there exists a vector w* such that ||w*|| = 1 and yi (w* · xi) , for 1 i n, then the number of modifications before convergence is at most (R/ )2. CISC 667, F 05, Lec 22, Liao 5

Proof: 1. wt· w* = wt-1 · w* + yi xi · w* wt-1 · w* + wt· w* t • || wt ||2 = || wt-1 ||2 + 2 yi xi · wt-1 + 2 || xi ||2 || wt-1 ||2 + 2 R 2 || wt ||2 t 2 R 2 • t R ||w*|| wt · w* t t (R/ )2. Note: – – – Without loss of generality, the separating plane is assumed to pass the origin, i. e. , no bias b is necessary. The learning rate seems to have no bearing on this upper bound. What if the training data is not linearly separable, i. e. , w* does not exist? CISC 667, F 05, Lec 22, Liao 6

Dual form • The final hypothesis w is a linear combination of the training points: w = i=1 to n i yixi where i are positive values proportional to the number of times misclassification of xi has caused the weight to be updated. • Vector can be considered as alternative representation of the hypothesis; i can be regarded as an indication of the information content of the example xi. • The decision function can be rewritten as h(x) = sign (w · x + b) = sign( ( j=1 to n j yjxj) · x + b) = sign( j=1 to n j yj (xj· x) + b) CISC 667, F 05, Lec 22, Liao 7

Rosenblatt’s Algorithm in dual form = 0; b = 0 R = max 1 i n || xi || error = 1; // flag for misclassification while (error) { // as long as modification is made in the for-loop error = 0; for (i = 1 to n) { if (yi ( j=1 to n j yj (xj· xi) + b) 0 ){ // xi is misclassified i = i + 1 // update the weight b = b + yi R 2 // update the bias error = 1; } } } return ( , b) // hyperplane that separates the data, where k is the number of // modifications. Notes: – The training examples enter the algorithm as dot products (xj· xi). – i is a measure of information content; xi with non-zero information content ( i >0) are called support vectors, as they are located on the boundaries. CISC 667, F 05, Lec 22, Liao 8

=0 Margin + >0 w b Separating hyperplane (w, b) Origin CISC 667, F 05, Lec 22, Liao 9

Larger margin is preferred: • converge more quickly • generalize better CISC 667, F 05, Lec 22, Liao 10

w · x+ + b = + 1 w · x- + b = 1 2 = [ (x+ · w ) - (x- · w ) ] = (x+ - x-) · w = || x+ - x- || ||w|| Therefore, maximizing the geometric margin || x+ - x- || is equivalent to minimizing ½ ||w||2, under linear constraints: yi (w · xi) +b 1 for i = 1, …, n. Min w, b < w · w > subject to yi <w · xi> +b 1 for i = 1, …, n CISC 667, F 05, Lec 22, Liao 11

Optimization with constraints Min w, b < w · w > subject to yi <w · xi> +b 1 for i = 1, …, n Lagrangian Theory: Introducing Lagrangian multiplier i for each constraint L(w, b, )= ½ ||w||2 i (yi (w · xi + b) 1), and then calculating L L L ------ = 0, w b This optimization problem can be solved as Quadratic Programming. … guaranteed to converge to the global minimum because of its being a convex Note: advantages over the artificial neural nets CISC 667, F 05, Lec 22, Liao 12

The optimal w* and b* can be found by solving the dual problem for to maximize: L( ) = i ½ i j yi yj xi · xj under the constraints: i 0, and i yi = 0. Once is solved, w* = i yi xi b* = ½ (max y =-1 w*· xi + min y=+1 w*· xi ) And an unknown x is classified as sign(w* · x + b*) = sign( i yi xi · x + b*) Notes: 1. 2. 3. Only the dot product for vectors is needed. Many i are equal to zero, and those that are not zero correspond to xi on the boundaries – support vectors! In practice, instead of sign function, the actual value of w* · x + b* is used when its absolute value is less than or equal to one. Such a value is called a discriminant. CISC 667, F 05, Lec 22, Liao 13

Non-linear mapping to a feature space Φ( ) L( ) = i ½ i j yi yj Φ (xi )·Φ (xj ) xi Φ(xj) xj CISC 667, F 05, Lec 22, Liao Φ(xi) 14

Nonlinear SVMs Input Space X= Feature Space x 1 x 2 CISC 667, F 05, Lec 22, Liao 15

Kernel function for mapping • For input X= (x 1, x 2), K(X, Y) = (X) · (Y) Define map (X) = (x 1 x 1, = (x 1 x 1, 2 x 1 x 2, x 2) · (y 1 y 1, 2 y 1 y 2, y 2) 2 x 1 x 2, x 2 x 2). • Define Kernel function as = (x 1 x 1 y 1 y 1 + 2 x 1 x 2 y 1 y 2 + x 2 x 2 y 2 y 2) 2 K(X, Y) = (X·Y). • It has K(X, Y) = (X) · = (x 1 y 1 + x 2 y 2) (Y) 2 = ((x , x ) · (y , y )) 1 2 • We can compute the dot product in feature space = (X·Y)2 without computing . CISC 667, F 05, Lec 22, Liao 16

Kernels Given a mapping Φ( ) from the space of input vectors to some higher dimensional feature space, the kernel K of two vectors xi, xj is the inner product of their images in the feature space, namely, K(xi, xj) = Φ (xi)·Φ (xj ). Since we just need the inner product of vectors in the feature space to find the maximal margin separating hyperplane, we use the kernel in place of the mapping Φ( ). Because inner product of two vectors is a measure of the distance between the vectors, a kernel function actually defines the geometry of the feature space (lengths and angles), and implicitly provides a similarity measure for objects to be classified. CISC 667, F 05, Lec 22, Liao 17

Mercer’s condition Since kernel functions play an important role, it is important to know if a kernel gives dot products (in some higher dimension space). For a kernel K(x, y), if for any g(x) such that g(x)2 dx is finite, we have K(x, y)g(x)g(y) dx dy 0, then there exist a mapping such that K(x, y) = (x) · (y) Notes: 1. 2. Mercer’s condition does not tell how to actually find . Mercer’s condition may be hard to check since it must hold for every g(x). CISC 667, F 05, Lec 22, Liao 18

More kernel functions some commonly used generic kernel functions – Polynomial kernel: K(x, y) = (1+x·y)p – Radial (or Gaussian) kernel: K(x, y) = exp(-||x-y||2/2 2) Questions: By introducing extra dimensions (sometimes infinite), we can find a linearly separating hyperplane. But how can we be sure such a mapping to a higher dimension space will generalize well to unseen data? Because the mapping introduces flexibility for fitting the training examples, how to avoid overfitting? Answer: Use the maximum margin hyperplane. (Vapnik theory) CISC 667, F 05, Lec 22, Liao 19

The optimal w* and b* can be found by solving the dual problem for to maximize: L( ) = i ½ i j yi yj K(xi , xj) under the constraints: i 0, and i yi = 0. Once is solved, w* = i yi xi b* = ½ max y =-1 [K(w*, xi) + min y=+1 (w*· xi )] And an unknown x is classified as sign(w* · x + b*) = sign( i yi K(xi , x) + b*) CISC 667, F 05, Lec 22, Liao 20

References and resources • Cristianini & Shawe-Tayor, “An introduction to Support Vector Machines”, Cambridge University Press, 2000. • www. kernel-machines. org – SVMLight – Chris Burges, A tutorial • J. -P Vert, A 3 -day tutorial • W. Noble, “Support vector machine applications in computational biology”, Kernel Methods in Computational Biology. B. Schoelkopf, K. Tsuda and J. -P. Vert, ed. MIT Press, 2004. CISC 667, F 05, Lec 22, Liao 21