An Introduction to Kernel Based Learning Algorithms K

An Introduction to Kernel. Based Learning Algorithms K. -R. Muller, S. Mika, G. Ratsch, K. Tsuda and B. Scholkopf Presented by: Joanna Giforos CS 8980: Topics in Machine Learning 9 March, 2006

Outline n n n Problem Description Nonlinear Algorithms in Kernel Feature Space Supervised Learning: q q n Supervised Learning: q n n Nonlinear SVM Kernel Fisher Discriminant Analysis Kernel Principle Component Analysis Applications Model specific kernels

Problem Description n 2 -class Classification: estimate a function, input-output training data such that using will correctly classify unseen examples. n i. e. , Find a mapping: n Assume: Training and test data are drawn from the same probability distribution

Problem Description n n A learning machine is a family of functions For a task of learning two classes f(x, ) 2 {-1, 1}, 8 x, Too complex ) Overfitting Not complex enough ) Underfitting Want to find the right balance between accuracy and complexity

Problem Description n Best is the one that minimizes the expected error: n Empirical Risk (training error): n Remp( )! R( ) as n!1

Structural Risk Minimization n Construct a nested family of function classes with non-decreasing VC dimension. , n Let n SRM chooses the function class and the function such that an upper bound on the generalization error is minimized. be the solutions of the empirical risk minimization.

Nonlinear Algorithms in Kernel Feature Space n n n Via a non-linear mapping the data is mapped into a potentially much higher dimensional feature space. Given this mapping, we can compute scalar products in feature spaces using kernel functions. does not need to be known explicitly ) every linear algorithm that only uses scalar products can implicitly be executed in by using kernels.

Nonlinear Algorithms in Kernel Feature Space: Example

Supervised Learning: Nonlinear SVM n Consider linear classifiers in feature space using dot products. n Conditions for classification without training error: n GOAL: Find and b such that the empirical risk and regularization term are minimized. n But we cannot explicitly access w in the feature space, so we introduce Lagrange multipliers, i, one for each of the above constraints.

Supervised Learning: Nonlinear SVM n Last class we saw that the nonlinear SVM primal problem is: n Which leads to the dual:

Supervised Learning: Nonlinear SVM n n n Using KKT second order optimality conditions on the dual SVM problem, we obtain: The solution is sparse in ) many patterns are outside the margin area and the optimal i’s are zero. Without sparsity, SVM would be impractical for large data sets.

Supervised Learning: Nonlinear SVM n The dual problem can be rewritten as: n Where n Since objective function is convex, every local max is a global max, but there can be several optimal solutions (in terms of i) n Once i’s are found using QP solvers, simply plug into prediction rule:

Supervised Learning: KFD n Discriminant analysis seeks to find a projection of the data in a direction that is efficient for discrimination. Image from: R. O. Duda, P. E. Hart and D. G. Stork, Pattern Classification, John Wiley & Sons, INC. , 2001.

Supervised Learning: KFD n n Solve Fisher’s linear discriminant in kernel feature space. Aims at finding linear projections such that the classes are well separated. q q How far are the projected means apart? (should be large) How big is the variance of the data in this direction? (should be small) n Recall, that this can be achieved by maximizing the Rayleigh quotient: n where

Supervised Learning: KFD n In kernel feature space training patterns: n To get: n Where, , express w in terms of mapped

Supervised Learning: KFD n Projection of a test point onto the discriminant is computed by: n Can solve the generalized eigenvalue problem: n But N and M may be large and non-sparse, can transform KFD into a convex QP problem. n Question – can we use numerical approximations to the eigenvalue problem?

Supervised Learning: KFD n Can reformulate as constrained optimization problem. FD tries to minimize the distance between the variance of the data along the projection whilst maximizing the distance between the means: n This QP is equivalent to J( ) since n q q M is a matrix of rank 1 (columns are linearly dependent) Solutions w in J( ) are invariant under scaling. ) Can fix the distance of the means to some arbitrary, positive value and just minimize the variance.

Connection Between Boosting and Kernel Methods n Can show that Boosting maximizes the smallest margin . n Recall, SVM attempts to maximize w n In general, using an arbitrary lp norm constraint on the weight vector leads to maximizing the lq distance between the hyperplane and the training points. q q Boosting uses l 1 norm SVM uses l 2 norm

Unsupervised Methods: Linear PCA § Principal Components Analysis (PCA) attempts to efficiently represent the data by finding orthonormal axes which maximally decorrelate the data § Given centered observations: n PCA finds the principal axes by diagonalizing the covariance matrix n Note that C is positive definite, and thus can be diagonalized with nonnegative eigenvalues.

Unsupervised Methods: Linear PCA n Eigenvectors lie in the span of x 1, …, xn: n Thus it can be shown that, n But is just a scalar, so all solutions v with 0 lie in the span of x 1, …, xn, i. e.

Unsupervised Methods: Kernel PCA n If we first map the data into another space, n Then assuming we can center the data, we can write the covariance matrix as: n Which can be diagonalized with nonnegative eigenvalues satisfying:

Unsupervised Methods: Kernel PCA n As in linear PCA, all solutions v with 0 lie in the span of (xi), …, (xm) i. e. n Substituting, we get: n Where K is the inner product kernel: n Premultiplying both sides by (xk)T, we finally get:

Unsupervised Methods: Kernel PCA n The resulting set of eigenvectors are then used to extract the Principle Components of a test point by:

Unsupervised Methods: Kernel PCA n Nonlinearities only enter the computation at two points: q q n In the calculation of the matrix K In the evaluation of new points Drawback of PCA: q For large data sets, storage and computational complexity issues. n n Can use sparse approximations of K. Question: Can we think of other unsupervised methods which can make use of kernels? q Kernel k-means, Kernal ICA, Spectral Clustering

Unsupervised Methods: Linear PCA

Unsupervised Methods: Kernel PCA

Applications n Support Vector Machines and Kernel Fisher Discriminant: q q q n Bioinformatics: protein classification OCR Face Recognition Content based image retrieval Decision Tree Predictive Modeling … Kernel PCA q q Denoising Compression Visualization Feature extraction for classification

Kernels for Specific Applications n Image Segmentation: Gaussian weighted 2 -distance between local color histograms. q Can be shown to be robust for color and texture discrimination n Text classification: Vector Space kernels n Structured Data (strings, trees, etc. ): Spectrum kernels n Generative models: P-kernels, Fisher kernels