Ch 7 Sparse Kernel Machines Pattern Recognition and

Contents l Maximum Margin Classifiers ¨ Overlapping Class Distributions ¨ Relation to Logistic Regression

Maximum Margin Classifiers l Problem settings ¨ Two-classification using linear models ¨ Assume that

Maximum Margin Solution For all data points, l The distance of a point to

Dual Representation l Introducing Lagrange multipliers, Find Appendix E for more details ¨ Min.

Classifying New Data l l Optimization subjects to Karush-Kuhn-Tucker (KKT) Conditions Appendix E or

Example of Separable Data Classification Figure 7. 2 (C) 2007, SNU Biointelligence Lab, http:

Overlapping Class Distributions Allow some misclassified examples soft margin l Introduce slack variables l

Soft Margin Solution l Minimize : trade-off between minimizing training errors and controlling model

Dual Representation l Dual representation l Classifying new data and obtaining b ( hard

Alternative Formulation l v-SVM (Schölkopf et al. , 2000) - Upper bound on the

Example of Nonseparable Data Classification (v-SVM) (C) 2007, SNU Biointelligence Lab, http: //bi. snu.

Solutions of the QP Problem l Chunking (Vapnik, 1982) ¨ Idea: the value of

Relation to Logistic Regression (Section 4. 3. 2) For data points on the correct

Relation to Logistic Regression (Cont’d) l From maximum likelihood logistic regression l Error function

Comparison of Error Functions Hinge error function Misclassification error Error function for logistic regression

Multiclass SVMs l One-versus-the-rest: K separate SVMs ¨ Can lead inconsistent results (Figure 4.

SVMs for Regression Simple linear regression: minimize l ε-insensitive error function quadratic error function

SVMs for Regression (Cont’d) l Minimize l (C) 2007, SNU Biointelligence Lab, http: //bi.

Dual Problem l l (C) 2007, SNU Biointelligence Lab, http: //bi. snu. ac. kr/

Predictions l l (from derivatives of the Lagrangian) ¨ KKT conditions: ¨ (C) 2007,

$Alternative Formulation l v-SVM (Schölkopf et al. , 2000) fraction of points lying outside$

Example of v-SVM Regression (C) 2007, SNU Biointelligence Lab, http: //bi. snu. ac. kr/

Relevance Vector Machines l SVM ¨ Outputs are decisions rather than posterior probabilities ¨

RVM for Regression RVM is a linear form in Chapter 3 with a modified

RVM for Regression (Cont’d) From the result (3. 49) for linear regression models l

RVM for Regression (Cont’d) l Two approaches ¨ By derivatives of marginal likelihood ¨

Example of RVM Regression RVM regression v-SVM regression More compact than SVM l Parameters

Mechanism for Sparsity l only isotropic noise, α = ∞ a finite value of

Sparse Solution l Pull out the contribution from αi in l Using (C. 7),

Sparse Solution (Cont’d) l For log marginal likelihood function L, Sparsity: measures the extent

Sequential Sparse Bayesian Learning Algorithm 1. Initialize 2. Initialize using , with , with

RVM for Classification Probabilistic linear classification model (Chapter 4) l with ARD prior l

RVM for Classification (Cont’d) l The posterior distribution is obtained by maximizing Iterative reweighted

RVM for Classification (Cont’d) l Marginal likelihood using Laplace approximation (Section 4. 4) l

Example of RVM Classification (C) 2007, SNU Biointelligence Lab, http: //bi. snu. ac. kr/

Slides: 36

Download presentation

Ch 7. Sparse Kernel Machines Pattern Recognition and Machine Learning, C. M. Bishop, 2006. Summarized by S. Kim Biointelligence Laboratory, Seoul National University http: //bi. snu. ac. kr/

Contents l Maximum Margin Classifiers ¨ Overlapping Class Distributions ¨ Relation to Logistic Regression ¨ Multiclass SVMs ¨ SVMs for Regression l Relevance Vector Machines ¨ RVM for Regression ¨ Analysis of Sparsity ¨ RVMs for Classification (C) 2007, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ 2

Maximum Margin Classifiers l Problem settings ¨ Two-classification using linear models ¨ Assume that training data set is linearly separable l Support vector machine approaches ¨ The decision boundary is chosen to be the one for which the margin is maximized support vectors (C) 2007, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ 3

Dual Representation l Introducing Lagrange multipliers, Find Appendix E for more details ¨ Min. points satisfy the derivatives of L w. r. t. w and b equal 0 l Dual representation (C) 2007, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ 5

Classifying New Data l l Optimization subjects to Karush-Kuhn-Tucker (KKT) Conditions Appendix E or l : support vectors Found by solving a quadratic programming problem O(N 3) (C) 2007, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ 6

Soft Margin Solution l Minimize : trade-off between minimizing training errors and controlling model complexity l ¨ KKT conditions: or (C) 2007, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ : support vectors 9

Alternative Formulation l v-SVM (Schölkopf et al. , 2000) - Upper bound on the fraction of margin errors - Lower bound on the fraction of support vectors (C) 2007, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ 11

Solutions of the QP Problem l Chunking (Vapnik, 1982) ¨ Idea: the value of Lagrangian is unchanged if we remove the rows and columns of the kernel matrix corresponding to Lagrange multipliers that have value zero ¨ Protected conjugate gradients (Burges, 1998) Decomposition methods (Osuna et al. , 1996) l Sequential minimal optimization (Platt, 1999) l (C) 2007, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ 13

Relation to Logistic Regression (Section 4. 3. 2) For data points on the correct side, For the remaining points, l l : hinge error function (C) 2007, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ 14

Multiclass SVMs l One-versus-the-rest: K separate SVMs ¨ Can lead inconsistent results (Figure 4. 2) ¨ Imbalanced training sets < Positive class: +1, negative class: -1/(K-1) (Lee et al. , 2001) l An objective function for training all SVMs simultaneously (Weston and Watkins, 1999) One-versus-one: K(K-1)/2 SVMs l Based on error-correcting output codes (Allwein et al. , 2000) l ¨ Generalization of the voting scheme of the one-versus-one (C) 2007, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ 17

$Alternative Formulation l v-SVM (Schölkopf et al. , 2000) fraction of points lying outside$

Relevance Vector Machines l SVM ¨ Outputs are decisions rather than posterior probabilities ¨ The extension to K>2 classes is problematic ¨ There is a complexity parameter C ¨ Kernel functions are centered on training data points and required to be positive definite l RVM ¨ Bayesian sparse kernel technique ¨ Much sparser models ¨ Faster performance on test data (C) 2007, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ 24

RVM for Regression (Cont’d) From the result (3. 49) for linear regression models l l α and β are determined using evidence approximation (type-2 maximum likelihood) (Section 3. 5) Maximize (C) 2007, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ 26

RVM for Regression (Cont’d) l Two approaches ¨ By derivatives of marginal likelihood ¨ EM algorithm Section 9. 3. 4 l Predictive distribution Section 3. 3. 2 (C) 2007, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ 27

Example of RVM Regression RVM regression v-SVM regression More compact than SVM l Parameters are determined automatically l Require more training time than SVM l (C) 2007, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ 28

Sparse Solution (Cont’d) l For log marginal likelihood function L, Sparsity: measures the extent to which overlaps with the other basis vectors Quality of : represents a measure of the alignment of the basis vector with the error between t and y-i l Stationary points of the marginal likelihood w. r. t. αi (C) 2007, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ 31

Sequential Sparse Bayesian Learning Algorithm 1. Initialize 2. Initialize using , with , with the remaining 3. Evaluate and for all basis functions 4. Select a candidate 5. If ( is already in the model), update 6. If , add to the model, and evaluate 7. If , remove from the model, and set 8. Update 9. Go to 3 until converged (C) 2007, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ 32

RVM for Classification Probabilistic linear classification model (Chapter 4) l with ARD prior l l - Initialize - Build a Gaussian approximation to the posterior distribution - Obtain an approximation to the marginal likelihood - Maximize the marginal likelihood (re-estimate ) until converged (C) 2007, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ 33

RVM for Classification (Cont’d) l The posterior distribution is obtained by maximizing Iterative reweighted least squares (IRLS) from Section 4. 3. 3 Resulting Gaussian approximation to the posterior distribution (C) 2007, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ 34

RVM for Classification (Cont’d) l Marginal likelihood using Laplace approximation (Section 4. 4) l Set the derivative of the marginal likelihood equal to zero, and rearranging then gives l If we define , Same in the regression case (C) 2007, SNU Biointelligence Lab, http: //bi. snu. ac. kr/ 35

Related presentations