Classification via Mathematical Programming Based Support Vector Machines

Classification via Mathematical Programming Based Support Vector Machines Glenn M. Fung November 26, 2002 Computer Sciences Dept. University of Wisconsin - Madison

Outline of Talk v (Standard) Support vector machines (SVM) u Classify by halfspaces v Proximal support vector machines (PSVM) u Classify by proximity to planes u Numerical experiments v Incremental PSVM classifiers u Synthetic dataset consisting of 1 billion points in 10 dimensional input space classified in less than 2 hours and 26 minutes seconds v Knowledge based linear SVMs u Incorporating knowledge sets into a classifier u Numerical experiments

Support Vector Machines Maximizing the Margin between Bounding Planes A+ ASupport vectors

Standard Support Vector Machine Algebra of 2 -Category Linearly Separable Case v Given m points in n dimensional space v Represented by an m-by-n matrix A v Membership of each in class +1 or – 1 specified by: v An m-by-m diagonal matrix D with +1 & -1 entries v Separate by two bounding planes, v More succinctly: where e is a vector of ones.

Standard Support Vector Machine Formulation v Solve the quadratic program for some min (QP) s. t. where : , denotes or , membership. v Margin is maximized by minimizing

Proximal Vector Machines (KDD 2002) Fitting the Data using two parallel Bounding Planes A+ A-

PSVM Formulation We have from the QP SVM formulation: min (QP) s. t. Solving for in terms of and gives: min This simple, but critical modification, changes the nature of the optimization problem tremendously!!

Advantages of New Formulation v Objective function remains strongly convex v An explicit exact solution can be written in terms of the problem data v PSVM classifier is obtained by solving a single system of linear equations in the usually small dimensional input space v Exact leave-one-out-correctness can be obtained in terms of problem data

Linear PSVM We want to solve: min v. Setting the gradient equal to zero, gives a nonsingular system of linear equations. v. Solution of the system gives the desired PSVM classifier

Linear PSVM Solution Here, v The linear system to solve depends on: which is of the size v is usually much smaller than

Linear Proximal SVM Algorithm Input Define Calculate Solve Classifier:

Nonlinear PSVM Formulation v Linear PSVM: (Linear separating surface: min ) (QP) s. t. By QP “duality”, . Maximizing the margin in the “dual space” , gives: min v Replace min by a nonlinear kernel :

The Nonlinear Classifier v The nonlinear classifier: v Where K is a nonlinear kernel, e. g. : v Gaussian (Radial Basis) Kernel : Ø The -entry of of data points and represents the “similarity”

Nonlinear PSVM v Similar to the linear case, setting the gradient equal to zero, we obtain: Defining slightly different: v Here, the linear system to solve is of the size However, reduced kernels techniques can be used (RSVM) to reduce dimensionality.

Non. Linear Proximal SVM Algorithm Input Define Calculate Solve Classifier:

Linear & Nonlinear PSVM MATLAB Code function [w, gamma] = psvm(A, d, nu) % PSVM: linear and nonlinear classification % INPUT: A, d=diag(D), nu. OUTPUT: w, gamma % [w, gamma] = psvm(A, d, nu); [m, n]=size(A); e=ones(m, 1); H=[A -e]; v=(d’*H)’ %v=H’*D*e; r=(speye(n+1)/nu+H’*H)v % solve (I/nu+H’*H)r=v w=r(1: n); gamma=r(n+1); % getting w, gamma from r

Linear PSVM Comparisons with Other SVMs Much Faster, Comparable Correctness Data Set mxn PSVM SSVM Ten-fold test % Time (sec. ) WPBC (60 mo. ) 110 x 32 68. 5 0. 02 68. 5 0. 17 62. 7 3. 85 Ionosphere 351 x 34 87. 3 0. 17 88. 7 1. 23 88. 0 2. 19 Cleveland Heart 297 x 13 85. 9 0. 01 86. 2 0. 70 86. 5 1. 44 Pima Indians 768 x 8 77. 5 0. 02 77. 6 0. 78 76. 4 37. 00 BUPA Liver 345 x 6 69. 4 0. 02 70. 0 0. 78 69. 5 6. 65 Galaxy Dim 4192 x 14 93. 5 0. 34 95. 0 5. 21 94. 1 28. 33

Linear PSVM vs LSVM 2 -Million Dataset Over 30 Times Faster Dataset Method Training Correctness % Testing Correctness % Time Sec. NDC “Easy” LSVM 90. 86 91. 23 658. 5 PSVM 90. 80 91. 13 20. 8 LSVM 69. 80 69. 44 655. 6 PSVM 69. 84 69. 52 20. 6 NDC “Hard”

Nonlinear PSVM: Spiral Dataset 94 Red Dots & 94 White Dots

Nonlinear PSVM Comparisons Data Set mxn PSVM SSVM Ten-fold test % Time (sec. ) LSVM Ten-fold test % Time (sec. ) Ionosphere 351 x 34 95. 2 4. 60 95. 8 25. 25 95. 8 14. 58 BUPA Liver 345 x 6 73. 6 4. 34 73. 7 20. 65 73. 7 30. 75 Tic-Tac-Toe 958 x 9 98. 4 74. 95 98. 4 395. 30 94. 7 350. 64 Mushroom * 8124 x 22 88. 0 35. 50 88. 8 307. 66 87. 8 503. 74 * A rectangular kernel was used of size 8124 x 215

Conclusion v PSVM is an extremely simple procedure for generating linear and nonlinear classifiers v PSVM classifier is obtained by solving a single system of linear equations in the usually small dimensional input space for a linear classifier v Comparable test set correctness to standard SVM v Much faster than standard SVMs : typically an order of magnitude less.

Incremental PSVM Classification (Second SIAM Data Mining Conference) v Suppose we have two “blocks” of data and v The linear system to solve depends on the compressed blocks: which are of the size

Linear Incremental Proximal SVM Algorithm Initialization Read from disk Compute and Store in memory Update in memory Yes No Discard: Compute output Keep:

Linear Incremental Proximal SVM Adding – Retiring Data v Capable of modifying an existing linear classifier by both adding and retiring data v Option of retiring old data is similar to adding new data u Financial Data: old data is obsolete v Option of keeping old data and merging it with the new data: u Medical Data: old data does not obsolesce.

Numerical experiments One-Billion Two-Class Dataset v Synthetic dataset consisting of 1 billion points in 10 dimensional input space v Generated by NDC (Normally Distributed Clustered) dataset generator v. Dataset divided into 500 blocks of 2 million points each. v. Solution obtained in less than 2 hours and 26 minutes v About 30% of the time was spent reading data from disk. v. Testing set Correctness 90. 79%

Numerical Experiments Simulation of Two-month 60 -Million Dataset v Synthetic dataset consisting of 60 million points (1 million per day) in 10 - dimensional input space v Generated using NDC v At the beginning, we only have data corresponding to the first month v Every day: u The oldest block of data is retired (1 Million) u A new block is added (1 Million) u A new linear classifier is calculated daily v Only an 11 by 11 matrix is kept in memory at the end of each day. All other data is purged.

Numerical experiments Separator changing through time

Numerical experiments Normals to the separating hyperplanes Corresponding to 5 day intervals

Conclusion v Proposed algorithm is an extremely simple procedure for generating linear classifiers in an incremental fashion for huge datasets. v The linear classifier is obtained by solving a single system of linear equations in the small dimensional input space. v The proposed algorithm has the ability to retire old data and add new data in a very simple manner. v Only a matrix of the size of the input space is kept in memory at any time

Support Vector Machines Linear Programming Formulation v Use the 1 -norm instead of the 2 -norm: min s. t. v This is equivalent to the following linear program: min s. t.

Conventional Data-Based SVM -15 x'w= g +1 + A -20 x'w= g -25 -30 x'w= g -1 - -35 A -40 -45 -20 -15 -10 -5 0 5

Knowledge-Based SVM via Polyhedral Knowledge Sets (NIPS 2002) -15 {x | B 1 x £ b 1 } + A -20 -25 x'w= g -1 -30 x'w= g +1 -35 - A -40 2 2 {x | C x £ c } 1 1 {x | C x £ c } -45 -20 -15 -10 -5 0 5

Incoporating Knowledge Sets Into an SVM Classifier v Suppose that the knowledge set: belongs to the class A+. Hence it must lie in the halfspace : v We therefore have the implication: v Will show that this implication is equivalent to a set of constraints that can be imposed on the classification problem.

Knowledge Set Equivalence Theorem

Proof of Equivalence Theorem ( Via Nonhomogeneous Farkas or LP Duality) Proof: By LP Duality:

Knowledge-Based SVM Classification

Knowledge-Based SVM Classification v Adding one set of constraints for each knowledge set to the 1 -norm SVM LP, we have:

Parametrized Knowledge-Based LP

Numerical Testing The Promoter Recognition Dataset v Promoter: Short DNA sequence that precedes a gene sequence. v A promoter consists of 57 consecutive DNA nucleotides belonging to {A, G, C, T}. v Important to distinguish between promoters and nonpromoters v This distinction identifies starting locations of genes in long uncharacterized DNA sequences.

The Promoter Recognition Dataset Numerical Representation v Simple “ 1 of N” mapping scheme for converting nominal attributes into a real valued representation: v Not most economical representation, but commonly used.

The Promoter Recognition Dataset Numerical Representation v Feature space mapped from 57 -dimensional nominal space to a real valued 57 x 4=228 dimensional space. 57 nominal values 57 x 4 =228 binary values

Promoter Recognition Dataset Prior Knowledge Rules v Prior knowledge consist of the following 64 rules:

Promoter Recognition Dataset Sample Rules where denotes position of a nucleotide, with respect to a meaningful reference point starting at position and ending at position Then:

The Promoter Recognition Dataset Comparative Algorithms v KBANN Knowledge-based artificial neural network [Shavlik et al] v BP: Standard back propagation for neural networks [Rumelhart et al] v O’Neill’s Method Empirical method suggested by biologist O’Neill [O’Neill] v NN: Nearest neighbor with k=3 [Cost et al] v ID 3: Quinlan’s decision tree builder[Quinlan] v SVM 1: Standard 1 -norm SVM [Bradley et al]

The Promoter Recognition Dataset Comparative Test Results

Wisconsin Breast Cancer Prognosis Dataset Description of the data v 110 instances corresponding to 41 patients whose cancer had recurred and 69 patients whose cancer had not recurred v 32 numerical features v The domain theory: two simple rules used by doctors:

Wisconsin Breast Cancer Prognosis Dataset Numerical Testing Results v Doctor’s rules applicable to only 32 out of 110 patients. v Only 22 of 32 patients are classified correctly by this rule (20% Correctness). v KSVM linear classifier applicable to all patients with correctness of 66. 4%. v Correctness comparable to best available results using conventional SVMs. v KSVM can get classifiers based on knowledge without using any data.

Conclusion v Prior knowledge easily incorporated into classifiers through polyhedral knowledge sets. v Resulting problem is a simple LP. v Knowledge sets can be used with or without conventional labeled data. v In either case KSVM is better than most knowledge based classifiers.

Breast Cancer Treatment Response Joint with Exon. Hit ( French Bio. Tech) v 35 patients treated by a drug cocktail v 9 partial responders; 26 nonresponders v 25 gene expression measurements made on each patient v 1 -Norm SVM classifier selected: 12 out of 25 genes v. Combinatorially selected 6 genes out of 12 v. Separating plane obtained: 2. 7915 T 11 + 0. 13436 S 24 -1. 0269 U 23 -2. 8108 Z 23 -1. 8668 A 19 -1. 5177 X 05 +2899. 1 = 0. v Leave-one-out-error: 1 out of 35 (97. 1% correctness)

Other papers: v A fast and Global Two Point Low Storage Optimization Technique for Tracing Rays in 2 D and 3 D Isotropic Media (Journal of Applied Geophysics) v Semi-Supervised Support Vector Machines for Unlabeled data Classification (Optimization Methods and Software) v Select a small subset of an unlabeled dataset to be labeled by an oracle or expert v Use the new labeled data and the remaining unlabeled data to train a SVM clasifier

Other papers: v Multicategory Proximal SVM Classifiers v Fast multicategory algorithm based on PSVM v Newton refinement step proposed v Data Selection for SVM Classifiers (KDD 2000) v. Reduce the number of support vectors of a linear SVM v Minimal Kernel Classifiers (JMLR) v Use a concave minimization formulation to reduce the SVM model complexity. v Useful for online testing where testing time is an issue.

Other papers: v A Feature Selection Newton Method for SVM Classification v LP SVM solved using a Newton method v Very sparse solutions are obtained v Finite Newton method for Lagrangian SVM Classifiers (Neurocomputing Journal) v Very fast performance, specially when n>m