Convex Optimization in Machine Learning MURI Meeting July

Convex Optimization in Machine Learning MURI Meeting July 2002 Gert Lanckriet (gert@eecs. berkeley. edu) L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett U. C. Berkeley

Convex Optimization in Machine Learning

Advanced Convex Optimization in Machine Learning SDP SOCP QCQP LP QP

Advanced Convex Optimization in Machine Learning

Linear Programming (LP)

Second Order Cone Programming (SOCP)

Semi-Definite Programming

Advanced Convex Optimization in Machine Learning

MPM: Problem Sketch (1) a. T z = b : decision hyperplane

MPM: Problem Sketch (2)

MPM: Problem Sketch (3) … should be minimized ! Probability of misclassification… … for worst-case classconditional density…

MPM: Main Result (5)

MPM: Geometric Interpretation

Robustness to Estimation Errors: Robust MPM (R-MPM)

Robust MPM (R-MPM)

Robust MPM (R-MPM)

MPM: Convex Optimization to solve the problem Linear Classifier Nonlinear Classifier Lemma Kernelizing Convex Optimization: Second Order Cone Program (SOCP) ) competitive with Quadratic Program (QP) SVMs

MPM: Empirical results a=1–b and TSA (test-set accuracy) of the MPM, compared to BPB (best performance in Breiman's report (Arcing classifiers, 1996)) and SVMs. (averages for 50 random partitions into 90% training and 10% test sets) • Comparable with existing literature, SVMs • a=1 -b is indeed smaller than the test-set accuracy in all cases (consistent with b as worst-case bound on probability of misclassification) • Kernelizing leads to more powerfull decision boundaries (alinear decision boundary < anonlinear decision boundary (Gaussian kernel))

Advanced Convex Optimization in Machine Learning

The idea (1) Machine learning Kernel-based machine learning

The idea (2)

The idea (4) training set (labelled) test set (unlabelled)

Hard margin SVM classifiers (3)

Hard margin SVM classifiers (4)

Hard margin SVM classifiers (5) SDP !

Hard margin SVM classifiers (7) training set (labelled) test set (unlabelled) Learning the kernel matrix !

Hard margin SVM classifiers (8) ?

Hard margin SVM classifiers (11) Learning Kernel Matrix with SDP !

Empirical results hard margin SVMs

See also