Convex Optimization in Machine Learning MURI Meeting July

  • Slides: 30
Download presentation
Convex Optimization in Machine Learning MURI Meeting July 2002 Gert Lanckriet (gert@eecs. berkeley. edu)

Convex Optimization in Machine Learning MURI Meeting July 2002 Gert Lanckriet (gert@eecs. berkeley. edu) L. El Ghaoui, M. Jordan, C. Bhattacharrya, N. Cristianini, P. Bartlett U. C. Berkeley

Convex Optimization in Machine Learning

Convex Optimization in Machine Learning

Advanced Convex Optimization in Machine Learning SDP SOCP QCQP LP QP

Advanced Convex Optimization in Machine Learning SDP SOCP QCQP LP QP

Advanced Convex Optimization in Machine Learning

Advanced Convex Optimization in Machine Learning

Linear Programming (LP)

Linear Programming (LP)

Second Order Cone Programming (SOCP)

Second Order Cone Programming (SOCP)

Semi-Definite Programming

Semi-Definite Programming

Advanced Convex Optimization in Machine Learning

Advanced Convex Optimization in Machine Learning

MPM: Problem Sketch (1) a. T z = b : decision hyperplane

MPM: Problem Sketch (1) a. T z = b : decision hyperplane

MPM: Problem Sketch (2)

MPM: Problem Sketch (2)

MPM: Problem Sketch (3) … should be minimized ! Probability of misclassification… … for

MPM: Problem Sketch (3) … should be minimized ! Probability of misclassification… … for worst-case classconditional density…

MPM: Main Result (5)

MPM: Main Result (5)

MPM: Geometric Interpretation

MPM: Geometric Interpretation

Robustness to Estimation Errors: Robust MPM (R-MPM)

Robustness to Estimation Errors: Robust MPM (R-MPM)

Robust MPM (R-MPM)

Robust MPM (R-MPM)

Robust MPM (R-MPM)

Robust MPM (R-MPM)

MPM: Convex Optimization to solve the problem Linear Classifier Nonlinear Classifier Lemma Kernelizing Convex

MPM: Convex Optimization to solve the problem Linear Classifier Nonlinear Classifier Lemma Kernelizing Convex Optimization: Second Order Cone Program (SOCP) ) competitive with Quadratic Program (QP) SVMs

MPM: Empirical results a=1–b and TSA (test-set accuracy) of the MPM, compared to BPB

MPM: Empirical results a=1–b and TSA (test-set accuracy) of the MPM, compared to BPB (best performance in Breiman's report (Arcing classifiers, 1996)) and SVMs. (averages for 50 random partitions into 90% training and 10% test sets) • Comparable with existing literature, SVMs • a=1 -b is indeed smaller than the test-set accuracy in all cases (consistent with b as worst-case bound on probability of misclassification) • Kernelizing leads to more powerfull decision boundaries (alinear decision boundary < anonlinear decision boundary (Gaussian kernel))

Advanced Convex Optimization in Machine Learning

Advanced Convex Optimization in Machine Learning

The idea (1) Machine learning Kernel-based machine learning

The idea (1) Machine learning Kernel-based machine learning

The idea (2)

The idea (2)

The idea (4) training set (labelled) test set (unlabelled)

The idea (4) training set (labelled) test set (unlabelled)

Hard margin SVM classifiers (3)

Hard margin SVM classifiers (3)

Hard margin SVM classifiers (4)

Hard margin SVM classifiers (4)

Hard margin SVM classifiers (5) SDP !

Hard margin SVM classifiers (5) SDP !

Hard margin SVM classifiers (7) training set (labelled) test set (unlabelled) Learning the kernel

Hard margin SVM classifiers (7) training set (labelled) test set (unlabelled) Learning the kernel matrix !

Hard margin SVM classifiers (8) ?

Hard margin SVM classifiers (8) ?

Hard margin SVM classifiers (11) Learning Kernel Matrix with SDP !

Hard margin SVM classifiers (11) Learning Kernel Matrix with SDP !

Empirical results hard margin SVMs

Empirical results hard margin SVMs

See also

See also

Convex Mirrors Convex Mirrors Convex Mirror F andConvex Mirrors Convex Mirrors Convex Mirror F and
CSE 203 B Convex Optimization Lecture 2 ConvexCSE 203 B Convex Optimization Lecture 2 Convex
CSE 203 B Convex Optimization Lecture 3 ConvexCSE 203 B Convex Optimization Lecture 3 Convex
CSE 203 B Convex Optimization Lecture 2 ConvexCSE 203 B Convex Optimization Lecture 2 Convex
CSE 203 B Convex Optimization Lecture 3 ConvexCSE 203 B Convex Optimization Lecture 3 Convex
CSE 203 B Convex Optimization Lecture 2 ConvexCSE 203 B Convex Optimization Lecture 2 Convex
MURI DI SOSTEGNO Prof Francesco OCCHICONE MURI DIMURI DI SOSTEGNO Prof Francesco OCCHICONE MURI DI
MURI LowLevel Control Biomimetic Robots HighLevel Control MURIMURI LowLevel Control Biomimetic Robots HighLevel Control MURI
MURI DI SOSTEGNO Prof Francesco OCCHICONE MURI DIMURI DI SOSTEGNO Prof Francesco OCCHICONE MURI DI
MURI Robot Fabrication LowLevel Control HighLevel MURI ControlMURI Robot Fabrication LowLevel Control HighLevel MURI Control
Machine Learning Machine Learning Machine learningstatistical learning artificialMachine Learning Machine Learning Machine learningstatistical learning artificial
convex concave convex concave Eigenfaces PhotobookEigenfaces MIT Mediaconvex concave convex concave Eigenfaces PhotobookEigenfaces MIT Media
Convex Hull obstacle start end Convex Hull 1Convex Hull obstacle start end Convex Hull 1
Convex hull smallest convex set containing all theConvex hull smallest convex set containing all the
computing CONVEX HULLS Presentation Outline 2 D Convexcomputing CONVEX HULLS Presentation Outline 2 D Convex
Convex Hull 20121023 Convex vs Concave A polygonConvex Hull 20121023 Convex vs Concave A polygon
CONVEX LENSES 18 1 Lenses n A convexCONVEX LENSES 18 1 Lenses n A convex
Convex Lens in Water Convex Lens in WaterConvex Lens in Water Convex Lens in Water
Convex Hull for Dynamic Data Convex Hull andConvex Hull for Dynamic Data Convex Hull and
CONVEX LENS Focus Convex lens is thicker inCONVEX LENS Focus Convex lens is thicker in
Convex Hull obstacle start 362021 end Convex HullConvex Hull obstacle start 362021 end Convex Hull
CONVEX MIRROR A convex mirror or diverging mirrorCONVEX MIRROR A convex mirror or diverging mirror
Unit 8 Lesson 7 Convex Lenses concave convexUnit 8 Lesson 7 Convex Lenses concave convex
Seeing Things in Convex Mirrors 1 Convex MirrorSeeing Things in Convex Mirrors 1 Convex Mirror