Locally Linear Support Vector Machines ubor Ladick Classification

Locally Linear Support Vector Machines Ľubor Ladický Classification task Locally Linear SVMs • The decision boundary should be smooth • Approximately linear in a sufficiently small region Given : { [x 1, y 1], [x 2, y 2], … [xn, yn] } xi - feature vector Philip H. S. Torr yi {-1, 1} - label Extension to Finite Kernels • The finite kernel classifier takes the form : • Weights W and b obtained as : The task is : To design y = H(x) that predicts a label y for a new vector x • Many approaches proposed (Boosting, Random Forests, Neural Nets, SVMs, . . ) • SVM seems to dominate the others Experiments Linear vs. Kernel SVMs • MNIST, LETTER & USPS datasets Kernel SVMs Linear SVMs • Fast training and evaluation • Applicable to large scale data sets • Low discriminative power • Slow training and evaluation • Not feasible for large scale data sets • Better performance for hard problems Motivation : • Anchor points obtained using K-means clustering • Coordinates evaluated on k. NN (k = 5) • Coordinates evaluated on k. NN (k = 8) (slow part) • Approximated Intersection kernel used • Coordinates obtained using weighted inverse distance • Spatial pyramid of BOW features • Curvature is bounded • Functions wi(x) and b(x) are Lipschitz • wi(x) and b(x) can be approximated using local coding • Raw data used • MNIST • Exploiting manifold learning techniques to design an SVM with • Good trade-off between the performance and speed • Scalability to large scale data sets (solvable with stochastic gradient descent) Local coding for manifold learning • Weights W and biases b are obtained as : • Points approximated as a weighted sum of anchor points • USPS / LETTER where v – anchor points γ v(x) - coordinates • Coordinates obtained using : • Distance based methods (Gemert et al. 2008, Zhou et al. 2009) • Reconstruction methods (Roweis et al. 2000, Yu et al. 2009, Gao et al. 2010) • For a normalised coding any Lipschitz function f can be approximated : (Yu et al. 2009) • CALTECH-101 (15 samples per class) [xk, yk] – training samples λ- regularisation weight S – number of samples Relation to Other Models • Generalisation of Linear SVM on x : • representable by W = (w w. . )T and b = (b’ b’. . )T • Generalisation of linear SVM on γ: • representable by W = 0 and b = w • Generalisation of model selecting Latent(MI) SVM : • Representable by γ= (0, 0, . . , 1, . . 0) • CALTECH-101 • Coordinates evaluated based on image histograms