Manifold Estimation Local and NonLocal Parametrized Tangent Learning

Manifold Estimation: Local (and Non-Local Parametrized) Tangent Learning Yavor Tchakalov Advanced Machine Learning Course by Dr. Jebara Fall 2006 Columbia University

Motivation Importance of amount of training data l Classical solutions l l Regularizers l A-priori l knowledge embedding Concept of tangent vectors l Compact representation of transformation invariance

Two classificaiton approaches l Learning techniques: building a model l Adjust a number of parameters to compute classification function l Memory-based techniques: l Training examples are stored in memory l New pattern => stored prototypes l Label is produced

Naïve Approach l Combine a l l l training dataset representing the input space simple distance measure: Euclidean dist Result l l prohibitively large prototype set poor accuracy

Classical Solution l Feature extractor l Compute representation that is minimally affected by certain transformations l Major bottleneck in classification l Invariant “true” distance measure l Deformable prototypes l Must know allowed transformations l Deformation search is expensive / unreliable

Transformation Manifold l l For instance: simple 16 x 16 grayscale image => 256 -D space Transformation Manifolds: l l l Dimensionality Non-linearity (i. e. geometric transformations of gray level image) Implications Solution: approximate the manifold by a tangent hyerplance at the prototype tangent distance: truly invariant w. r. t. transformations used to define manifolds

Tangent hyperplane Hyperplane fully defined by original prototype (α=0) and first derivate of transformation Tayler’s expansion of transformation around α=0

Tangent Distance l l Compute the minimum distance between tangent hyperplanes that approximate transformation manifolds (hence invariant to these transformations) Three benefits: l l l Linear subspaces: simple analytical expressions can be computed and stored Minimal distance is a simple least-squares problem Distance is locally invariant but not glodablly invariant

Illustration

Implementation l Prototype approximation l Distance: linear least squares optimization problem

Illustration (I)

Illustration (II)

Results l Implementation caveats l l Pre-computation of tangent vectors Smoothing

Non-local Parameterized Tangent Learning l Problems with a large class of local manifold learning Nyström’s formula vector differences of neighbours l l l Instances of such problems Non-local learning: minimize relative projection error Goal: Transduction