Transfer Learning for Image Classification Transfer Learning Approaches

Transfer Learning for Image Classification

Transfer Learning Approaches Leverage data from related tasks to improve performance: q Improve generalization. q Reduce the run-time of evaluating a set of classifiers. Two Main Approaches: q Learning Shared Hidden Representations. q Sharing Features.

Sharing Features: efficient boosting procedures for multiclass object detection Antonio Torralba Kevin Murphy William Freeman

Snapshot of the idea Goal: q Reduce the computational cost of multiclass object recognition q Improve generalization performance Approach: q Make boosted classifiers share weak learners q Some Notation:

Training a single boosted classifier q Consider training a single boosted classifier: Candidate weak learners Weighted stumps Fit an additive model

Training a single boosted classifier Minimize the exponential loss Greedy Approach Gentle Boosting

Standard Multiclass Case: No Sharing Additive model for each class Minimize the sum of exponential losses Each class has its own set of weak learners:

Multiclass Case: Sharing Features subset of classes corresponding additive classifier for the k class At iteration t add one weak learner to one of the additive models: Minimize the sum of exponential losses Naive approach: Greedy Heuristic:

Sharing features for multiclass object detection Torralba, Murphy, Freeman. CVPR 2004

Learning efficiency

Sharing features shows sub-linear scaling of features with objects (for area under ROC = 0. 9). Red: shared features Blue: independent features

How the features are shared across objects Basic features: Filter responses computed at different locations of the image

Uncovering Shared Structures in Multiclass Classification Yonatan Amit Michael Fink Nathan Srebro Shimon Ullman

Structure Learning Framework Linear Classifiers Linear Transformations Class Parameters Structural Parameters q Find optimal parameters:

Multiclass Loss Function q Hinge Loss: q Maximal Hinge Loss:

Snapshot of the idea q Main Idea: Enforce Sharing by finding low rank parameter matrix W Transformation on x Transformation on w q Consider the m by d parameter matrix: q Can be factored: q Rows of theta form a basis

Low Rank Regularization Penalty q Rank of a d by m matrix is the smallest z, such that: q A regularization penalty designed to minimize the rank of W’ would tend to produce solutions where a few basis are shared by all classes. q Minimizing the rank would lead to a hard combinatorial problem q Instead use a trace norm penalty: eigen value of W’

Putting it all together No longer in the objective q For optimization they use a gradient based method that minimizes a smooth approximation of the objective

Mammals Dataset

Results

Transfer Learning for Image Classification via Sparse Joint Regularization Ariadna Quattoni Michael Collins Trevor Darrell

Training visual classifiers when a few examples are available q Problem: q Image classification from a few examples can be hard. q A good representation of images is crucial. q Solution: q We learn a good image representation using: unlabeled data + labeled data from related problems

Snapshot of the idea: q Use unlabeled dataset + kernel function to compute a new representation: q q q Complex features, high dimensional space Some of them will be very discriminative (hopefully) Most will be irrelevant q If we knew the relevant features we could learn from fewer examples. q Related problems may share relevant features. q We can use data from related problems to discover them !!

Semi-supervised learning Step 1: Learn representation Visual Representation Large dataset of unlabeled data Unsupervised learning Step 2: Train Classifier h-dimensional training set Small training set of labeled images Compute Train Classifier

Semi-supervised learning: Raina et al. [ICML 2007] proposed an approach that learns a sparse set of high level features (i. e. linear combinations of the original features) from unlabeled data using a sparse coding technique. q q Balcan et al. [ML 2004] proposed a representation based on computing kernel distances to unlabeled data points.

Learning visual representations using unlabeled data only q q q Unsupervised learning in data space Good thing: q Lower dimensional representation preserves relevant statistics of the data sample. Bad things: q The representation might still contain irrelevant features, i. e. features that are useless for classification.

Learning visual representations from unlabeled data + labeled data from related categories Step 1: Learn a representation Large Dataset of unlabeled images Kernel Function Labeled images from related categories Discriminative representation Create New Representation Select Discriminative Features of the New Representation

Our contribution q Main differences with previous approaches: q Our choice of joint regularization norm allows us to express the joint loss minimization as a linear program (i. e. no need for greedy approximations. ) q While previous approaches build joint sparse classifiers on the feature space our method discovers discriminative features in a space derived using the unlabeled data and uses these discriminative features to solve future problems.

Overview of the method Step I: Use the unlabeled data to compute a new representation space [Kernel SVD]. q q Step II: Use the labeled data from related problems to discover discriminative features in the new space [Joint Sparse Regularization]. q Step III: Compute the new discriminative representation for the samples of the target problem. q Step IV: Train the target classifier using the representation of step III.

Step I: Compute a representation using the unlabeled data q Perform Kernel SVD on the Unlabeled Data. q A) Compute kernel matrix of unlabeled images: U K q B) Compute a projection matrix A by taking all the eigen vectors of K.

Step I: Compute a representation using the unlabeled data q C) Project labeled data from related problems to the new space: D q Notational shorthand:

Sidetrack q Another possible method for learning a representation from the unlabeled data would be to create a projection matrix Q by taking the h eigen vectors of A corresponding to the h largest eigenvalues. q We will call this approach the Low Rank Baseline q Our method differs significantly from the low rank approach in that we use training data from related problems to select discriminative features in the new space.

Step II: Discover relevant features by joint sparse approximation q A classifier is a function: where: and is the input space, i. e. the representation learnt from unlabeled data. is a binary label, in our application is going to be 1 if an image belongs to a particular topic and -1 otherwise. q A loss function:

Step II: Discover relevant features by joint sparse approximation q Consider learning a single sparse linear classifier (on the space learnt from the unlabeled data) of the form: q A sparse model will have only a few features with non-zero coefficients. q A natural choice for parameter estimation would be: Classification error L 1 penalizes non-sparse solutions q Donoho [2004] has proven (in a regression setting) that the solution with smallest L 1 norm is also the sparsest solution.

Step II: Discover relevant features by joint sparse approximation q Goal: Find a subset of features R such that each problem in: can be well approximated by a sparse classifier whose non-zero coefficients correspond to features in R. q Solution : Regularized Joint loss minimization: Classification error on training set k penalizes solutions that utilize too many features

Step II: Discover relevant features by joint sparse approximation q How do we penalize solutions that use too many features? Coefficients for feature 2 Coefficients for classifier 2 q Problem : not a proper norm, would lead to a hard combinatorial problem.

Step II: Discover relevant features by joint sparse approximation q Instead of using the #non-zero-rows pseudo-norm we will use a convex relaxation [Tropp 2006] q This norm combines: An L 1 norm on the maximum absolute values of the coefficients promotes sparsity on max values. Use few features The L∞ norm on each row promotes non-sparsity on the rows Share features q The combination of the two norms results in a solution where only a few features are used but the features used will contribute in solving many classification problems.

Step II: Discover relevant features by joint sparse approximation q Using the L 1 - L∞ norm we can rewrite our objective function as: q For any convex loss this is a convex function, in particular when considering the hinge loss: the optimization problem can be expressed as a linear program.

Step II: Discover relevant features by joint sparse approximation q Linear program formulation ( hinge loss): q Objective: q Max value constraints: and q Slack variables constraints: and

Step III: Compute the discriminative features representation q Define the set of relevant features to be: q Create a new representation by taking all the features in x’ corresponding to the indexes in R

Experiments: Dataset Reuters Dataset 10382 images, 108 topics. Predict 10 most frequent topics; (binary prediction) Data Partitions 3000 unlabeled images. 2382 images as testing data. 5000 images as source of supervised training data labeled training sets of sizes: 1, 5, 10 , 15… 50. Training set with n positive examples and 2*n

Dataset Super. Bowl [341] Sharon [321] Golden globes [167] Grammys [170] Danish Cartoons Australian open [178] [209] Figure skating Academy Awards [146] [135] Trapped coal miners [196] Iraq [ 125]

Baseline Representation q ‘Bag of words’ representation that combines: color, texture and raw local image information q Sampling: § Sample image patches on a fixed grid § For each image patch compute: üColor features based on HSV color histograms üTexture features based on mean responses of Gabor filters at different scales and orientations üRaw features, normalized pixel values q Create visual dictionary: for each feature type we do vector quantization and create a dictionary V of 2000 visual words.

Baseline representation q Compute baseline representation: q Sample image patches over a fix grid q For every feature type map each patch to its closest visual word in the corresponding dictionary. q The final representation is given by: where is the number of times that an image patch was mapped to the i-th color word

Setting Step 1: Learn a representation using the unlabeled datataset and labeled datatasets from 9 topics. q q Step 2: Train a classifier for the 10 th held out topic using the learnt representation. q As evaluation we use the equal error rate averaged over the 10 topics.

Experiments q Three models, all linear SVMs q Baseline model (RFB): q. Uses raw representation. q Low Rank baseline (LRB): q Uses as a representation: where Q is consists of the h highest eigenvectors of the matrix A computed in the first step of the algorithm q Sparse Transfer Model (SPT) q Uses Representation computed by our algorithm q For both LRB and SPT we used and RBF kernel when computing the Representation from unlabeled data.

Results:

Results: Mean Equal Error rate per topic for classifiers trained with five positive examples; for the RFB model and the SPT model. Super. Bowl; GG: Golden Globes; DC: Danish Cartoons; Gr: Grammys; AO: Australian Open; Sh: Sharon; FS: Figure Skating; AA: Academy Awards; Ir: Iraq.

Results

Conclusion q Summary: q We described a method for learning discriminative sparse image representations from unlabeled images + images from related tasks. q The method is based on learning a representation from the unlabeled data and performing joint sparse approximation on the data from related tasks to find a subset of discriminative features. q The induced representation improves performance when learning with very few examples.

Future work q Combine different representations q Develop an efficient algorithm for solving the joint optimization that will scale to very large datasets.

Joint Sparse Approximation q. Discovers image representations which improve learning with few examples q. The LP formulation is feasible for small problems but becomes intractable for larger data-sets.

Outline n A Joint Sparse Approximation Model for Multi-task Learning n An Efficient Algorithm n Experiments

Joint Sparse Approximation as a Constrained Convex Optimization Problem A convex function Convex constraints q We will use a Projected Sub. Gradient method. Main advantages: simple, scalable, guaranteed convergence rates. q Projected Sub. Gradient methods have been recently proposed: q L 2 regularization, i. e. SVM [Shalev-Shwartz et al. 2007] q L 1 regularization [Duchi et al. 2008]

Euclidean Projection into the L 1 -∞ ball q Projection: 1 2 Inspecting the Lagrangian shows that it can be reduced to: q We reduced the projection to finding new maximums for each feature across tasks and using them to truncate A. q The total mass removed from a feature across tasks should be the same for all features whose coefficients don’t vanish.

Euclidean Projection into the L 1 -∞ ball 3 Features, 6 problems, Q=14 Regret t 1 t 2 t 3 t 4 t 5 t 6 Feature III

Euclidean Projection into the L 1 -∞ ball

An Efficient Algorithm For Finding μ q Recall that we need to find a vector of maximums µ that sums to Q and a constant θ such that the mass loss for every feature is the same. q Consider the mass loss function and its inverse: This is a piece wise linear function So is its inverse q Consider a function that takes a mass loss and computes the corresponding sum of µ ( the norm of the new matrix) This is also piece wise linear q We can construct this function and pick the θ for which

Complexity q The total cost of the algorithm is dominated by the initial sort of each row of A: q The merge of these rows to build the piece wise linear function: q The total cost is in the order of: q Notice that we only need to consider non-zero rows of A, so d is really the number of non-zero rows rather than the actual number of features

Outline n A Joint Sparse Approximation Model for Multi-task Learning n An Efficient Algorithm n Experiments

Synthetic Experiments q All tasks consisted of predicting functions of the form: q To generate jointly sparse vectors we randomly selected 10% of the features to be the relevant feature set V. Then for each task we randomly selected a subset v and zeroed all parameters outside v. q We use the same projected subgradient method, and compared three different types of projection steps: q L 1−∞ projection q L 2 projection q L 1 projection

Synthetic Experiments Test Error Performance on predicting relevant features

Dataset: News story prediction Super. Bowl Golden globes Sharon Grammys Danish Australian Cartoons open Figure skating Academy Awards Trapped coal miners Iraq q 40 tasks q Raw image representation: Bag of visual words q Linear Kernel q 3000 dimensions

Image Classification Experiments

Conclusion and Future Work q Presented a simple and effective algorithm for training joint models with L 1−∞ constraints. q The algorithm scales linearly with the number of examples and O(dm log(dm) ) with the number of problems and dimensions. q The experiments on a real image dataset show that this algorithm can find solutions that are jointly sparse, resulting in lower test error. q We believe our approach can be extended to work on an online multi-task learning setting.

Future Work: Online Multitask Classification [Cavallanti et al. 2008] q There are m binary classification tasks indexed by 1 , …. , M q At each time step t=1, 2, …, T the learner receives a task index k and the corresponding instance vector. q Based on this information the learner outputs a binary prediction an then observes the correct label yk q We are interested in comparing the learner’s mistake count to that of the optimal predictors:

n Thanks!

Lagrangian

Lagrangian