Can Machines Transfer Knowledge from Task to Task

Can Machines Transfer Knowledge from Task to Task? Isabelle Guyon Clopinet, California http: //clopinet. com/ul 1 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

CREDITS Data donors: Handwriting recognition (AVICENNA) -- Reza Farrahi Moghaddam, Mathias Adankon, Kostyantyn Filonenko, Robert Wisnovsky, and Mohamed Chériet (Ecole de technologie supérieure de Montréal, Quebec) contributed the dataset of Arabic manuscripts. The toy example (ULE) is the MNIST handwritten digit database made available by Yann Le. Cun and Corinna Costes. Object recognition (RITA) -- Antonio Torralba, Rob Fergus, and William T. Freeman, collected and made available publicly the 80 million tiny image dataset. Vinod Nair and Geoffrey Hinton collected and made available publicly the CIFAR datasets. See the techreport Learning Multiple Layers of Features from Tiny Images, by Alex Krizhevsky, 2009, for details. Human action recognition (HARRY) -- Ivan Laptev and Barbara Caputo collected and made publicly available the KTH human action recognition datasets. Marcin Marszałek, Ivan Laptev and Cordelia Schmid collected and made publicly available the Hollywood 2 dataset of human actions and scenes. Text processing (TERRY) -- David Lewis formatted and made publicly available the RCV 1 -v 2 Text Categorization Test Collection. Ecology (SYLVESTER) -- Jock A. Blackard, Denis J. Dean, and Charles W. Anderson of the US Forest Service, USA, collected and made available the (Forest cover type) dataset. Web platform: Server made available by Prof. Joachim Buhmann, ETH Zurich, Switzerland. Computer admin. : Thomas Fuchs, ETH Zurich. Webmaster: Olivier Guyon, Mister. P. net, France. Platform: Causality Wokbench. Co-orgnizers: • David W. Aha, Naval Research Laboratory, USA. • Gideon Dror, Academic College of Tel-Aviv Yaffo, Israel. • Vincent Lemaire, Orange Research Labs, France. • Graham Taylor, NYU, New-York. USA. • Gavin Cawley, University of east Anglia, UK. • Danny Silver, Acadiau University, Canada. • Vassilis Athitsos, UT Arlington, Texas. , USA. Protocol review and advising: • Olivier Chapelle, Yahoo!, California, USA. • Gerard Rinkus, Brandeis University, USA. • Urs Mueller, Net-Scale Technilogies, USA. • Yoshua Bengio, Universite de Montreal, Canada. • David Grangier, NEC Labs, USA. • Andrew Ng, Stanford Univ. , Palo Alto, California, USA. • Yann Le. Cun, NYU. New-York, USA. • Richard Bowden, University of Surrey, UK. • Philippe Dreuw, Aachen University, Germany. • Ivan Laptev, INRIA, France. • Jitendra Malik, UC Berkeley, USA. • Greg Mori, Simon Fraser University, Canada. • Christian Vogler, ILSP, Athens, Greece 2 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

What is the problem? 3 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Can learning about. . . 4 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

help us learn about… 5 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Can learning about… publicly available data 6 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

help us learn about… personal data Philip and Thomas Philip Anna Martin Solene Bernhard Anna, Thomas and GM Omar, Thomas Philip Thomas Unsupervised and Transfer Learning Challenge 7 http: //clopinet. com/ul

Transfer learning Common data representation Philip and Thomas Philip Anna Martin Solene Bernhard Anna, Thomas and GM Omar, Thomas Philip Thomas Unsupervised and Transfer Learning Challenge 8 http: //clopinet. com/ul

How? 9 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Vocabulary Source task labels Target task labels 10 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Vocabulary Source task labels Target task labels 11 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Vocabulary Labels available? Source task labels Tasks the same? Target task labels Domains the same? 12 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Taxonomy of transfer learning No labels in source domain Self-taught TL Inductive TL Labels available in source domain Labels available in target domain Transfer Learning Labels avail. ONLY in source domain Same source and target task Multi-task TL Transductive TL Semi-supervised TL No labels in both source and target domains Different source and target tasks Unsupervised TL Unsupervised and Transfer Learning Challenge Cross-task TL Adapted from: A survey on transfer learning, Pan-Yang, 2010. 13 http: //clopinet. com/ul

Taxonomy of transfer learning No labels in source domain Self-taught TL Inductive TL Labels available in source domain Labels available in target domain Transfer Learning Labels avail. ONLY in source domain Same source and target task Multi-task TL Transductive TL Semi-supervised TL No labels in both source and target domains Different source and target tasks Unsupervised TL Unsupervised and Transfer Learning Challenge Cross-task TL Adapted from: A survey on transfer learning, Pan-Yang, 2010. 14 http: //clopinet. com/ul

Unsupervised transfer learning 15 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

What can you do with NO labels? • No learning at all: – Normalization of examples or features – Construction of features (e. g. products) – Generic data transformations (e. g. taking the log, Fourier transform, smoothing, etc. ) • Unsupervised learning: – Manifold learning to reduce dimension (and/or orthogonalize features) – Sparse coding to expand dimension – Clustering to construct features – Generative models and latent variable models Unsupervised and Transfer Learning Challenge 16 http: //clopinet. com/ul

Unsupervised transfer learning 1) Source domain P R 17 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Unsupervised transfer learning 1) P 18 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Unsupervised transfer learning 1) P 2) Target domain P C John Task labels 19 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Unsupervised transfer learning Target domain P C Emily 20 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Manifold learning • • PCA ICA Kernel PCA Kohonen maps Auto-encoders MDS, Isomap, LLE, Laplacian Eigenmaps Regularized principal manifolds Unsupervised and Transfer Learning Challenge 21 http: //clopinet. com/ul

Deep Learning Greedy layer-wise unsupervised pre-training of multi-layer neural networks and Bayesian networks, including: • Deep Belief Networks (stacks of Restricted Boltzmann machines) reconstructor • Stacks of auto-encoders preprocessor 22 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Clustering • K-means and variants w. cluster overlap (Gaussian mixtures, fuzzy C-means) • Hierarchical clustering • Graph partitioning • Spectral clustering 23 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Example: K-means • Start with random cluster centers. • Iterate: o Assign the examples to their closest center to form clusters. o Re-compute the centers by averaging the cluster members. • Create features, e. g. fk= exp –g ||x-xk|| Clusters of ULE valid after 5 it. 24 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

ALC=0. 79 log 2(num. tr. ex. ) Raw data: 784 features AUC Results on ULE: do better! ALC=0. 84 log 2(num. tr. ex. ) K-means: 20 features Current best: AUC=1, ALC=0. 96 Unsupervised and Transfer Learning Challenge 25 http: //clopinet. com/ul

Unsupervised learning (resources) • Unsupervised Learning. Z. Ghahramani. http: //www. gatsby. ucl. ac. uk/~zoubin/course 04/ul. pdf • Nonlinear dimensionality reduction. http: //en. wikipedia. org/wiki/Nonlinear_dimensionality_reduction • Out-of-Sample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering. Y. Bengio et al. http: //books. nips. cc/papers/files/nips 16/NIPS 2003_AA 23. pdf • Data Clustering: A Review. Jain et al. http: //citeseerx. ist. psu. edu/viewdoc/summary? doi=10. 1. 1. 18. 2720 • Why Does Unsupervised Pre-training Help DL? D. Erhan et al. http: //jmlr. csail. mit. edu/papers/volume 11/erhan 10 a. pdf • Efficient sparse coding algorithms. H. Lee et al. 26 http: //www. eecs. umich. edu/~honglak/nips 06 sparsecoding. pdf http: //clopinet. com/ul Unsupervised and Transfer Learning Challenge

Taxonomy of transfer learning No labels in source domain Self-taught TL Inductive TL Labels available in source domain Labels available in target domain Transfer Learning Labels avail. ONLY in source domain Same source and target task Multi-task TL Transductive TL Semi-supervised TL No labels in both source and target domains Different source and target tasks Unsupervised TL Unsupervised and Transfer Learning Challenge Cross-task TL Adapted from: A survey on transfer learning, Pan-Yang, 2010. 27 http: //clopinet. com/ul

Cross-task transfer learning 28 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

How can you do it? • Data representation learning: – Deep neural networks – Deep belief networks (re-use the internal representation created by the hidden units and/or output units) • Similarity or kernel learning: – Siamese neural networks – Graph-theoretic methods 29 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Data representation learning 1) Source domain P C Sea Source task labels 30 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Data representation learning 1) P 31 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Data representation learning 1) P 2) Target domain P C John Target task labels 32 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Data representation learning Target domain P C Emily 33 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Kernel learning 1) P S Source domain same or different Source task labels P 34 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Kernel learning 1) P 35 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Kernel learning 1) P 2) Target domain P C John Target task labels 36 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Kernel learning Target domain P C Emily 37 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Cool results in cross-task transfer learning Genuine or not Source task NLP (almost) from scratch. Collobert et al. 2011, submitted to JMLR Unsupervised and Transfer Learning Challenge Target tasks pos=Part-Of-Speech tagging chunk=Chunking ner=Named Entity Recognition srl=Semantic Role Labeling 38 http: //clopinet. com/ul

Cross-task transfer (resources) • A Survey on Transfer Learning. Pan and Yang. http: //www 1. i 2 r. astar. edu. sg/~jspan/publications/TLsurvey_0822. pdf • Distance metric learning: A comprehensive survey. Yang-Jin. http: //citeseerx. ist. psu. edu/viewdoc/summary? doi=10. 1. 1. 91. 47 32 • Signature Verification using a "Siamese" Time Delay Neural Network. Bromley et al. http: //citeseerx. ist. psu. edu/viewdoc/summary? doi=10. 1. 1. 28. 4792 • Learning the kernel matrix with semi-definite programming, Lanckriet et al. http: //jmlr. csail. mit. edu/papers/volume 5/lanckriet 04 a/lanckriet 04 a. pdf 39 • NLP (almost) from scratch. Collobert et al. 2011, http: //clopinet. com/ul Unsupervised and Transfer Learning Challenge http: //leon. bottou. org/morefiles/nlp. pdf.

Taxonomy of transfer learning No labels in source domain Self-taught TL Inductive TL Labels available in source domain Labels available in target domain Transfer Learning Labels avail. ONLY in source domain Same source and target task Multi-task TL Transductive TL Semi-supervised TL No labels in both source and target domains Different source and target tasks Unsupervised TL Unsupervised and Transfer Learning Challenge Cross-task TL Adapted from: A survey on transfer learning, Pan-Yang, 2010. 40 http: //clopinet. com/ul

Multi-task learning 41 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Multi-task learning Source domain P Target domain C Sea John Source task labels Target task labels 42 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Multi-task learning Target domain P C Emily 43 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Cool results in multi-task learning One-Shot Learning with a Hierarchical Nonparametric Bayesian Model, Salakhutdinov-Tenenbaum-Torralba, 2010 44 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Taxonomy of transfer learning No labels in source domain Self-taught TL Inductive TL Labels available in source domain Labels available in target domain Transfer Learning Labels avail. ONLY in source domain Same source and target task Multi-task TL Transductive TL Semi-supervised TL No labels in both source and target domains Different source and target tasks Unsupervised TL Unsupervised and Transfer Learning Challenge Cross-task TL Adapted from: A survey on transfer learning, Pan-Yang, 2010. 45 http: //clopinet. com/ul

Self-taught learning 46 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Self-taught learning Source domain P Target domain C John Target task labels 47 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Self-taught learning Target domain P C Emily 48 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Cool results in self-taught learning Source task Target task Unsupervised Semi-supervised Multi-task Self-taught learning. R. Raina et al. 2007 49 http: //clopinet. com/ul Unsupervised and Transfer Learning Challenge

Inductive transfer learning (resources) • Multitask learning. R. Caruana. http: //www. cs. cornell. edu/~caruana/mlj 97. pdf • Learning deep architectures for AI. Y. Bengio. http: //www. iro. umontreal. ca/~lisa/pointeurs/TR 1312. pdf • Transfer Learning Techniques for Deep Neural Nets. S. M. Gutstein thesis. http: //robust. cs. utep. edu/~gutstein/sg_home_files/thesis. pdf • One-Shot Learning with a Hierarchical Nonparametric Bayesian Model. R. Salakhutdinov et al. http: //dspace. mit. edu/bitstream/handle/1721. 1/60025/ MIT-CSAIL-TR-2010 -052. pdf? sequence=1 • Self-taught learning. R. Raina et al. 50 http: //www. stanford. edu/~rajatr/papers/icml 07_Self. Tau http: //clopinet. com/ul Unsupervised and Transfer Learning Challenge ght. Learning. pdf

Dec 2010 -April 2011 http: //clopinet. com/ul Competitors Development data Validation data Challenge data Source task labels Data representations Validation target task labels Challenge target task labels • Goal: Learning data representations or kernels. • Phase 1: Unsupervised learning (until Feb. 28) • Phase 2: Cross-task transfer learning (from Mar. 1) • Prizes: $6000 + free registrations + travel awards • Dissemination: Workshops at ICML and IJCNN; proc. in JMLR W&CP. Evaluators 51 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

July 2011, ICML - Dec 2011, NIPS http: //clopinet. com/tl Competitors Development Data (source + target data) All task labels Validation data (target only) Challenge data (target only) Predictions Multi-task learning setting: - Synthetic, Real-world - Supervised learning - Binary classification problems. - 5 -10 secondary tasks, 1 primary -Impoverished primary task data in development set -Diversity of tasks with varying degree of relatedness to primary task Target task validation labels Target task challenge labels 52 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Challenge June 2011 -June. 2012 http: //clopinet. com/gs (in preparation) STEP 1: Develop a “generic” sign language recognition system that can learn new signs with a few examples. STEP 2: At conference: teach the system new signs. STEP 3: Live evaluation in front of audience. 53 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul

Conclusion • Transfer learning algorithms offer solutions to problems in which – a lot of training samples are available for a source task, – fewer training samples are available for a similar but different target task. • We stated a program of challenges featuring problems in which transfer learning is applicable. 54 Unsupervised and Transfer Learning Challenge http: //clopinet. com/ul