STAT 541 Multivariate Analysis Kernels in Multivariate Analysis

  • Slides: 22
Download presentation
STAT 541 Multivariate Analysis Kernels in Multivariate Analysis Oleg Melnikov Rice University Department of

STAT 541 Multivariate Analysis Kernels in Multivariate Analysis Oleg Melnikov Rice University Department of Statistics Fall 2013 RW

Today’s Agenda • • Inner products Kernels Hilbert space Reproducing kernel map RKHS: Reproducing

Today’s Agenda • • Inner products Kernels Hilbert space Reproducing kernel map RKHS: Reproducing Kernel Hilbert Space Mercer Theorem (without proof) Kernel Trick Kernel PCA Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 2

Inner products • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW

Inner products • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 3

Inner products • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW

Inner products • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 4

Concept check: Find imposters • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall

Concept check: Find imposters • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 5

Kernels (or kernel functions) • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall

Kernels (or kernel functions) • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 6

Examples of Kernels • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013

Examples of Kernels • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 7

Positive (semi-)definite kernels • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013

Positive (semi-)definite kernels • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 8

A space exploration • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013

A space exploration • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 9

• Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021

• Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 10

• Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021

• Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 11

Example: Gaussian RK map • Source: Learning with Kernels… Smola A. J. 2001 Stat

Example: Gaussian RK map • Source: Learning with Kernels… Smola A. J. 2001 Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 12

• Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021

• Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 13

Dimension reduction Feature Space , … PCA, CCA, LDA Kernel methods : k. PCA,

Dimension reduction Feature Space , … PCA, CCA, LDA Kernel methods : k. PCA, k. CCA, k. LD A , … • PCA, LDA, CCA: feature extraction and dimension reduction – Note: dimension reduction is in feature space, not the input space • So, the model uses fewer features, not observed variables Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 14

Kernel trick • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW

Kernel trick • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 15

Kernel trick • Source: Kernel Methods for Pattern Analysis, John Shawe, 2004 Stat 541,

Kernel trick • Source: Kernel Methods for Pattern Analysis, John Shawe, 2004 Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 12/17/2021 RW 16

Mercer condition • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW

Mercer condition • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 17

Mercer theorem • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW

Mercer theorem • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 18

PCA vs k. PCA • Source: Learning with. Kernels SVM, … by A. J.

PCA vs k. PCA • Source: Learning with. Kernels SVM, … by A. J. Smola, 2001 Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 19

k. PCA example Fig 1 • Fig 3 Fig 2 Source: Wikipedia, Author: Petter

k. PCA example Fig 1 • Fig 3 Fig 2 Source: Wikipedia, Author: Petter Strandmark Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 20

PCA: Review • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW

PCA: Review • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 21

Kernel PCA (k. PCA) • Source: Wikipedia, Author: Petter Strandmark Stat 541, Multivariate Analysis

Kernel PCA (k. PCA) • Source: Wikipedia, Author: Petter Strandmark Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 22

STAT 541 Multivariate Analysis Kernels in Multivariate AnalysisSTAT 541 Multivariate Analysis Kernels in Multivariate Analysis
Stat One Stat One Stat One Stat OneStat One Stat One Stat One Stat One
Multivariate Analysis 1 Multivariate Analysis n n MultivariateMultivariate Analysis 1 Multivariate Analysis n n Multivariate
Methoden der Psychologie Multivariate Analysemethoden Multivariate Distanz MultivariateMethoden der Psychologie Multivariate Analysemethoden Multivariate Distanz Multivariate
On Kernels Margins and Lowdimensional Mappings or KernelsOn Kernels Margins and Lowdimensional Mappings or Kernels
Marginalized Kernels Graph Kernels Max Planck Institute forMarginalized Kernels Graph Kernels Max Planck Institute for
NLP Text Similarity Text Kernels Unigram Kernels SameNLP Text Similarity Text Kernels Unigram Kernels Same
Discrete Kernels Kernels for Sequences Similarity between sequencesDiscrete Kernels Kernels for Sequences Similarity between sequences
stat grammar Expr g prog stat stat exprstat grammar Expr g prog stat stat expr
Glyco POST HELM Stat One Stat One StatGlyco POST HELM Stat One Stat One Stat
Multivariate Data Analysis Chapter 6 Multivariate Analysis ofMultivariate Data Analysis Chapter 6 Multivariate Analysis of
Business Research Methods Multivariate Analysis Multivariate Statistical AnalysisBusiness Research Methods Multivariate Analysis Multivariate Statistical Analysis
Multivariate Analysis Introduction What is Multivariate Analysis TheMultivariate Analysis Introduction What is Multivariate Analysis The
Multivariate Analysis Introduction What is Multivariate Analysis TheMultivariate Analysis Introduction What is Multivariate Analysis The
Multivariate Analysis Introduction What is Multivariate Analysis TheMultivariate Analysis Introduction What is Multivariate Analysis The
Multivariate Analysis Techniques Multivariate Analysis Techniques All statisticalMultivariate Analysis Techniques Multivariate Analysis Techniques All statistical
EVDA 541 ARST 541 MAPPING MATERIAL DIMENSIONS MACHINEDRAWINGMACHINESEVDA 541 ARST 541 MAPPING MATERIAL DIMENSIONS MACHINEDRAWINGMACHINES
CIS 541 Numerical Methods Mathematical Preliminaries OSUCIS 541CIS 541 Numerical Methods Mathematical Preliminaries OSUCIS 541
Multivariate Linear Regression Chapter 8 1 Multivariate AnalysisMultivariate Linear Regression Chapter 8 1 Multivariate Analysis
Applied Multivariate Analysis Introduction 1 Nature of MultivariateApplied Multivariate Analysis Introduction 1 Nature of Multivariate
Multivariate Statistical Analysis Multivariate Normal Distribution Jenpei LiuMultivariate Statistical Analysis Multivariate Normal Distribution Jenpei Liu