STAT 541 Multivariate Analysis Kernels in Multivariate Analysis

Today’s Agenda • • Inner products Kernels Hilbert space Reproducing kernel map RKHS: Reproducing

Inner products • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW

Concept check: Find imposters • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall

Kernels (or kernel functions) • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall

Examples of Kernels • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013

Positive (semi-)definite kernels • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013

A space exploration • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013

• Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021

Example: Gaussian RK map • Source: Learning with Kernels… Smola A. J. 2001 Stat

Dimension reduction Feature Space , … PCA, CCA, LDA Kernel methods : k. PCA,

Kernel trick • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW

Kernel trick • Source: Kernel Methods for Pattern Analysis, John Shawe, 2004 Stat 541,

Mercer condition • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW

Mercer theorem • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW

PCA vs k. PCA • Source: Learning with. Kernels SVM, … by A. J.

k. PCA example Fig 1 • Fig 3 Fig 2 Source: Wikipedia, Author: Petter

PCA: Review • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW

Kernel PCA (k. PCA) • Source: Wikipedia, Author: Petter Strandmark Stat 541, Multivariate Analysis

Slides: 22

Download presentation

STAT 541 Multivariate Analysis Kernels in Multivariate Analysis Oleg Melnikov Rice University Department of Statistics Fall 2013 RW

Today’s Agenda • • Inner products Kernels Hilbert space Reproducing kernel map RKHS: Reproducing Kernel Hilbert Space Mercer Theorem (without proof) Kernel Trick Kernel PCA Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 2

Inner products • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 3

Inner products • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 4

Concept check: Find imposters • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 5

Kernels (or kernel functions) • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 6

Examples of Kernels • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 7

Positive (semi-)definite kernels • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 8

A space exploration • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 9

• Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 10

• Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 11

Example: Gaussian RK map • Source: Learning with Kernels… Smola A. J. 2001 Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 12

• Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 13

Dimension reduction Feature Space , … PCA, CCA, LDA Kernel methods : k. PCA, k. CCA, k. LD A , … • PCA, LDA, CCA: feature extraction and dimension reduction – Note: dimension reduction is in feature space, not the input space • So, the model uses fewer features, not observed variables Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 14

Kernel trick • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 15

Kernel trick • Source: Kernel Methods for Pattern Analysis, John Shawe, 2004 Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 12/17/2021 RW 16

Mercer condition • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 17

Mercer theorem • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 18

PCA vs k. PCA • Source: Learning with. Kernels SVM, … by A. J. Smola, 2001 Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 19

k. PCA example Fig 1 • Fig 3 Fig 2 Source: Wikipedia, Author: Petter Strandmark Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 20

PCA: Review • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 21

Kernel PCA (k. PCA) • Source: Wikipedia, Author: Petter Strandmark Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 12/17/2021 22