STAT 541 Multivariate Analysis Kernels in Multivariate Analysis

  • Slides: 22
Download presentation
STAT 541 Multivariate Analysis Kernels in Multivariate Analysis Oleg Melnikov Rice University Department of

STAT 541 Multivariate Analysis Kernels in Multivariate Analysis Oleg Melnikov Rice University Department of Statistics Fall 2013 RW

Today’s Agenda • • Inner products Kernels Hilbert space Reproducing kernel map RKHS: Reproducing

Today’s Agenda • • Inner products Kernels Hilbert space Reproducing kernel map RKHS: Reproducing Kernel Hilbert Space Mercer Theorem (without proof) Kernel Trick Kernel PCA Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 11/28/2020 2

Inner products • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW

Inner products • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 11/28/2020 3

Inner products • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW

Inner products • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 11/28/2020 4

Concept check: Find imposters • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall

Concept check: Find imposters • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 11/28/2020 5

Kernels (or kernel functions) • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall

Kernels (or kernel functions) • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 11/28/2020 6

Examples of Kernels • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013

Examples of Kernels • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 11/28/2020 7

Positive (semi-)definite kernels • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013

Positive (semi-)definite kernels • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 11/28/2020 8

A space exploration • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013

A space exploration • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 11/28/2020 9

• Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 11/28/2020

• Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 11/28/2020 10

• Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 11/28/2020

• Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 11/28/2020 11

Example: Gaussian RK map • Source: Learning with Kernels… Smola A. J. 2001 Stat

Example: Gaussian RK map • Source: Learning with Kernels… Smola A. J. 2001 Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 11/28/2020 12

• Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 11/28/2020

• Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 11/28/2020 13

Dimension reduction Feature Space , … PCA, CCA, LDA Kernel methods : k. PCA,

Dimension reduction Feature Space , … PCA, CCA, LDA Kernel methods : k. PCA, k. CCA, k. LD A , … • PCA, LDA, CCA: feature extraction and dimension reduction – Note: dimension reduction is in feature space, not the input space • So, the model uses fewer features, not observed variables Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 11/28/2020 14

Kernel trick • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW

Kernel trick • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 11/28/2020 15

Kernel trick • Source: Kernel Methods for Pattern Analysis, John Shawe, 2004 Stat 541,

Kernel trick • Source: Kernel Methods for Pattern Analysis, John Shawe, 2004 Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 11/28/2020 RW 16

Mercer condition • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW

Mercer condition • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 11/28/2020 17

Mercer theorem • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW

Mercer theorem • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 11/28/2020 18

PCA vs k. PCA • Source: Learning with. Kernels SVM, … by A. J.

PCA vs k. PCA • Source: Learning with. Kernels SVM, … by A. J. Smola, 2001 Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 11/28/2020 19

k. PCA example Fig 1 • Fig 3 Fig 2 Source: Wikipedia, Author: Petter

k. PCA example Fig 1 • Fig 3 Fig 2 Source: Wikipedia, Author: Petter Strandmark Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 11/28/2020 20

PCA: Review • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW

PCA: Review • Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 11/28/2020 21

Kernel PCA (k. PCA) • Source: Wikipedia, Author: Petter Strandmark Stat 541, Multivariate Analysis

Kernel PCA (k. PCA) • Source: Wikipedia, Author: Petter Strandmark Stat 541, Multivariate Analysis Oleg Melnikov, Rice University, Fall 2013 RW 11/28/2020 22