Spectral Clustering Eyal David Image Processing seminar May

Spectral Clustering Eyal David Image Processing seminar May 2008

Lecture Outline l l l Motivation Graph overview and construction Demo Spectral Clustering Demo Cool implementations 2

3 A Tutorial on Spectral ClusteringArik Azran

Spectral Clustering Example – 2 Spirals Dataset exhibits complex cluster shapes Þ K-means performs very poorly in this space due bias toward dense spherical clusters. In the embedded space given by two leading eigenvectors, clusters are trivial to separate. Spectral Clustering - Derek Greene 4

Lecture Outline l l l Motivation Graph overview and construction Graph demo Spectral Clustering demo Cool implementation 5

6 Matthias Hein and Ulrike von Luxburg August 2007

7 Matthias Hein and Ulrike von Luxburg August 2007

8 Matthias Hein and Ulrike von Luxburg August 2007

9 Matthias Hein and Ulrike von Luxburg August 2007

10 Matthias Hein and Ulrike von Luxburg August 2007

11 Matthias Hein and Ulrike von Luxburg August 2007

12 Matthias Hein and Ulrike von Luxburg August 2007

13 Matthias Hein and Ulrike von Luxburg August 2007

14 Matthias Hein and Ulrike von Luxburg August 2007

15 Matthias Hein and Ulrike von Luxburg August 2007

16 Matthias Hein and Ulrike von Luxburg August 2007

17 Matthias Hein and Ulrike von Luxburg August 2007

Demo (Live example)

Lecture Outline l l l Motivation Graph overview and construction Demo Spectral Clustering Demo Cool implementations 19

20 Matthias Hein and Ulrike von Luxburg August 2007

21 Matthias Hein and Ulrike von Luxburg August 2007

22 Matthias Hein and Ulrike von Luxburg August 2007

23 Matthias Hein and Ulrike von Luxburg August 2007

24 Matthias Hein and Ulrike von Luxburg August 2007

25 Matthias Hein and Ulrike von Luxburg August 2007

26 Matthias Hein and Ulrike von Luxburg August 2007

27 Matthias Hein and Ulrike von Luxburg August 2007

28 Matthias Hein and Ulrike von Luxburg August 2007

Eigenvectors & Eigenvalues 29

30 Matthias Hein and Ulrike von Luxburg August 2007

31 Matthias Hein and Ulrike von Luxburg August 2007

Demo (Live example)

Spectral Clustering Algorithm Ng, Jordan, and Weiss l Motivation l Given a set of points l We would like to cluster them into k subsets 33 Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm

Algorithm l l Form the affinity matrix Define if l l Scaling parameter chosen by user Define D a diagonal matrix whose (i, i) element is the sum of A’s row i 34 Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm

Algorithm l Form the matrix l Find , the k largest eigenvectors of L These form the columns of the new matrix X l l Note: have reduced dimension from nxn to nxk 35 Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm

Algorithm l Form the matrix Y l Renormalize each of X’s rows to have unit length l l Y Treat each row of Y as a point in Cluster into k clusters via K-means 36 Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm

Algorithm l Final Cluster Assignment l Assign point to cluster j iff row i of Y was assigned to cluster j 37 Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm

Why? l If we eventually use K-means, why not just apply K-means to the original data? l This method allows us to cluster non-convex regions 38 Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm

l Some Examples 39

40

41

42

43

44

45

46

47

User’s Prerogative l l Affinity matrix construction Choice of scaling factor l l l Realistically, search over gives the tightest clusters and pick value that Choice of k, the number of clusters Choice of clustering method 48 Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm

How to select k? l Eigengap: the difference between two consecutive eigenvalues. l Most stable clustering is generally given by the value k that maximises the expression Largest eigenvalues of Cisi/Medline data λ 1 Þ Choose k=2 λ 2 49 Spectral Clustering - Derek Greene

Recap – The bottom line 50 Matthias Hein and Ulrike von Luxburg August 2007

Summary l l Spectral clustering can help us in hard clustering problems The technique is simple to understand The solution comes from solving a simple algebra problem which is not hard to implement Great care should be taken in choosing the “starting conditions” 51

The End