Machine Learning Spectral Clustering Eric Xing Lecture 8
- Slides: 34
Machine Learning Spectral Clustering Eric Xing Lecture 8, August 13, 2010 Reading: Eric Xing © Eric Xing @ CMU, 2006 -2010
Data Clustering 2 Eric Xing © Eric Xing @ CMU, 2006 -2010
3 Eric Xing © Eric Xing @ CMU, 2006 -2010
Data Clustering l Two different criteria l Compactness, e. g. , k-means, mixture models l Connectivity, e. g. , spectral clustering Connectivity Compactness 4 Eric Xing © Eric Xing @ CMU, 2006 -2010
Spectral Clustering Data Similarities 5 Eric Xing © Eric Xing @ CMU, 2006 -2010
Weighted Graph Partitioning l i Some graph terminology l Wij j Objects (e. g. , pixels, data points) iÎ I = vertices of graph G i l Edges (ij) = pixel pairs with Wij > 0 l Similarity matrix W = [ Wij ] l Degree di = SjÎG Wij d. A = SiÎA di degree of A l Eric Xing G Assoc(A, B) = SiÎA SjÎB Wij © Eric Xing @ CMU, 2006 -2010 A A B 6
Cuts in a Graph l (edge) cut = set of edges whose removal makes a graph disconnected l weight of a cut: cut( A, B ) = SiÎA SjÎB Wij=Assoc(A, B) l Normalized Cut criteria: minimum cut(A, Ā) More generally: 7 Eric Xing © Eric Xing @ CMU, 2006 -2010
Graph-based Clustering l Data Grouping i Wij j G = {V, E} l Image sigmentation l Affinity matrix: l Degree matrix: l Laplacian matrix: l (bipartite) partition vector: 8 Eric Xing © Eric Xing @ CMU, 2006 -2010
Affinity Function l Affinities grow as s grows l How the choice of s value affects the results? l What would be the optimal choice for s? Eric Xing © Eric Xing @ CMU, 2006 -2010 9
Clustering via Optimizing Normalized Cut l The normalized cut: l Computing an optimal normalized cut over all possible y (i. e. , partition) is NP hard l Transform Ncut equation to a matrix form (Shi & Malik 2000): Rayleigh quotient Subject to: l Still an NP hard problem 10 Eric Xing © Eric Xing @ CMU, 2006 -2010
Relaxation Rayleigh quotient Subject to: l Instead, relax into the continuous domain by solving generalized eigenvalue system: l Which gives: l Note that l The second smallest eigenvector is the real valued solution to this problem!! Eric Xing Rayleigh quotient theorem so, the first eigenvector is y 0=1 with eigenvalue 0. © Eric Xing @ CMU, 2006 -2010 11
Algorithm 1. Define a similarity function between 2 nodes. i. e. : 2. Compute affinity matrix (W) and degree matrix (D). 3. Solve l Do singular value decomposition (SVD) of the graph Laplacian Þ Use the eigenvector with the second smallest eigenvalue, , to bipartition the graph. 4. l For each threshold k, l Compute Ncut(Ak, Bk) Output l Eric Xing Ak={i | yi among k largest element of y*} Bk={i | yi among n-k smallest element of y*} 12 © Eric Xing @ CMU, 2006 -2010
Ideally … y 2 i i i A A 13 Eric Xing © Eric Xing @ CMU, 2006 -2010
Example (Xing et al, 2001) 14 Eric Xing © Eric Xing @ CMU, 2006 -2010
Poor features can lead to poor outcome (Xing et al 2001) 15 Eric Xing © Eric Xing @ CMU, 2006 -2010
Cluster vs. Block matrix A B 16 Eric Xing © Eric Xing @ CMU, 2006 -2010
Compare to Minimum cut l Criterion for partition: A Problem! Weight of cut is directly proportional to the number of edges in the cut. B Cuts with lesser weight than the ideal cut Ideal Cut First proposed by Wu and Leahy 17 Eric Xing © Eric Xing @ CMU, 2006 -2010
Superior Performance? l K-means and Gaussian mixture methods are biased toward convex clusters 18 Eric Xing © Eric Xing @ CMU, 2006 -2010
Ncut is superior in certain cases 19 Eric Xing © Eric Xing @ CMU, 2006 -2010
Why? 20 Eric Xing © Eric Xing @ CMU, 2006 -2010
General Spectral Clustering Data Similarities 21 Eric Xing © Eric Xing @ CMU, 2006 -2010
Representation l Partition matrix X: l Pair-wise similarity matrix W: l Degree matrix D: l Laplacian matrix L: segments pixels 22 Eric Xing © Eric Xing @ CMU, 2006 -2010
A Spectral Clustering Algorithm Ng, Jordan, and Weiss 2003 l Given a set of points S={s 1, …sn} l Form the affinity matrix l Define diagonal matrix Dii= Sk aik l Form the matrix l Stack the k largest eigenvectors of L to for the columns of the new matrix X: l Renormalize each of X’s rows to have unit length and get new matrix Y. Cluster rows of Y as points in R k 23 Eric Xing © Eric Xing @ CMU, 2006 -2010
SC vs Kmeans 24 Eric Xing © Eric Xing @ CMU, 2006 -2010
Why it works? l K-means in the spectrum space ! Eric Xing © Eric Xing @ CMU, 2006 -2010 25
Eigenvectors and blocks l l Block matrices have block eigenvectors: 1= 2 2= 2 1 1 0 0 . 71 0 0 0 1 1 0 . 71 eigensolver 3= 0 4= 0 Near-block matrices have near-block eigenvectors: 1= 2. 02 2= 2. 02 3= -0. 02 4= -0. 02 1 1 . 2 0 . 71 0 1 1 0 -. 2 . 69 -. 14 . 2 0 1 1 . 14 . 69 0 -. 2 1 1 0 . 71 Eric Xing eigensolver © Eric Xing @ CMU, 2006 -2010 26
Spectral Space l l Can put items into blocks by eigenvectors: 1 1 . 2 0 . 71 0 1 1 0 -. 2 . 69 -. 14 . 2 0 1 1 . 14 . 69 0 -. 2 1 1 0 . 71 e 2 Clusters clear regardless of row ordering: 1 . 2 1 0 . 71 0 . 2 1 0 1 . 14 . 69 1 0 1 -. 2 . 69 -. 14 0 1 -. 2 1 0 . 71 e 2 Eric Xing © Eric Xing @ CMU, 2006 -2010 e 1 e 2 27
More formally … l Recall generalized Ncut l Minimizing this is equivalent to spectral clustering segments pixels 28 Eric Xing © Eric Xing @ CMU, 2006 -2010
Spectral Clustering l Algorithms that cluster points using eigenvectors of matrices derived from the data l Obtain data representation in the low-dimensional space that can be easily clustered l Variety of methods that use the eigenvectors differently (we have seen an example) l Empirically very successful l Authors disagree: l l Which eigenvectors to use l How to derive clusters from these eigenvectors Two general methods Eric Xing 29 © Eric Xing @ CMU, 2006 -2010
Method #1 l Partition using only one eigenvector at a time l Use procedure recursively l Example: Image Segmentation l Uses 2 nd (smallest) eigenvector to define optimal cut l Recursively generates two clusters with each cut 30 Eric Xing © Eric Xing @ CMU, 2006 -2010
Method #2 l Use k eigenvectors (k chosen by user) l Directly compute k-way partitioning l Experimentally has been seen to be “better” 31 Eric Xing © Eric Xing @ CMU, 2006 -2010
Toy examples Images from Matthew Brand (TR-2002 -42) 32 Eric Xing © Eric Xing @ CMU, 2006 -2010
User’s Prerogative l Choice of k, the number of clusters l Choice of scaling factor l Realistically, search over and pick value that gives the tightest clusters l Choice of clustering method: k-way or recursive bipartite l Kernel affinity matrix 33 Eric Xing © Eric Xing @ CMU, 2006 -2010
Conclusions l l Good news: l Simple and powerful methods to segment images. l Flexible and easy to apply to other clustering problems. Bad news: l High memory requirements (use sparse matrices). l Very dependant on the scale factor for a specific problem. 34 Eric Xing © Eric Xing @ CMU, 2006 -2010