WK 9 Principle Component Analysis Contents PCA GHA

WK 9 – Principle Component Analysis Contents PCA GHA APEX CS 476: Networks of Neural Computation WK 9 – Principle Component Analysis Kernel PCA Conclusions Dr. Stathis Kasderidis Dept. of Computer Science University of Crete Spring Semester, 2009 CS 476: Networks of Neural Computation, CSD, UOC, 2009

Contents PCA GHA • Introduction to Principal Component Analysis • Generalised Hebbian Algorithm • Adaptive Principal Components Extraction APEX • Kernel Principal Components Analysis Kernel PCA • Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009

Principal Component Analysis Contents PCA GHA APEX Kernel PCA Conclusions • The PCA method is a statistical method for Feature Selection and Dimensionality Reduction. • Feature Selection is a process whereby a data space is transformed into a feature space. In principal both spaces have the same dimensionality. • However, in the PCA method, the transformation is design in such way that the data set be represented by a reduced number of “effective” features and yet retain most of the intrinsic information contained in the data; in other words the data set undergoes a dimensionality reduction. CS 476: Networks of Neural Computation, CSD, UOC, 2009

Principal Component Analysis-1 Contents PCA GHA APEX Kernel PCA Conclusions • Suppose that we have a x of dimension m and we wish to transmit it using l numbers, where l<m. If we simply truncate the vector x, we will cause a mean square error equal to the sum of the variances of the elements eliminated from x. • So, we ask: Does there exist an invertible linear transformation T such that the truncation of Tx is optimum in the mean-squared sense? • Clearly, the transformation T should have the property that some of its components have low variance. • Principal Component Analysis maximises the rate CS 476: Networks of Neural Computation, CSD, UOC, 2009

Principal Component Analysis-2 Contents PCA GHA APEX Kernel PCA Conclusions of decrease of variance and is the right choice. • Before we present neural network, Hebbian-based, algorithms that do this we first present the statistical analysis of the problem. • Let X be an m-dimensional random vector representing the environment of interest. We assume that the vector X has zero mean: E[X]=0 Where E is the statistical expectation operator. If X has not zero mean we first subtract the mean from X before we proceed with the rest of the analysis. CS 476: Networks of Neural Computation, CSD, UOC, 2009

Principal Component Analysis-3 Contents PCA GHA APEX Kernel PCA Conclusions • Let q denote a unit vector, also of dimension m, onto which the vector X is to be projected. This projection is defined by the inner product of the vectors X and q: A=XTq=q. TX Subject to the constraint: ||q||=(q. Tq)½=1 • The projection A is a random variable with a mean and variance related to the statistics of vector X. Assuming that X has zero mean we can calculate the mean value of the projection A: E[A]=q. TE[X]=0 CS 476: Networks of Neural Computation, CSD, UOC, 2009

Principal Component Analysis-4 Contents • The variance of A is therefore the same as its mean -square value and so we can write: PCA 2=E[A 2]=E[(q. TX)(XTq)]=q. TE[XXT]q=q. TR q GHA • The m-by-m matrix R is the correlation matrix of the random vector X, formally defined as the expectation of the outer product of the vector X with itself, as shown: APEX Kernel PCA Conclusions R=E[XXT] • We observe that the matrix R is symmetric, which means that: RT=R CS 476: Networks of Neural Computation, CSD, UOC, 2009

Principal Component Analysis-5 Contents PCA GHA APEX Kernel PCA Conclusions • From this property it follows that for any m-by-1 vectors a and b we have: a. TRb= b. TRa • From the above we see that the variance 2 of A is a function of the unit vector q; we can then thus write: (q)= 2= q. TR q • From the above we can think of (q) as a variance probe. • To minimise the variance of A we must find the vectors q which are the extremal points of (q), CS 476: Networks of Neural Computation, CSD, UOC, 2009

Principal Component Analysis-6 Contents Subject to the constraint of unit length. GHA • If q is a vector such that (q) has an extreme value, then for any small q of the unit vector q, we find that, to the first order in q: APEX (q+ q )= (q) PCA Kernel PCA Conclusions • Now from the definition of the variance probe we have: (q+ q )= (q+ q)TR (q+ q)= q. TRq+2( q)TRq+ ( q)TR q Where in the previous line we have made use of the symmetric property of matrix R. CS 476: Networks of Neural Computation, CSD, UOC, 2009

Principal Component Analysis-7 Contents PCA GHA APEX Kernel PCA Conclusions • Ignoring the second-order term ( q)TR q and invoking the definition of (q) we may write: (q+ q )= q. TRq+2( q)TRq= (q) +2( q)TRq • The above relation implies that: ( q)TRq=0 • Note that just any perturbation q of q is not admissible; rather we restrict to use those for which the Euclidean norm of the perturbed vector q+ q remains equal to unity: || q+ q ||=1 Or: (q+ q)T (q+ q)=1 CS 476: Networks of Neural Computation, CSD, UOC, 2009

Principal Component Analysis-8 Contents • Taking into account that q is already a vector of unit length, this means that: PCA ( q)T q=0 GHA • This means that perturbation q must be orthogonal to q and therefore only a small change in the direction of q is permitted. APEX Kernel PCA Conclusions • Combining the previous two equations we can now write: ( q)TR q- ( q)T q=0 ( q)T(R q- q)=0 Where is a scaling constant for the elements of R. • We can now write: CS 476: Networks of Neural Computation, CSD, UOC, 2009

Principal Component Analysis-9 Contents PCA GHA APEX Kernel PCA Conclusions R q= q • This means that q is an eigenvector and is an eigenvalue of R. • The matrix R has real and non-negative eigenvalues (because it is symmetric). Let the eigenvalues of matrix R be denoted by i and the corresponding vectors by qi where the eigenvalues are arranged in a decreasing order: 1 > 2 > … > m so that 1= max. CS 476: Networks of Neural Computation, CSD, UOC, 2009

Principal Component Analysis-10 Contents PCA GHA • We can then write matrix R as: • Combining the previous results we can see that the variance probes are the same as the eigenvalues: APEX Kernel PCA Conclusions (qj)= j , for j=1, 2, …, m • To summarise the previous analysis we have two important results: • The eigenvectors of the correlation matrix R pertaining to the zero-mean random variable X define the unit vectors qj , representing the principal directions along which the variance probes (qj) have their extreme values; CS 476: Networks of Neural Computation, CSD, UOC, 2009

Principal Component Analysis-11 • The Contents PCA GHA APEX Kernel PCA Conclusions associated eigenvalues define the extremal values of the variance probes. • We now we want to investigate the representation of a data vector x which is a realisation of the random vector X. • With m eigenvectors qj we have m possible projection directions. The projections of x into the eigenvectors are given by: j=qj. Tx= x. Tqj , j=1, 2, …, m • The numbers j are called the principal components. To reconstruct the original vector x from the projections we combine all projections into CS 476: Networks of Neural Computation, CSD, UOC, 2009

Principal Component Analysis-12 Contents PCA a single vector: =[ 1, 2, …, m]T GHA =[x. Tq 1, x. Tq 2, …, x. Tqm]T APEX =QTx Kernel PCA Conclusions Where Q is the matrix which is constructed by the (column) eigenvectors of R. • From the above we see that: x=Q • This is nothing more than a coordinate CS 476: Networks of Neural Computation, CSD, UOC, 2009

Principal Component Analysis-13 Contents PCA GHA APEX Kernel PCA Conclusions transformation from the input space, of vector x, to the feature space of the vector . • From the perspective of the pattern recognition the usefulness of the PCA method is that it provides an effective technique for dimensionality reduction. • In particular we may reduce the number of features needed for effective data representation by discarding those linear combinations in the previous formula that have small variances and retain only these terms that have large variances. • Let 1, 2, …, l denote the largest l eigenvalues of R. We may then approximate the vector x by CS 476: Networks of Neural Computation, CSD, UOC, 2009

Principal Component Analysis-14 Contents truncating the previous sum to the first l terms: PCA GHA APEX Kernel PCA Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009

Generalised Hebbian Algorithm Contents PCA GHA APEX • We will present now a neural network method which solves the PCA problem. It belongs to the so-called reestimation algorithms class of PCA methods. • The network which solves the problem is shown below: Kernel PCA Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009

Generalised Hebbian Algorithm -1 Contents PCA GHA APEX Kernel PCA Conclusions • For the feedforward network shown we make two structural assumptions: • Each neuron in the output layer of the network is linear; • The network has m inputs and l outputs, both of which are specified. Moreover, the network has fewer outputs than inputs (i. e. l < m). • It can be shown that under these assumptions and by using a special form of Hebbian learning the network truly learns to calculate the principal components in its output nodes. • The GHA can be summarised as follows: CS 476: Networks of Neural Computation, CSD, UOC, 2009

Generalised Hebbian Algorithm -2 1. Contents PCA GHA 2. Initialise the synaptic weights of the network, wji, to small random values at time n=1. Assign a small positive value to the learning rate parameter ; For n=1, j=1, 2, …, l and i=1, 2, …, m, compute: APEX Kernel PCA Conclusions 3. Where xi(n) is the ith component of the m-by-1 input vector x(n) and l is the desire number of principal compenents; Increment n by 1, go to step 2, and continue until the synaptic weights wji reach their steady state CS 476: Networks of Neural Computation, CSD, UOC, 2009

Generalised Hebbian Algorithm -3 Contents PCA GHA APEX values. For large n, the weight wji of neuron j converges to the ith component of the eigenvector associated with jth eigenvalue of the correlation matrix of the input vector x(n). The output neurons represent the eigenvalues of correlation matrix with decreasing order from 1 towards l. Kernel PCA Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009

Adaptive Principal Components Extraction Contents PCA GHA APEX Kernel PCA Conclusions • Another algorithm for extracting the principal components is the adaptive principal components extraction (APEX) algorithm. This network uses both feedforward and feedback connections. • The algorithm is iterative in nature and if we are given the first (j-1) principal components the jth one can be easily computed. • This algorithm belongs to the class of decorrelating algorithms. • The network that implements the algorithm is shown next: CS 476: Networks of Neural Computation, CSD, UOC, 2009

Adaptive Principal Components Extraction-1 Contents PCA GHA APEX Kernel PCA Conclusions • The network structure is defined as follows: • Each neuron is assumed to be linear (in the output layer); • Feedforward connections exist from the input nodes to each of the neurons 1, 2, …, j, with j<m. The feedforward connections operate with a Hebbian rule. They are CS 476: Networks of Neural Computation, CSD, UOC, 2009

Adaptive Principal Components Extraction-2 excitatory and therefore provide amplification. Contents These connections are represented by the wj(n) vector. PCA • GHA Lateral connections exist from the individual outputs of neurons 1, 2, …, j-1 to neuron j of the output layer, thereby applying feedback to the network. These connections are represented by the aj(n) vector. The lateral connections operate with an anti. Hebbian learning rule which has the effect of making them inhibitory. APEX Kernel PCA Conclusions • The algorithm is summarised as follows: 1. Initialise the feedforward weight vector wj and the feedback weight vector aj to small random numbers at time n=1, where j=1, 2, …, m. Assign a small CS 476: Networks of Neural Computation, CSD, UOC, 2009

Adaptive Principal Components Extraction-3 Contents PCA positive value to the learning rate parameter ; 2. Set j=1, and for n=1, 2, …, compute: GHA APEX where x(n) is the input vector. For large n, we have w 1(n) q 1, where q 1 is the eigenvector asociated with the largest eigenvalue 1 of the correlation matrix of x(n); Kernel PCA Conclusions 3. Set j=2, and for n=1, 2, …, compute: CS 476: Networks of Neural Computation, CSD, UOC, 2009

Adaptive Principal Components Extraction-4 Contents PCA GHA APEX Kernel PCA Conclusions 4. Increment j by 1, go to step 3, and continue until j=m, where m is the desired number of principal components. (Note that j=1 corresponds to eigenvector associated with the largest eigenvalue, which is taken care in step 2). For large n we have wj(n) qj and aj(n) 0, where qj is the eigenvector associated with the jth eigenvalue of the correlation matrix of x(n). CS 476: Networks of Neural Computation, CSD, UOC, 2009

Kernel Principal Components Analysis Contents • A last algorithm which uses kernels (more on the SVM lecture) will be given below. We simply summarise the algorithm. • This algorithm can be considered as a non-linear PCA methods as we first project the input space in a feature space using a non-linear transform (x) and then we perform a linear PCA analysis in the feature space. This is different from the previous methods in that they calculate a linear transformation between the input and the feature spaces. • Summary of the kernel PCA method: PCA GHA APEX Kernel PCA Conclusions 1. Given the training examples {xi}i=1 , compute the CS 476: Networks of Neural Computation, CSD, UOC, 2009

Kernel Principal Components Analysis-1 the N-by-N kernel matrix K={K(xi, xj)}, where: Contents PCA GHA K(xi, xj)= T(xi) (xj) 2. Ka= a APEX where is an eigenvalue of the kernel matrix K and a is the associated eigenvector; Kernel PCA Conclusions Solve the eigenvalue problem: 3. Normalise the eigenvectors so computed by requiring that: ak. T ak=1/ k , k=1, 2, …, p where p is the smallest nonzero eigenvalue of the matrix K, assuming that the eigenvalues are arranged in decreasing order; CS 476: Networks of Neural Computation, CSD, UOC, 2009

Kernel Principal Components Analysis-2 4. Contents For the extraction of the principal components of a test point x, compute the projections: PCA GHA where ak, j is the jth element of eigenvector ak. APEX Kernel PCA Conclusions CS 476: Networks of Neural Computation, CSD, UOC, 2009

Conclusions Contents PCA GHA APEX Kernel PCA Conclusions • Typically we use PCA methods for dimension reduction as a pre-processing step before we apply other methods, for example in a pattern recognition problem. • There are batch and adaptive numerical methods for the calculation of the PCA. An example for the first class is the Singular Value Decomposition (SVD) method while the GHA algorithm is for example and adaptive method. • It is used mainly for finding out clusters in highdimensional spaces, as it is difficult to visualise these clusters otherwise. CS 476: Networks of Neural Computation, CSD, UOC, 2009