ICA Alphan Altinok Outline PCA ICA Foundation Ambiguities

ICA Alphan Altinok

Outline Ø PCA Ø ICA § § § Foundation Ambiguities Algorithms Examples Papers

PCA & ICA Ø PCA § Projects d-dimensional data onto a lower dimensional subspace in a way that is optimal in Σ|x 0 – x|2. Ø ICA § Seek directions in feature space such that resulting signals show independence.

PCA Ø Compute d-dimensional μ (mean). Ø Compute d x d covariance matrix. Ø Compute eigenvectors and eigenvalues. Ø Choose k largest eigenvalues. § k is the inherent dimensionality of the subspace governing the signal and (d – k) dimensions generally contain noise. Ø Form a d x k matrix A with k columns of eigenvalues. Ø The representation of data by principal components consists of projecting data into k-dimensional subspace by x = At (x – μ).

PCA Ø A simple 3 -layer neural network can form such a representation when trained.

ICA Ø While PCA seeks directions that represents data best in a Σ|x 0 – x|2 sense, ICA seeks such directions that are most independent from each other. Ø Used primarily for separating unknown source signals from their observed linear mixtures. Ø Typically used in Blind Source Separation problems. ICA is also used in feature extraction.

ICA – Foundation Ø q source signals s 1(k), s 2(k), …, sq(k) § § with 0 means k is the discrete time index or pixels in images scalar valued mutually independent for each value of k Ø h measured mixture signals x 1(k), x 2(k), …, xh(k) Ø Statistical independence for source signals § p[s 1(k), s 2(k), …, sq(k)] = П p[si(k)]

ICA – Foundation Ø The measured signals will be given by § xj(k) = Σsi(k)aij + nj(k) Ø For j = 1, 2, …, h, the elements aij are unknown. Ø Define vectors x(k) and s(k), and matrix A § Observed: x(k) = [x 1(k), x 2(k), …, xh(k)] § Source: s(k) = [s 1(k), s 2(k), …, sq(k)] § Mixing matrix: A = [a 1, a 2, …, aq] Ø The equation above can be stated in vector-matrix form § x(k) = As(k) + n(k) = Σsi(k)ai + n(k)

Ambiguities with ICA Ø The ICA expansion § x(k) = As(k) + n(k) = Σsi(k)ai + n(k) Ø Amplitudes of separated signals cannot be determined. Ø There is a sign ambiguity associated with separated signals. Ø The order of separated signals cannot be determined.

ICA – Using NNs Ø Prewhitening – transform input vectors x(k) by § v(k) = V x(k) § Whitening matrix V can be obtained by NN or PCA Ø Separation (NN or contrast approximation) Ø Estimation of ICA basis vectors (NN or batch approach)

ICA – Fast Fixed Point Algorithm Ø FFPA converges rapidly to the most accurate solution allowed by the data structure.

ICA – Example

ICA – Example

ICA – Example

ICA – Example

ICA – Example

ICA – Example Ø BSS of recorded speech and music signals. speech / music speech / speech in difficult environment mic 1 mic 2 separated 1 separated 2 http: //www. cnl. salk. edu/~tewon/ica_cnl. html

ICA – Example Ø Source images Ø separation demo http: //www. open. brain. riken. go. jp/demos/research. BSRed. html

ICA – Papers Ø Hinton – A New View of ICA § Interprets ICA as a probability density model. § Overcomplete, undercomplete, and multi-layer non-linear ICA becomes simpler. Ø Cardoso – Blind Signal Separation, Statistical Principles § § § Modelling identifiability. Contrast functions. Estimating functions. Adaptive algorithms. Performance issues.

ICA – Papers Ø Hyvarinen – ICA Applied to Feature Extraction from Color and Stereo Images § Seeks to extend ICA by contrasting it to the processing done in neural receptive fields. Ø Hyvarinen – Survey on ICA Ø Lawrence – Face Recognition, A Convolutional Neural Network Approach § Combines local image sampling, a SOM, and a convolutional NN that provides partial invariance to translation, rotation, scaling, and deformations.

ICA – Papers Ø Sejnowski – Independent Component Representations for Face Recognition Ø Sejnowski – A Comparison of Local vs Global Image Decompositions for Visual Speechreading Ø Bartlett – Viewpoint Invariant Face Recognition Using ICA and Attractor Networks Ø Bartlett – Image Representations for Facial Expression Coding

ICA – Links Ø http: //sig. enst. fr/~cardoso/ Ø http: //www. cnl. salk. edu/~tewon/ica_cnl. html Ø http: //nucleus. hut. fi/~aapo/ Ø http: //www. salk. edu/faculty/sejnowski. html