Informatics and Mathematical Modelling Intelligent Signal Processing Approximate

Informatics and Mathematical Modelling / Intelligent Signal Processing Approximate L 0 constrained NMF/NTF Morten Mørup Informatics and Mathematical Modeling Technical University of Denmark Work done in collaboration with Ph. D Kristoffer Hougaard Madsen Informatics and Mathematical Modeling Technical University of Denmark Professor Lars Kai Hansen Informatics and Mathematical Modeling Technical University of Denmark Morten Mørup ISCAS 2008 1

Informatics and Mathematical Modelling / Intelligent Signal Processing Non-negative Matrix Factorization (NMF) V WH, V≥ 0, W≥ 0, H≥ 0 (Lee & Seung – Nature 1999) NMF gives Part based representation! Morten Mørup 2

Informatics and Mathematical Modelling / Intelligent Signal Processing NMF based on Multiplicative updates Step size parameter Morten Mørup 3

Informatics and Mathematical Modelling / Intelligent Signal Processing fast Non-Negative Least Squares, f. NNLS Active Set procedure (Lawson and Hanson, 1974) Morten Mørup 4

Informatics and Mathematical Modelling / Intelligent Signal Processing NMF not in general unique!! V=WH=(WP)(P-1 H)=W’H’ (Donoho & Stodden, 2003) Morten Mørup 5

Informatics and Mathematical Modelling / Intelligent Signal Processing FIX: Impose sparseness (Hoyer, 2001, 2004 Eggert et al. 2004) n Ensures uniqueness n Eases interpretability (sparse representation factor effects pertain to fewer dimensions) n Can work as model selection (Sparseness can turn off excess factors by letting them become zero) n Resolves over complete representations (when model has many more free variables than data points) L 1 used as convex proxy for the L 0 norm, i. e. card(H) Morten Mørup 6

Informatics and Mathematical Modelling / Intelligent Signal Processing Least Angle Regression and Selection(LARS)/Homotopy Method Morten Mørup 7

Informatics and Mathematical Modelling / Intelligent Signal Processing Controlling sparsity degree (Patric Hoyer 2004) Controlling sparsity degree (Mørup et al. , 2008) Sparsity can now be controlled by evaulating the full regularization path of the NLARS Morten Mørup 8

Informatics and Mathematical Modelling / Intelligent Signal Processing New Algorithm for sparse NMF: 1: Solve for each column of H using NLARS and obtain solutions for all values of (i. e. the entire regularization path) 2: Select -solution giving the desired degree of sparsity 3: Update W such that ||Wd||F=1, according to Repeat from step 1 until convergence Morten Mørup 9 (Eggert et al. 2004)

Informatics and Mathematical Modelling / Intelligent Signal Processing CBCL face database Morten Mørup USPS handwritten digits 10

Informatics and Mathematical Modelling / Intelligent Signal Processing Morten Mørup 11

Informatics and Mathematical Modelling / Intelligent Signal Processing Conclusion n New efficient algorithm for sparse NMF based on the proposed non-negative version of the LARS algorithm n The obtained full regularization path admit to use L 1 as a convex proxy for the L 0 norm to control the degree of sparsity given by n The proposed method is more efficient than previous methods to control degree of sparsity. Furhtermore, NLARS is even comparable in speed to the classic efficient f. NNLS method. n Proposed method directly generalizes to tensor decompositions through models such as Tucker and PARAFAC when using an alternating least squares approach. Morten Mørup 12