Sparsity Control for Robust Principal Component Analysis Gonzalo

Sparsity Control for Robust Principal Component Analysis Gonzalo Mateos and Georgios B. Giannakis ECE Department, University of Minnesota Acknowledgments: NSF grants no. CCF-1016605, EECS-1002180 Asilomar Conference November 10, 2010 1

Principal Component Analysis n Motivation: (statistical) learning from high-dimensional data DNA microarray Traffic surveillance n Principal component analysis (PCA) [Pearson’ 1901] q Extraction of low-dimensional data structure q Data compression and reconstruction q PCA is non-robust to outliers [Jolliffe’ 86] n Our goal: robustify PCA by controlling outlier sparsity 2

Our work in context n Contemporary applications q Anomaly detection in IP networks [Huang et al’ 07], [Kim et al’ 09] q Video surveillance, e. g. , [Oliver et al’ 99] Original Robust PCA `Outliers’ n Robust PCA q Robust covariance matrix estimators [Campbell’ 80], [Huber’ 81] q Computer vision [Xu-Yuille’ 95], [De la Torre-Black’ 03] q Low-rank matrix recovery from sparse errors [Wright et al’ 09] n Huber’s M-class and sparsity in linear regression [Fuchs’ 99] 3

PCA formulations n Training data: n Minimum reconstruction error: n Dimensionality reduction operator n Reconstruction operator n Maximum variance: n Factor analysis model: Solution: 4

Robustifying PCA n Least-trimmed squares (LTS) regression [Rousseeuw’ 87] LTS-based PCA for robustness (LTS PCA) is the -th order statistic among Trimming constant determines breakdown point n Q: How should we go about minimizing ? (LTS PCA) is nonconvex; existence of minimizer(s)? A: Try all subsets of size , solve, and pick the best n Simple but intractable beyond small problems 5

Modeling outliers n Introduce auxiliary variables q Inliers obey q Inlier noise: s. t. inlier outlier ; outliers something else are zero-mean i. i. d. random vectors n Remarks q and are unknown q If outliers sporadic, then vector is sparse! n Natural (but intractable) estimator 6

LTS PCA as sparse regression n Lagrangian form (P 0) q Tuning controls sparsity in , thus number of outliers Proposition 1: If solves (P 0) with , then solves (LTS PCA) too. chosen such that n Justifies the model and its estimator (P 0); ties sparsity with robustness 7

Just relax! n (P 0) is NP-hard relax (P 2) q Role of sparsity controlling is central n Q: Does (P 2) yield robust estimates ? A: Yap! Huber estimator is a special case 8

Entrywise outliers n Use -norm regularization (P 1) Original Outlier pixels Robust PCA (P 2) Robust PCA (P 1) Entire image rejected Outlier pixels rejected 9

Alternating minimization (P 1) q q update: reduced-rank Procrustes rotation update: coordinatewise soft-thresholding Proposition 2: Alg. 1’s iterates converge to a stationary point of (P 1). 10

Refinements n Nonconvex penalty terms approximate better in (P 0) n Options: SCAD [Fan-Li’ 01], or sum-of-logs [Candes etal’ 08] n Iterative linearization-minimization of around n Iteratively reweighted version of Alg. 1 n Warm start: solution of (P 1) or (P 2) n Bias reduction in (cf. weighted Lasso [Zou’ 06]) n Discard outliers identified in n Re-estimate missing data problem 11

Online robust PCA n Motivation: Real-time data and memory limitations n Exponentially-weighted robust PCA n Approximation [Yang’ 95] n At time , do not re-estimate past outlier vectors 12

Video surveillance Original PCA Data: http: //www. cs. cmu. edu/~ftorre/ Robust PCA `Outliers’ 13

Online PCA in action n Inliers: n Outliers: and Angle between C(n) and C n Figure of merit: angle between 14

Concluding summary n Sparsity control for robust PCA q q q LTS PCA as -(pseudo)norm regularized regression (NP-hard) Relaxation (group)-Lassoed PCA Sparsity controlling role of central M-type estimator n Batch and online robust PCA algorithms n i) Outlier identification, ii) Robust subspace tracking n Refinements via nonconvex penalty terms n Tests on real video surveillance data for anomaly extraction n Ongoing research n Preference measurement: conjoint analysis and collaborative filtering n Robustifying kernel PCA and blind dictionary learning 15