Compressed Sensing Theory Geometric Interpretations April 7 th

Compressed Sensing Theory Geometric Interpretations April 7 th 2016, Ilmenau © Fraunhofer IIS Prof. Giovanni Del Galdo

Compressed Sensing Theory Geometric Interpretations April 7 th 2016, Ilmenau Prof. Giovanni Heavily inspired by… With contributions from… the tutorial of Dr. Dejan E. Lazich given at the 22 nd meeting of the ITG section on “Applied Information Theory” on Oct. 7 th, 2013 Dr. Florian © Fraunhofer IIS Römer Anastasia Lavrenko Alexandra Craciun Magdalena Prus Mohamed Gamal Ibrahim Roman Alieiev Del Galdo

Goals of this lecture n Answer the following questions: n Why is the desired sparse solution unique? n Why is the hyperplane of solutions tangent to the L 1 -ball at the right point? n What is the impact of measurement noise? © Fraunhofer IIS 3 ?

Complete Data Model Reconstruction strategy © Fraunhofer IIS 4

Solution Set for the Equality Constraint n x must lie on a line © Fraunhofer IIS 5

Lp-Norms and Lp-Balls n Definition of Lp-norm: n Lp-Balls for N = 2 © Fraunhofer IIS 6

© Fraunhofer IIS 7

Minimization Problem: p=0. 5 © Fraunhofer IIS 8

Minimization Problem: p=0. 5 L 2 2 D animation © Fraunhofer IIS 9

Minimization Problem: p=0. 5 L 2 2 D animation n The “arms” of the L 0. 5 ball reach out making sparse solutions favored wrt non-sparse © Fraunhofer IIS 10

Minimization Problem: p=1 n Same effect as L 0. 5, although not as prominent © Fraunhofer IIS 11

Minimization Problem: p=1 © Fraunhofer IIS 12

Minimization Problem: p=1 © Fraunhofer IIS 13

Minimization Problem: p=2 © Fraunhofer IIS 14

Minimization Problem: p=2 © Fraunhofer IIS 15

Minimization Problem: p=2 n This is the LS solution © Fraunhofer IIS 16 n

© Fraunhofer IIS 17

© Fraunhofer IIS 18

© Fraunhofer IIS 19

Escape Velocities for the L 1 -ball n Vertices move faster than edges, which move faster than sides n They correspond to 1 -sparse, 2 -sparse, and 3 -sparse respectively © Fraunhofer IIS 20

Choice of p-Norm n p>1 lead to non-sparse solutions n p<=1 lead to sparse solutions n p=0 = cardinality n logical choice n gives the sparsest solution n NP-hard n p=1 n identical to p=0 for practical cases n linear and convex problem © Fraunhofer IIS 21

Goals of this lecture n Answer the following questions: n Why is the desired sparse solution unique? n Why is the hyperplane of solutions tangent to the L 1 -ball at the right point? n What is the impact of measurement noise? © Fraunhofer IIS 22 ?

Outline n General geometrical considerations n affine subspace of solutions for a generalized System of Linear Equations (SLE) n subspaces spanned by sparse vectors n intersection between the subspace of solutions with the subspaces spanned by the sparse vectors, i. e. , When does an SLE have sparse solutions? n Geometrical considerations specific to Compressed Sensing n solutions for a system originating from a compressed sensing scenario n impact of the noise: designing the sensing matrix n Conclusions © Fraunhofer IIS 23

System of Linear Equations (SLE) underdetermined fully determined overdetermined infinite number of solutions one unique solution no solutions ! assuming B to be full rank © Fraunhofer IIS 24

Solutions to the i-th Linear Equation solution to the homogeneous problem solution to the inhomogeneous problem © Fraunhofer IIS 25

Solutions to the Homogeneous Linear Equation n Any point x 0 on the plane orthogonal to bi is a solution to the homogeneous linear equation © Fraunhofer IIS 26

Solutions to the Inhomogeneous Linear Equation n Distance between the planes: n Any point on the affine plane is a solution to the inhomogeneous linear equation © Fraunhofer IIS 27

Solutions to the Inhomogeneous SLE n Each equation removes a degree of freedom n There exists one unique solution © Fraunhofer IIS 28

Shooting for Higher Dimensional Spaces n In order to predict the characteristics of the affine subspace of solutions we cannot rely on intuition alone ND n Therefore, we need to introduce more powerful tools n the definition of affine subspaces n the rules applying to the intersection of affine subspaces 3 D © Fraunhofer IIS 29

K-Flats, Affine Subspaces in N-dimensional Ambient Space n The origin-free generalization of points, lines, planes, and K-dimensional subspaces are termed: n affine subspaces of dimension 0, 1, 2, and K, respectively or n 0 -flats, 1 -flats, 2 -flats, and K-flats, respectively n A K-flat is determined by K+1 linearly independent points n K points are linearly independent iff n taking a subset, no 3 points lie on a line (1 -flat) n taking a subset, no 4 points lie on a plane (2 -flat) n … n taking a subset, no k points lie on a (k-2)-flat, for 2<=k<=K n Ambient space of dimension N: the N-flat which contains the universe considered © Fraunhofer IIS 30

K-Flats, Affine Subspaces in N-dimensional Ambient Space n A line is a 1 -flat and is determined by 2 linearly independent points n A plane is a 2 -flat and is determined by 3 linearly independent points n 3 points are linearly independent if no 2 points overlap and no 3 points lie on a line © Fraunhofer IIS 31

Intersection of Affine Subspaces n Assume a p-flat and a q-flat in N-dimensional ambient space n Ignoring special cases, the intersection of the two flats is (p+q-N)-flat, if p+q-N >= 0 empty, if p+q-N < 0 n If two flats obey the rules above, they are said to be in General Position n Randomly generated subspaces are in General Position with probability one © Fraunhofer IIS 32

Intersection of Affine Subspaces: Examples Two lines in 3 D space p=1 q=1 N=3 Intersection = empty …as 1 + 1 – 3 = -1 The lines are skew © Fraunhofer IIS 33

Intersection of Affine Subspaces: Examples Two lines in 2 D space p=1 q=1 N=2 Intersection = 0 -flat …as 1 +1 – 2 = 0 The lines intersect in a point Degenerate case: Two overlapping lines Intersection = 1 -flat © Fraunhofer IIS 34

Intersection of Affine Subspaces: Examples Two planes in 3 D space p=2 q=2 N=3 Intersection = 1 -flat …as 2 + 2 – 3 = 1 The planes intersect in a line © Fraunhofer IIS 35

Intersection of Affine Subspaces: Examples Two planes in 4 D space p=2 q=2 N=4 Intersection = 0 -flat, i. e. , a point …as 2 + 2 – 4 = 0 The two planes intersect in a point © Fraunhofer IIS 36 A special case can be visualized! The rest of the plane is in the fourth dimension

Degenerate Cases: Examples n Two 1 -flats (lines) overlapping in 2 D The General Position would be an intersection in one point n Two 2 -flats (planes) intersecting on a line in 5 D The General Position would be an empty intersection n Affine subspaces generated randomly are degenerate with probability zero © Fraunhofer IIS 37

Flats, their Intersection for a SLE n Each linear constraint gives rise to a (N-1)-flat n The subspace of solutions is the intersection of the M (N-1)-flats in N-dimensional space. It is an (N-M)-flat! number of linear constraints M © Fraunhofer IIS 38 dimensionality of the intersection 1 N-1 2 N-2 3 N-3 … … M N-M

Flats, their Intersection for a SLE © Fraunhofer IIS 39 number of equations M number of dimensions N dimensions of the subspace of solutions, i. e. , the intersection underdetermined N-M infinite number of solutions determined 0 one unique solution overdetermined empty no solution

Recap: Geometrical Interpretation on the SLE n Given a SLE featuring n an N dimensional unknown vector x n M linear equations n The solutions to each linear equation span an affine subspace of dimensionality N-1, i. e. , an (N-1)-flat n The solutions to the whole SLE span the intersection of the M (N-1)-flats n Given an underdetermined system of equations n the affine subspace of solutions has size N-M n we call it: the “Solutions Monster” (N-M)-flat Solutions Monster © Fraunhofer IIS 40

Outline n General geometrical considerations n affine subspace of solutions for a generalized System of Linear Equations (SLE) n subspaces spanned by sparse vectors n intersection between the subspace of solutions with the subspaces spanned by the sparse vectors, i. e. , When does an SLE have sparse solutions? n Geometrical considerations specific to Compressed Sensing n solutions for a system originating from a compressed sensing scenario n impact of the noise: designing the sensing matrix n Conclusions © Fraunhofer IIS 41

Affine Subspaces for K-sparse Vectors n The K-sparse vectors span specific affine subspaces in N-dimensional space n 1 -sparse vectors span 1 -flats © Fraunhofer IIS 42

Affine Subspaces for K-sparse Vectors n 2 -sparse vectors span 2 -flats © Fraunhofer IIS 43

Outline n General geometrical considerations n affine subspace of solutions for a generalized System of Linear Equations (SLE) n subspaces spanned by sparse vectors n intersection between the subspace of solutions with the subspaces spanned by the sparse vectors, i. e. , When does an SLE have sparse solutions? n Geometrical considerations specific to Compressed Sensing n solutions for a system originating from a compressed sensing scenario n impact of the noise: designing the sensing matrix n Conclusions © Fraunhofer IIS 44

Affine Subspaces for K-sparse Vectors n In general: K-sparse vectors span K-flats n Given a system of linear equations in General Position: fla 1 - ? t What is the probability of having sparse solutions? I. e. , when will the solutions monster touch the flats containing sparse vectors? (N-M)-flat Solutions Monster © Fraunhofer IIS 45

Intersection: Solution Monster <-> Sparse Flats n Assuming a SLE of M linear constraints in N-dimensional space Sparsity K Corresponding affine subspace Dimensionality of the intersection between the Solution Monster and each of the sparse flats 1 1 -flat Empty 2 2 -flat Empty … … … M M-flat 0 -flat = point M+1 -flat =line … … … K K-flat (K-M)-flat n Assuming General Position the results in the table occur with Prob. = 1 © Fraunhofer IIS 46

Intersection: Solution Monster <-> Sparse Flats 1 -flat 2 -fl at 1 -flat f 2 (N-M)-flat Solutions to the SLE 2 -fla 1 -flat t n For K < M the Solutions Monster does not intersect with any K-flat carrying K-sparse solutions © Fraunhofer IIS 47

Recap: Solution Monster vs. Sparse Flats n In the General Position, the Solutions Monster dodges all sparse vectors up to K = M-1 Me too! (N-M)-flat Solutions to the SLE © Fraunhofer IIS 48

Outline n General geometrical considerations n affine subspace of solutions for a generalized System of Linear Equations (SLE) n subspaces spanned by sparse vectors n intersection between the subspace of solutions with the subspaces spanned by the sparse vectors, i. e. , When does an SLE have sparse solutions? n Geometrical considerations specific to Compressed Sensing n solutions for a system originating from a compressed sensing scenario n impact of the noise: designing the sensing matrix n Conclusions © Fraunhofer IIS 49

SLE in a Compressed Sensing Scenario n Let us now take an underdetermined system of equations originating from a sparse solution x n The Solution Monster corresponding to the homogeneous system will be in General Position (N-M)-flat Solutions Monster © Fraunhofer IIS 50

SLE in a Compressed Sensing Scenario n Let us now take an underdetermined system of equations originating from a sparse solution x n The Solution Monster corresponding to the homogeneous system will be in General Position (N-M)-flat Solutions Monster n However d will move it into a specific position… © Fraunhofer IIS 51

SLE in a Compressed Sensing Scenario (N-M)-flat Solutions Monster © Fraunhofer IIS 52 fla 1 - ! t n The Solution Monster will therefore NOT BE in General Position w. r. t. the specific 1 -flat, however it will be in General Position with probability one w. r. t. all other sparse flats

Intersection: Solution Monster <-> Sparse Flats (N-M)-flat Solutions to the SLE t at lat f 2 fla 2 -fl 1 -flat 2 -fla t n For K < M the Solutions Monster does not intersect with any K-flat carrying K-sparse solutions BUT the one that generated the SLE © Fraunhofer IIS 53

Recap: Geometrical Interpretation on the SLE originating from a Compressed Sensing Scenario n Given a SLE originating from the measurement of a sparse vector x n N: dimensionality of x n M<N: number of equations n Among all solutions to the SLE, only one is K-sparse with K<M and is equal to the desired x © Fraunhofer IIS 54

Outline n General geometrical considerations n affine subspace of solutions for a generalized System of Linear Equations (SLE) n subspaces spanned by sparse vectors n intersection between the subspace of solutions with the subspaces spanned by the sparse vectors, i. e. , When does an SLE have sparse solutions? n Geometrical considerations specific to Compressed Sensing n solutions for a system originating from a compressed sensing scenario n impact of the noise: designing the sensing matrix n Conclusions © Fraunhofer IIS 55

SLE in a Compressed Sensing Scenario with Noise measurement noise signal noise n The system may experience two kinds of noise: n signal noise: perturbing the sparse vector x n measurement noise: perturbing the data vector d directly n Signal noise can be interpreted as a measurement noise, colored by B © Fraunhofer IIS 56

SLE in a Compressed Sensing Scenario with Noise n The system may experience two kinds of noise: n signal noise: perturbing the sparse vector x, indirectly perturbing d n measurement noise: perturbing the data vector d directly n Signal noise can be interpreted as a measurement noise, colored by B © Fraunhofer IIS 57

SLE in a Compressed Sensing Scenario with Noise (N-M)-flat Solutions Monster © Fraunhofer IIS 58 fla 1 - ! t n Considering measurement noise only, the Solutions Monster will be moved away from x, as the noise perturbs d

SLE in a Compressed Sensing Scenario with Noise 1 - fla t n Considering measurement noise only, the Solutions Monster will be moved away from x, as the noise perturbs d (N-M)-flat Solutions Monster © Fraunhofer IIS 59

Intersection: Solution Monster <-> Sparse Flats (N-M)-flat Solutions to the SLE t at lat f 2 fla 2 -fl 1 -flat 2 -fla t n In a noisy case, the reconstruction algorithm finds the closest sparse solution to the Solution Monster © Fraunhofer IIS 60

Intersection: Solution Monster <-> Sparse Flats (N-M)-flat Solutions to the SLE t at lat f 2 fla 2 -fl 1 -flat 1 - 1 -flat reconstruction 1 -flat 2 -fla t n If the noise is not too large, the support is correctly reconstructed and some errors are made in the amplitudes of the non-zero elements © Fraunhofer IIS 61

Intersection: Solution Monster <-> Sparse Flats (N-M)-flat Solutions to the SLE t at lat f 2 fla 2 -fl 1 -flat 2 -fla t n If the noise is too large, the support is incorrectly reconstructed © Fraunhofer IIS 62

Resilience Against Noise n A good way to measure the resilience of the SLE is to measure its “self coherence” n It tells us how similar the two most similar atoms of B are, so that due to the noise they can be mistaken for one another n This should be minimized when designing the measurement kernel * Assuming norm one columns of B © Fraunhofer IIS 63

Designing the Sensing Matrix B n Alternatively we can compute the angle between the vectors bk and bh giving us a similar interpretation, namely, we’d like the angle to be as large as possible so that no “aliasing” in terms of support will occur * Assuming norm one columns of B © Fraunhofer IIS 64

Designing the Sensing Matrix B Random matrix Vandermonde structure k k h h © Fraunhofer IIS 65

Designing the Sensing Matrix B n Structured sensing matrix: Allows us to minimize the errors on support recovery n Ideal for applications in which this is relevant, such as any detection application, e. g. , DOA estimation or spectrum sensing n Random sensing matrix: Usually achieves larger angles (i. e. , lower self coherence), however the recovery might place the non-zero elements far away from the correct support n Ideal for applications in which this is not relevant, such as any signal reconstruction application, e. g. , image acquisition © Fraunhofer IIS 66

Conclusions n A SLE in N-dimensional space featuring M equations… n gives rise to an affine subspace of solutions of dimension (N-M) n possesses no K-sparse solutions with probability 1, for K<M n For a SLE originating from a sparse vector x n no other K-sparse solution but x exists, for K<M n reconstruction methods aim at finding the closest sparse K-flat to the subspace of solutions n measurement noise moves the subspace of solutions away from x n the distances to the sparse K-flats should be considered when designing the sensing matrix © Fraunhofer IIS 67

Thank you for your attention! From me too! Solutions Monster © Fraunhofer IIS 68