Beyond Bandlimited Sampling Nonideal Sampling Smoothness and Sparsity

Beyond Bandlimited Sampling: Nonideal Sampling, Smoothness and Sparsity Yonina Eldar Department of Electrical Engineering Technion-Israel Institute of Technology http: //www. ee. technion. ac. il/people/Yonina. Eldar/ yonina@ee. technion. ac. il Rice, April 2008

Sampling: “Analog Girl in a Digital World…” Judy Gorman 99 Analog world Digital world Sampling A 2 D Signal processing Denoising Image analysis … Reconstruction D 2 A (Interpolation) 2

Introduction Input Signals Nonlinearities Reconstruction Main Problem Can we reconstruct x(t) from c[n]? No! Unless we know something about x(t)) bandlimited piece-wise linear Different priors lead to different reconstructions 3

Introduction Input Signals Nonlinearities Reconstruction Our Point-Of-View The field of sampling was traditionally associated with methods implemented either in the frequency domain, or in the time domain Sampling can be viewed in a broader sense of projection onto any subspace or union of subspaces Can choose the subspaces to yield interesting new possibilities (below Nyquist sampling of sparse signals, pointwise samples of non bandlimited signals, perfect compensation of nonlinear effects …) 4

Introduction Input Signals Nonlinearities Reconstruction Bandlimited Sampling Theorems Cauchy (1841): Periodic & N-BL Whittaker (1915) - Shannon (1948): π-BL Extensions focusing primarily on bandlimited signals with nonuniform grids A. J. Jerri, “The Shannon sampling theorem - its various extensions and applications: A tutorial review”, Proc. IEEE, pp. 1565 -1595, Nov. 1977. 5

Introduction Input Signals Nonlinearities Reconstruction Towards More Robust DSPs … Limitations of standard sampling theorems: Input bandlimited Ideal sampling Ideal reconstruction (ideal LPF) Impractical Towards more robust DSPs: General inputs Nonideal sampling: general pre-filters, nonlinear distortions Simple interpolation kernels 6

Introduction Nonlinearities Input Signals Reconstruction Beyond Bandlimited Two key ideas in bandlimited sampling: Avoid aliasing Fourier domain analysis Misleading concepts! Suppose that with Signal is clearly not bandlimited Aliasing in frequency and time Perfect reconstruction possible from samples Aliasing is not the issue … 7

Introduction Nonlinearities Input Signals Reconstruction Fourier Domain Can Be Misleading Original + Initial guess Nonideal sampling Nonlinear distortion t=n Reconstructed signal Replace Fourier analysis by functional analysis, Hilbert space algebra, and convex optimization 8

Introduction Input Signals Nonlinearities Reconstruction Moving To An Abstract Hilbert Space What classes of inputs would we like to treat? Subspace prioir Bandlimited Spline spaces Shift invariant subspace: General subspace in a Hilbert space Smoothness constraints: Sparse vector model (compressed sensing): Analog Version? 9

Introduction Input Signals Nonlinearities Reconstruction Broad Sampling Framework Beyond Ideal Bandlimited Sampling Broad class of input signals: Subspace priors, smoothness priors Sparse analog signals: Signals restricted to bands Analog sampling + compressed sensing General pre-filters Perfect compensation of nonlinear distortions Nonideal reconstruction filters 10

Introduction Input Signals Nonlinearities Reconstruction Outline 1. Input Signals: Subspace methods: Perfect recovery from linear generalized samples Smoothness priors: Minimax approximations Sparsity priors: Brief overview of compressed sensing (CS) for finite vectors CS of analog signals: Blind sampling of multiband signals 2. Nonlinearties: Perfect recovery in the presence of nonlinear distortions 3. Nonideal reconstruction: Minimax approximation with simple kernels 11

Introduction Input Signals Nonlinearities Reconstruction Non-Ideal Linear Sampling Non- ideal sampling Sampling functions s(-t) t=n. T Examples: Generalized antialiasing filter Local averaging 0 Sampling space S: In the sequel: T=1 (Riesz: Any linear and bounded acquisition) Electrical circuit Δ 12

Introduction Input Signals Nonlinearities Reconstruction Perfect Reconstruction Given s(t) which signals can be perfectly reconstructed? Key observation: sampling space Knowing is equivalent to knowing (Riesz basis): If x(t) is in S then if is a frame and perfect reconstruction is possible 13

Introduction Input Signals Nonlinearities Reconstruction Shannon Revisited Perfect reconstruction scheme: Bandlimited sampling: 14

Introduction Nonlinearities Input Signals Reconstruction Mismatched Sampling What if x(t) lies in a subspace where A is generated by a(t) ? If then PR impossible since If then PR possible (Christansen and Eldar, 2005) Perfect Reconstruction in a Subspace: 15

Introduction Input Signals Nonlinearities Reconstruction Examples Point-wise sampling of c[n]=x[n] corresponding to s(t)=δ(t) Can recover x(t) as long as : (Unser and Aldroubi 94) Bandlimited sampling: Can x(t) be recovered even though it is not bandlimited? 16

Introduction Input Signals Nonlinearities Reconstruction Perfect Recovery 1. Compute convolutional inverse of 2. Convolve the samples with 3. Reconstruct with 17

Introduction Input Signals Nonlinearities Reconstruction Summary: Perfect Recovery In A Subspace General input signals (not necessarily BL) General samples (anti-aliasing filters) Results hold also for nonuniform sampling and more general spaces 18

Introduction Nonlinearities Input Signals Reconstruction Smoothness Prior (Eldar 2007) No subspace information but Many consistent solutions Motivation: Want to be close to x(t) Minimize the worst-case difference: Complicated problem but … simple solution optimal interpolation kernel 19

Introduction Input Signals Nonlinearities Reconstruction From Smoothness to Compressed Sensing Non ideal sampling of analog signals PR with subspace prior Approximations with smoothness prior Sparsity prior : Discrete signals Sparse prior: Samples: Can x be reconstructed from c? If every 2 K columns of A are linearly independent then there is a unique K-sparse signal (Donoho and Elad 03) Key observation: c can be relatively short and still contain the entire information about x 20

Introduction Input Signals Nonlinearities Reconstruction Joint Sparsity Multiple measurement vectors (MMV): C=AX Each column of X is K-sparse The non-zero values share a common location set Theorem Let X have . If then X is the unique sparsest solution set (Chen and Huo 06, Mishali and Eldar 07) 21

Introduction Input Signals Nonlinearities Reconstruction Algorithms SMV Efficient algorithms: Basis pursuit Matching pursuit Others To overcome NP-hard MMV Efficient algorithms: M-Basis pursuit M-Matching pursuit Others Results and Algorithms Inherently Discrete 22

Introduction Input Signals Nonlinearities Reconstruction Analog Compressed Sensing What is analog compressed sensing? A signal with a multiband structure in some basis no more than N bands, max width B, bandlimited to Previous methods for analog CS involve discretization or finite models (R. Baraniuk , J. Laska, S. Kirolos, M. Duarte, T. Ragheb, Y. Massoud, A. Gilbert, M. Iwen, M. Strauss, J. Tropp, M. Wakin, D. Baron) Our model is inherently continuous: each band has an uncountable number of non-zero elements No finite basis! 23

Introduction Input Signals Nonlinearities Reconstruction Goals 1 Minimal rate Blind Sampling 2 Perfect reconstruction Blind Reconstruction 3 Blind system 24

Introduction Input Signals Nonlinearities Reconstruction Non-Blind Scenario Theorem Landau (1967) Average sampling rate is constant with Lebesgue measure Minimal-rate sampling and reconstruction (NB) with known band locations (Lin and Vaidyanathan 98) Subspace scenario Half blind system (Herley and Wong 99, Venkataramani and Bresler 00) 25

Introduction Input Signals Nonlinearities Reconstruction Minimal Sampling Rate Question: What is the minimal sampling rate that allows blind perfect reconstruction with arbitrary sampling/reconstruction methods? Theorem Mishali and Eldar (2007) The minimal sampling rate is doubled (for ) Minimal sampling rate for our set M: 2 NB 26

Introduction Nonlinearities Input Signals Reconstruction Sampling Multi-Coset: Periodic Non-uniform on the Nyquist grid In each block of samples, only are kept, as described by 2 3 Analog signal Point-wise samples 0 0 3 2 0 2 3 27

Introduction Input Signals Nonlinearities Reconstruction The Sampler DTFT of sampling sequences Length. known Constant in vector form unknowns matrix known 28

Introduction Input Signals Nonlinearities Reconstruction Objectives Goal: Recover Problems: 1. Undetermined system – non unique solution (p<L) 2. Continuous set of linear systems Observation: is sparse 29

Introduction Input Signals Nonlinearities Reconstruction Uniqueness Theorem (Mishali and Eldar, 2007) 30

Introduction Input Signals Nonlinearities Reconstruction We Say NO to Discretization ! Choose a dense grid of Solve for each Interpolate Disadvantages: Loose perfect reconstruction Large computational complexity Sensitivity to noise 31

Introduction Input Signals Nonlinearities Reconstruction Paradigm Solve finite problem Reconstruct 0 1 2 3 4 5 6 32

Introduction Input Signals Nonlinearities Reconstruction Once S is Known… Solve finite problem Reconstruct exactly by 33

Introduction Input Signals Nonlinearities Reconstruction Continuous to Finite CTF block MMV Solve finite problem Continuous Reconstruct Finite 34

Introduction Input Signals Nonlinearities Reconstruction CTF Fundamental Theorem Mishali and Eldar (2007) 35

Introduction Input Signals Nonlinearities Reconstruction Algorithm CTF Continuous-to-finite block: Compressed sensing for analog signals Perfect reconstruction at minimal rate Blind system: band locations are unkown 36

Introduction Input Signals Nonlinearities Reconstruction Summary: Perfect Reconstruction Until Now: Perfect reconstruction from subspace samples, sparse samples Minimax reconstruction from smooth signals Limitations: Linear sampling Ideal interpolation kernels Coming up …. Perfect compensation for nonlinear distortions Minimax interpolation with simple kernels 37

Introduction Nonlinearities Input Signals Reconstruction Nonlinear Sampling s(-t) Memoryless nonlinear distortion t=n Saturation in CCD sensors Dynamic range correction Optical devices High power amplifiers Many applications… No theory! 38

Introduction Input Signals Nonlinearities Reconstruction Perfect Reconstruction Theorem (uniqueness): If and m(t) is invertible and smooth enough then y(t) can be recovered exactly (Dvorkind, Eldar, Matusiak 2007) Setting: m(t) is invertible with bounded derivative y(t) is lies in a subspace A Uniqueness same as in linear case! Proof: Based on extended frame perturbation theory and geometrical ideas 39

Introduction Nonlinearities Input Signals Reconstruction Algorithm: Linearization Transform the problem into a series of linear problems: 1. Initial guess y 0 2. Linearization: Replace m(t) by its derivative around y 0 3. Solve linear problem and update solution yn error in samples yn+1 solving linear problem correction Questions: 1. Does the algorithm converge? 2. Does it converge to the true input? 40

Introduction Input Signals Nonlinearities Reconstruction Optimization Based Approach Main idea: 1. Minimize error in samples 2. From uniqueness if Perfect reconstruction where global minimum of Difficulties: 1. Nonlinear, nonconvex problem 2. Defined over an infinite space Theorem : Under the previous conditions any stationary point of is unique and globally optimal (Dvorkind, Eldar, Matusiak 2007) Our algorithm traps a stationary point! 41

Introduction Input Signals Nonlinearities Reconstruction Simulation Example Optical sampling system: optical modulator ADC 42

Introduction Input Signals Nonlinearities Reconstruction Simulation Third iteration: with First iteration: Initialization 43

Introduction Input Signals Nonlinearities Reconstruction Constrained Reconstruction reconstruction space Consistent reconstruction (Unser and Aldroubi 94, Eldar 03, 04) Unique solution possible only if: Problem: Resulting error can be quite large 44

Introduction Input Signals Nonlinearities Reconstruction Minimax Reconstruction (Eldar and Dvorkind, 2005) Motivation: Want to be close to x(t) Best approximation in W is but can’t be attained from Minimize the worst-case difference: Complicated problem but …. Simple solution: Comparison: 45

Introduction Nonlinearities Input Signals Reconstruction Example: Audio Processing Down-up sampling with non-ideal filtering: LPF 1 8[k. Hz] H /2 LPF 2 x 2 Original signal No processing (NE=0. 81) Consistent (NE=0. 87) Regret (NE=0. 28) Orthogonal Projection (NE=0. 27) 46

Conclusion Beyond Ideal Bandlimited Sampling Broad class of input signals: Subspace priors, smoothness priors Compressed sensing for analog signals Compensations for many practical distortions Applicable to a wide host of sampling problems Can beat Nyquist and aliasing using the right tools! 47

References Y. C. Eldar and T. Dvorkind, "A Minimum Squared-Error Framework for Generalized Sampling, " IEEE Trans. Signal Processing, vol. 54, no. 6, pp. 2155 -2167, June 2006. M. Mishali and Y. C. Eldar, "Blind Multi-Band Signal Reconstruction: Compressed Sensing for Analog Signals, “ submitted to IEEE Trans. on Signal Processing, Sep. 2007. T. G. Dvorkind, Y. C. Eldar and E. Matusiak, "Nonlinear and Non-Ideal Sampling: Theory and Methods, " submitted to IEEE Trans. on Signal Processing, Nov. 2007. M. Mishali and Y. C. Eldar, "Reduce and Boost: Recovering Arbitrary Sets of Jointly Sparse Vectors", submitted to IEEE Trans. on Signal Processing, Feb. 2008. Y. C. Eldar and M. Unser, "Nonideal Sampling and Interpolation from Noisy Observations in Shift-Invariant Spaces, " IEEE Trans. Signal Processing, Vol. 54, No. 7, pp. 2636 -2651, July 2006. Y. C. Eldar, "Sampling and Reconstruction in Arbitrary Spaces and Oblique Dual Frame Vectors ", J. Fourier Analys. Appl. , vol. 1, no. 9, pp. 77 -96, Jan. 2003. O. Christensen and Y. C. Eldar, "Oblique Dual Frames and Shift-Invariant Spaces, " Applied and Computational Harmonic Analysis, vol. 17/1, pp. 48 -68, July 2004. 48

Details: x m’(t) y where: Maximal angle between the spaces 49