Sparsity Based Poisson Denoising and Inpainting Raja Giryes

Sparsity Based Poisson Denoising and Inpainting Raja Giryes, Tel Aviv University Joint work with Michael Elad, Technion

2 Agenda • • • Problem Definition – Poisson Denoising Existing Poisson Denoising Methods Poisson Greedy Algorithm Experimental results Poisson Inpainting

3 Denoising Problem • Original unknown image • is a noisy measurement of x. • The goal is to recover x from.

4 Gaussian Denoising Problem • where is a zero-mean white Gaussian noise with variance , i. e. , each element.

5 Gaussian Noisy Measurements • Another perspective for the Gaussian denoising problem: • Look at the measurements as Gaussian distributed with mean equal to the original signal • The variance determines the noise power.

6 Poisson Noisy Measurements • The measurements are Poisson Distributed • Poisson noise is not an additive noise, unlike the Gaussian case. • The noise power is measured by the peak value:

7 Poisson Denoising Problem • Noisy image distribution: • is an integer. • large • small • . .

8 Poisson Denoising Problem

9 Poisson Denoising Problem

10 Poisson Denoising Applications • • • Tomography – CT, PET and SPECT Astrophysics Fluorescence Microscopy Night Vision Spectral Imaging etc.

11 Tomography Slices of skeletal SPECT image [Takalo , Hytti and Ihalainen 2011]

12 Fluorescence Microscopy C. elegans embryo labeled with three fluorescent dyes [Luisier, Vonesch, Blu and Unser 2010]

13 Astrophysics XMM/Newton image of the Kepler SN 1604 supernova [Starck, Donoho and Candès 2003]

14 Agenda • • • Problem Definition – Poisson Denoising Existing Poisson Denoising Methods Poisson Greedy Algorithm Experimental results Poisson Inpainting

15 Denoising Methods • Many denoising methods exists. • However, most of them assume a Gaussian model for the noise. • We have two options: • Use a transformation that converts the noise to be Gaussian. • Work directly with the Poisson model.

16 The Anscombe Transform • The Anscombe transform converts Poisson distributed noise into an approximately Gaussian distributed data with variance 1 using the following formula elementwise [Anscombe, 1948]. Valid only when peak>4

17 Poisson Log-likelihood • We will work directly with the Poisson data. • By maximizing the log-likelihood of the Poisson distribution we get the following minimization problem A prior is needed • Reminder: In the Gaussian case we had

18 Sparsity Prior for Poisson Denoising (1) • Regular sparsity prior leads to which is a non-negative optimization problem • Instead we use that yields the following • D is a given dictionary. • counts the non-zero elements

19 Sparsity Prior for Poisson Denoising (2) • The minimization problem is likely to be NP-hard. • Approximations are needed.

20 l 1 Relaxation • One option is to use l 1 relaxation • is a relaxation parameter. • This problem can be solved using the SPIRAL algorithm [Harmany et al. , 2012].

21 Non-local PCA (NLPCA) • GMM (Gaussian Mixture Model) based method. • Cluster the noisy patches into small number of large groups. • For each cluster train a PCA subspace • Non-local Sparse PCA (NLSPCA) ▫ Uses l 1 regularization with NLPCA. • Binning ▫ Aggregate nearby pixels to improve SNR. ▫ Denoise down-sampled image. ▫ Interpolate recovered image to return to initial size. [Salmon, Harmany, Deledalle, Willett 2013]

22 Agenda • • • Problem Definition – Poisson Denoising Existing Poisson Denoising Methods Poisson Greedy Algorithm Experimental results Poisson Inpainting Novel Part

23 Exponential Sparsity Prior Zero entries Non-zero entries [Salmon et al. 2012, Giryes and Elad 2012]

24 Poisson Greedy Algorithm - Summary • Divide the image into set of overlapping patches. • Cluster (using Gaussian filtering) the noisy patches into large number of small groups. • Each group of patches is assumed to have the same non-zero locations (support) in their representations. • A global dictionary is used for all groups of patches. • Having the reconstructed patches we form the final image by averaging.

25 Dictionary Learning • Joint dictionary D and representation with a fixed support learning [Smith, Elad 2013]. • after we have the representation of all the patches and their supports we minimize: • Global initial dictionary for all images • Trained using the following image

26 Our Algorithm vs. NLPCA Poisson Greedy Algorithm NLPCA • Large number of clusters. • Small cluster size. • Clustering using Gaussian filtering. • Global dictionary for all patches. • Dictionary learning based approach. • • Small number of clusters. Large cluster size. Clustering using k-means. Local dictionary for each cluster. • GMM based approach.

27 Algorithm Summary. . … Gaussian filtering . . … Extracting overlapping patches Applying Poisson greedy algorithm for each group Patch grouping . . … . . … Dictionary learning Averaging patches . . …

28 Poisson Greedy Algorithm-Sparse Coding • Input: Group of noisy patches • Initialization: • While t<k • t=t+1 • Find new support element and representations: • Update the support • Form patches estimate:

29 Boot-strapped Stopping Criterion • Ideally we want to select different number of non -zeros for each patch. • We want to add elements to the support till the error with respect to the original patch (in the original image) stops decreasing. • We do not have access to the original image. • Use the patches of the estimated image from the previous iteration.

30 Agenda • • • Problem Definition – Poisson Denoising Existing Poisson Denoising Methods Poisson Greedy Algorithm Denoising Results Poisson Inpainting Novel Part

31 Experiment- Parameter Setting • • • Patches of size 20 by 20. Patches clustered to groups of size 50. Initial cardinality of the patches is k=2. 5 dictionary learning iterations. Repeat the process one time with re-clustering based on the recovered image.

32 Noisy Image Max y value = 7 Peak = 1

33 Poisson Greedy Algorithm. Dictionary learned atoms: Method Peak = 1 Our 22. 59 db NLSPCA 20. 37 db BM 3 Dbin 19. 41 db [Giryes and Elad 2013].

34 Poisson Greedy Algorithm Method Peak = 1 Our 22. 59 db NLSPCA 20. 37 db BM 3 Dbin 19. 41 db [Salmon et al. 2013].

35 Poisson Greedy Algorithm Method Peak = 1 Our 22. 59 db NLSPCA 20. 37 db BM 3 Dbin 19. 41 db [Makitalo and Foi 2011]

36 Original Image

37 Noisy Image Max y value = 3 Peak = 0. 2

38 Poisson Greedy Algorithm Method Peak = 0. 2 Our 24. 16 db NLSPCA 22. 98 db BM 3 Dbin 23. 16 db [Giryes and Elad 2013].

39 Poisson Greedy Algorithm Method Peak = 0. 2 Our 24. 16 db NLSPCA 22. 98 db BM 3 Dbin 23. 16 db [Salmon et al. 2013].

40 Poisson Greedy Algorithm Method Peak = 0. 2 Our 24. 16 db NLSPCA 22. 98 db BM 3 Dbin 23. 16 db [Makitalo and Foi 2011]

41 Original Image

42 Noisy Image Max y value = 8 Peak = 2

43 Poisson Greedy Algorithm Method Peak = 2 Our 24. 76 db NLSPCA 23. 23 db BM 3 Dbin 24. 23 db [Giryes and Elad 2013].

44 Poisson Greedy Algorithm Method Peak = 2 Our 24. 76 db NLSPCA 23. 23 db BM 3 Dbin 24. 23 db [Salmon et al. 2013].

45 Poisson Greedy Algorithm Method Peak = 2 Our 24. 76 db NLSPCA 23. 23 db BM 3 Dbin 24. 23 db [Makitalo and Foi 2011]

46 Original Image

47 Recovery Results • 8 Test images. • 6 peak levels (0. 1, 0. 2, 0. 5, 1, 2, 4). • Best average recovery error for 5 out of 6 peak values. • Second best for peak =1 (difference of 0. 02 db).

48 Agenda • • • Problem Definition – Poisson Denoising Existing Poisson Denoising Methods Poisson Greedy Algorithm Denoising Results Poisson Inpainting Novel Part

49 The Poisson Inpainting Problem • Some of the pixels in are occluded. • The mask defines missing and given pixels in the measured image + Noisy Image =

50 Poisson Inpainting Objective • The Poisson Inpainting minimization problem • We approximate this problem using a greedy algorithm as before.

51 Noise Estimation for Inpainting • Having a recovery , we replace each unknown pixel in with a noisy pixel generated from. + • We get a noisy image for which we can apply the regular dictionary update steps.

52 Inpainting Results 23. 86 d. B Peak = 1, 20% Missing Pixels

53 Inpainting Results 22. 83 d. B Peak = 1, 40% Missing Pixels

54 Inpainting Results 21. 02 d. B Peak = 1, 60% Missing Pixels

55 Inpainting Results Average over four different test images

56 Inpainting Results 24. 34 d. B Peak = 1

57 Inpainting Results 23. 58 d. B Peak = 2

58 Inpainting Results 22. 72 d. B Peak = 2

59 Inpainting Results 19. 76 d. B Peak = 1

60 Conclusion • • • Poisson based denoising Sparse representation for Poisson noise Greedy Poisson algorithm State-of-the-art denoising results Poisson inpainting algorithm

61 Questions?