SOS Boosting of Image Denoising Algorithms Yaniv Romano

SOS Boosting of * Image Denoising Algorithms Yaniv Romano Electrical Engineering Department The Technion – Israel Institute of technology Haifa 32000, Israel SPARS 2015 Cambridge, UK * Joint work with Michael Elad The research leading to these results has received funding from the European Research Council under European Union's Seventh Framework Program, ERC Grant agreement no. 320649, and by the Intel Collaborative Research Institute for Computational Intelligence

Leading Image Denoising Methods Are built upon powerful patch-based (local) image models: § § § Non-Local Means (NLM): self-similarity within natural images K-SVD: sparse representation modeling of image patches BM 3 D: combines a sparsity prior and non local self-similarity Kernel-regression: offers a local directional filter EPLL: exploits a GMM model of the image patches … Today we present a way to improve various such state-of-the-art image denoising methods, simply by applying the original algorithm as a “black-box” several times Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 3

Background

Boosting Methods for Denoising Denoise In image denoising, there are two sources of possible problems: q Residual noise in the output image, and q Residual content in the method noise Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 5

Existing Boosting Algorithms SAIF [Talebi et al. (’ 12)] chooses automatically the local improvement mechanism: Diffusion or Twicing Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 6

SOS Boosting of Image Denoising Algorithms SIAM Journal on Imaging Sciences, 2015

Strengthen - Operate - Subtract Boosting q Given any denoiser, how can we improve its performance? Denoise Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 8

Strengthen - Operate - Subtract Boosting q Given any denoiser, how can we improve its performance? Denoise Previous Result I. Strengthen the signal Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 9

Strengthen - Operate - Subtract Boosting q Given any denoiser, how can we improve its performance? Denoise Previous Result I. Strengthen the signal II. Operate the denoiser Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 10

Strengthen - Operate - Subtract Boosting q Given any denoiser, how can we improve its performance? Denoise Previous Result I. Strengthen the signal II. Operate the denoiser III. Subtract the previous estimation from the outcome Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 11

Strengthen - Operate - Subtract Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 12

Image Denoising – A Matrix Formulation q Observation: Denoising Algorithm Non-Linear Part (decisions/switches) Spatially adaptive weighted averages True for NLM, Kernel-regression, BM 3 D, K-SVD, and many other methods q We study the convergence of the SOS using only the linear part: q Kernel-based methods can be represented as row-stochastic positive definite matrices [Milanfar (’ 13)] • Has eigenvalues in the range [0, …, 1] q What about sparsity-based denoising methods [Elad & Aharon (‘ 06)]? Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 13

Sparsity Model – The Basics • D … = Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad D 15

K-SVD Image Denoising Noisy Image Initial Dictionary [Elad & Aharon (‘ 06)] Using KSVD Update the Dictionary Denoise each patch Using OMP Denoised Patch A linear combination of few atoms Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 16

K-SVD Image Denoising Noisy Image Initial Dictionary [Elad & Aharon (‘ 06)] Using KSVD Reconstructed Image Update the Dictionary Denoise each patch Using OMP Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 17

K-SVD: A Matrix Formulation SOS Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 18

Back to the convergence Study… For these denoising algorithms, the SOS boosting converges to q What about the non-linear part and its influence? More on this can be found in our paper … Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 19

Generalization q We introduce two parameters that modify § The steady-state outcome § The requirements for convergence (the eigenvalues range), and § The rate of convergence q The parameter , affects the steady-state outcome: q The second parameter, , controls the rate-of-convergence, without affecting the steady-state: Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 20

Generalization Largest eigenvalue of the error’s transition matrix Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 21

Graph-Based Interpretation

Graph Laplacian Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 23

Graph Laplacian Regularization q The regularization can be defined as [Elmoataz et al. (’ 08), Bougleux et al. (‘ 09)] Seeks for an estimation that is close to the noisy version While promoting similar pixels to remain similar q Another option is to integrate the filter also in the data fidelity term [Kheradmand Milanfar (’ 13)] Using the adaptive filter as a weight-matrix Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 24

Graph Laplacian Regularization q Another natural option is to minimize the following cost function Seeks for an estimation that is close to the denoised version Its closed-form solution is the steady-state outcome of the SOS The SOS boosting acts as a graph Laplacian regularizer q More on this topic can be found in our paper … Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 25

Experiments

Results q We successfully boost several state-of-the-art denoising algorithms: § K-SVD, NLM, BM 3 D, and EPLL § Without any modifications, simply by applying the original software as a “black-box” q We manually tuned two parameters § – signal emphasis factor § – noise level, which is an input to the denoiser • Since the noise level of Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad is higher than the one of 27

Quantitative Comparison q Average boosting in PSNR* over 5 images (higher is better): Noise std 10 20 25 50 75 100 Improved Methods K-SVD NLM BM 3 D EPLL 0. 13 0. 22 0. 26 0. 77 1. 26 0. 81 0. 44 0. 34 0. 41 0. 30 0. 56 0. 36 0. 01 0. 02 0. 03 0. 07 0. 11 0. 14 0. 13 0. 25 0. 26 0. 30 0. 32 0. 27 Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 28

Visual Comparison: K-SVD q Original K-SVD results, 29. 06 d. B Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 29

Visual Comparison: K-SVD q SOS K-SVD results, 29. 41 d. B Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 30

Visual Comparison: EPLL q Original EPLL results, Forman House 32. 44 d. B 32. 07 d. B Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 31

Visual Comparison: EPLL q SOS EPLL results, Forman House 32. 78 d. B 32. 38 d. B Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 32

Visual Comparison: All Noisy image KSVD (31. 20) SOS KSVD (31. 91) NLM (30. 02) BM 3 D (31. 88) SOS NLM (30. 56) SOS BM 3 D (31. 94) Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad EPLL (30. 88) SOS EPLL (31. 15) 33

Visual Comparison: All Noisy image KSVD (33. 72) NLM (31. 64) BM 3 D (34. 66) EPLL (33. 62) SOS KSVD (34. 4) SOS NLM (32. 3) SOS BM 3 D (34. 7) SOS EPLL (34. 1) Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 34

Time to Conclude

Conclusions Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 36

We are Done… Thank you! Questions? Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 37

Reducing the “Local-Global” Gap Patch-Disagreement as a Way to Improve K-SVD Denoising ICASSP, 2015

Reaching a Consensus q It turns out that the SOS boosting reduces the local/global gap, which is a shortcoming of many patch-based methods: § Local processing of patches VS. the global need in a whole denoised result q We define the local disagreements by Disagreement patch Local independent denoised patch Globally averaged patch q The disagreements § Naturally exist since each noisy patch is denoised independently § Are based on the intermediate results Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 40

“Sharing the Disagreement” q Inspired by the “Consensus and Sharing” problem from game-theory: § There are several agents, each one of them aims to minimize its individual cost (i. e. , representing the noisy patch sparsely) § These agents affect a shared objective term, describing the overall goal (i. e. , obtaining the globally denoised image) q Imitating this concept, we suggest sharing the disagreements Noisy patches Noisy image Patch-based denoising Patch Avg. Est. Image Disagreement per-patch Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 41

Connection to SOS Boosting q Interestingly, for a fixed filter matrix and the SOS boosting are equivalent , “sharing the disagreement” q The connection to the SOS is far from trivial because § The SOS is blind to the intermediate results (the independent denoised patches, before patch-averaging) § The intermediate results are crucial for “sharing the disagreement” approach The SOS boosting reduces the local-global gap Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 42

Guaranteed improvement?

The General Case [Milanfar (’ 12)] SOS Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 44

The General Case [Milanfar (’ 12)] SOS Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 45

MSE of the SOS boosting Larger than 1 SOS Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad Smaller than 1 46

Gaining Improvement The SOS boosting is always able to improve a suboptimal denoiser SOS Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 47

Minimizing the MSE SOS Boosting of Image Denoising Algorithms By Yaniv Romano and Michael Elad 48