Image Enhancement Method via Blur and Noisy Image

Image Enhancement Method via Blur and Noisy Image Fusion Marius Tico, Kari Pulli Nokia Research Center Palo Alto, CA, USA

Outline • Introduction • Related work • Proposed method • Photometric calibration • Luminance fusion • Color fusion • Experiments and examples • Conclusions 2 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

Introduction • Image capturing in dim light is challenging, especially for miniature cameras • Tuning camera parameters tradeoffs between different quality factors • Aperture increase: ensures more light but reduces depth of field • ISO sensitivity increase: amplifies the noise • Exposure time increase: may result in motion blur 3 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

Related work • De-noise a single image captured with short exposure (and high ISO): • e. g. , [Donoho and Johnstone, 1994], [Starck, Candes, and Donoho, 2002], [Portilla, et al. 2003] • Due to short exposure and quantization color may not be recovered • De-blurring a single image captured with long exposure time (and small ISO) • Blind image de-convolution [You and Kaveh '96 & '99], [Chan and Wong '98], [Fergus ’ 06], [Shan ‘ 08] • Computationally complex, plus reliability problems • Assume spatially invariant blur PSF • Fusing blurry / noisy image pairs [Tico, et al. ‘ 06], [Yuan. et al. ‘ 07] • Avoid blind de-convolution by estimating the blur PSF from the two input images • Assume blur PSF is spatially invariant • Using additional hardware • Extra video camera for motion estimation [Ben-Ezra, Nayar ‘ 04] • Flash: effective only for close objects, and change the mood of the scene • Opto-mechanical stabilizer systems: inefficient for long exposure times 4 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

The proposed method • Short exposed frame: noisy but not affected by motion blur • Long exposed frame: better colors, less noise, but blurry Time Short exposed: dark, noisy and less color 5 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli Long exposed: good color but blurry

The block diagram of the proposed solution 6 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli Image fusion Registration Photocalibration

Photometric calibration • Build the joint histogram (comparagram) 7 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

Photometric calibration (cont’d) • Identify most likely correspondences (xi, yi) between pixel values in the two images 8 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

Photometric calibration (cont’d) • Estimate the Brightness Transfer Function (BTF) 9 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

Photometric calibration (cont’d) Calibration curves Short exposed Long exposed 10 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli Calibrated short exposed

Image fusion 11 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli Image fusion Registration Photocalibration

Luminance fusion • Formally we can write the following model for the two images • By applying an orthogonal wavelet transform the model becomes • Taking advantage of the de-correlation in the wavelet domain we propose a MMSE diagonal estimator of the form where • By minimizing the mean square error results the expression for the optimal weight 12 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

Luminance fusion (cont’d) • Prefer the noisy image near edges, and the blurry image in smooth areas • Edges can be detected based on the difference between the two images 13 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

Different levels of blur 24. 61 d. B 14 33. 76 d. B (3 x 3) 29. 81 d. B (5 x 5) 35. 09 d. B 34. 03 d. B Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli 25. 58 d. B (11 x 11) 33. 50 d. B 22. 78 d. B (21 x 21) 33. 40 d. B

Different levels of noise 27. 80 d. B (7 x 7) 15 28. 13 24. 61 d. B 35. 32 d. B 33. 62 d. B Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli 22. 12 d. B 32. 47 d. B 20. 21 d. B 31. 48 d. B

Comparative simulations 24. 92 d. B Wavelet HT 24. 73 d. B Curvelet HT 27. 43 d. B 31. 05 d. B Wiener 29. 44 d. B 31. 19 d. B 21. 02 d. B (7 x 7) 16 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli [Portilla, et al. 2003]

Comparative simulations (cont’d) • The CPU times measured on Intel Core 2 Duo 2. 20 GHz Proposed method 31. 19 d. B 17 CPU Time: 0. 8 sec Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli Portilla, et al. 2003 31. 05 d. B CPU Time: 31 sec

Noise variance estimation • Noise variance in the photo-calibrated image = noise variance in the short exposed image = brightness transfer function = short exposed image in pixel Short exposed - calibrated 18 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli Noise variance

Color fusion • Emphasize colors from long exposed image except the areas where the long exposed image is saturated • Weighting functions • Saturation weight function (left) • Blurriness weight function (right) 19 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

Color fusion (cont’d) • Example of color weighting in accordance to the two rules (saturation and blurriness ) Short exposed 20 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli Color weight Long exposed

Example Short exposed 21 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli Result Long exposed

Example (local blur) Short exposed 22 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli Result Long exposed

Example (local blur) De-blurring 23 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli Result Long exposed

Low Light Imaging Short exposed: 1/30 sec, ISO 100 24 Photometric aligned short exposed Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli Long exposed: 1 sec, ISO 200 Output 24

Conclusions • We proposed a new approach of image restoration by fusing two differently degraded images • Short exposed image affected by noise • Long exposed image that may be affected by motion blur • In contrast to previous blurred/noisy image fusion our approach is not applying convolution on the blurry image • The main advantages: de- • Can deal with spatially variant blur due to parallax, or object motion • Lower computational complexity • Since the proposed approach is not dependent of blur spatial invariance it can be used also for fusing images with different aperture • Small aperture image, affected by noise but capturing a large depth of field • Large aperture image, less noisy but affected by blur due to narrow depth of field 25 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

Thank you! 26 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli

References • De-noise a single image captured with short exposure (and high ISO) • Several de-noising methods available in the literature, e. g. , • D. L. Donoho and I. M. Johnstone, “Ideal spatial adaptation by wavelet shrinkage, ” Biometrika, vol. 81, pp. 425– 455, 1994. • Jean-Luc Starck, Emmanuel J. Candes, and David L. Donoho, “The Curvelet Transform for Image Denoising, ” IEEE Trans. on Image Processing, vol. 11, no. 6, pp. 670– 684, 2002. • J. Portilla, V. Strela, M. Wainwright, E. P. Simoncelli, “Image denoising using scale mixtures of Gaussians in the wavelet domain”, IEEE Trans. on Image Processing 12, No. 11, 1338– 1351, 2003. • Some are too complex a mobile device computational power • De-blurring a single image captured with long exposure time (and small ISO) • Blind image de-convolution • Q. Shan, J. Jia, and A. Agarwala, “High-quality motion deblurring from a single image”, SIGGRAPH, 2008. • R. Fergus, B. Singh, A. Hertzmann, S. T. Roweis, and W. T. Freeman, “Removing camera shake from a single photograph”, SIGGRAPH, 2006. • High complexity and insufficient robustness for consumer applications • Using additional camera for motion estimation • M. Ben-Ezra, S. K. Nayar, "Motion-based motion deblurring", IEEE Trans. on PAMI, 26, No. 6, 689 -698, 2004 • Using specially designed CMOS sensors • X. Liu, A. Gamal, "Synthesis of High Dynamic Range Motion Blur Free Image From Multiple Captures", IEEE Trans. on Circuits and Systems I: Findamental Theory and applications, vol. 50, no. 4, 530 -539, 2003. • Fusing blurry / noisy image pairs • Marius Tico, Mejdi Trimeche, and Markku Vehvil¨ainen, “Motion blur identification based on differently exposed images”, ICIP, 2006. • Lu Yuan, Jian Sun, Long Quan, and Heung-Yeung Shum, “Image deblurring with blurred/noisy image pairs, ” SIGGRAPH 2007. 27 Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli