Noise Estimation from a Single Image Ce Liu

Noise Estimation from a Single Image Ce Liu William T. Freeman Richard Szeliski Sing Bing Kang

Parameter Tweaking in Computer Vision q Computer vision algorithms suffer from hand tuning parameters for particular images or image sequences q We want vision algorithms that behave properly under varying lighting conditions, blur levels and noise levels q Our work is one step in that direction § Given an image, estimate the noise level § Modify vision algorithms to be independent of noise

Image Noise Is Important in Vision q In image denoising the noise is assumed to be known as Additive Gaussian White Noise (AWGN) q However, in real applications the noise is unknown and non-additive q Many other computer vision algorithms also explicitly or implicitly assume the type and level of image noise q Hard to make vision algorithms fully automatic without knowing noise

Noise Level Function (NLF) q The standard deviation of noise s is a function of image brightness I q Measurable by fixing the camera and taking multiple shots of a static scene s q For each pixel: § Mean: I § Standard deviation: s q NLF depends on camera, ISO, shutter speed, aperture q Our goal is to estimate NLF from a single image § How to estimate noise without separating noise and signal? I

An Example Image

Piecewise Smooth Image Prior Affine model = + Patch Signal Brightness mean I Standard deviation For each RGB channel: s s Red Residual s Green 0. 2 0. 1 0 0 0. 5 1 I 0 0 0. 5 Brightness 1 I 0 Standard deviation s Blue 0 0. 5 1 I

Piecewise Smooth Image Prior = + Standard deviation Patch s s Red Signal Residual s Green 0. 2 0. 1 0 0 0. 5 1 I 0 0 0. 5 Brightness 1 I 0 Blue 0 0. 5 1 I

Piecewise Smooth Image Prior = + Standard deviation Patch s s Red Signal Residual s Green 0. 2 0. 1 0 0 0. 5 1 I 0 0 0. 5 Brightness 1 I 0 Blue 0 0. 5 1 I

Segmentation-based Approach Observed image

Segmentation-based Approach Oversegmentation

Segmentation-based Approach Signal

Segmentation-based Approach Residual= noise + unmodelled image variation

Estimate NLFs s s Residual std. dev. s I I Brightness q q q Assume brightness mean I is accurate estimate Standard deviation s is an over-estimate: (may contain signal) The lower envelope is the upper bound of NLF I

Issues s s Residual std. dev. s I I Brightness q Should the curve be strictly and tightly below the points? I

Issues s s Residual std. dev. s I I Brightness q Should the curve be strictly and tightly below the points? q How to handle the missing data? I

Issues s s Residual std. dev. s I I Brightness q Should the curve be strictly and tightly below the points? q How to handle the missing data? q Correlation between RGB channels? I

Solutions s s Residual std. dev. s I I Brightness q Formulate the inference problem in a probabilistic framework q Learn the prior of noise level functions I

Outline q Over-segmentation and per-segment variance analysis q Learning the priors of noise level functions (NLF) § Synthesize CCD noise § Sample noise level functions § Learn the prior of noise level functions q Inference: estimate the upper bound of NLF § Bayesian MAP to estimate NLFs for RGB channels q Applications § Adaptive bilateral filtering § Canny edge detection

Camera Noise L Scene Radiance Atmospheric Attenuation Lens/ geometric Distortion CCD Imaging/ Bayer Pattern Fixed Pattern Noise Quantization Noise I Digital Image A/D Converter Shot Noise Dark Current Noise Camera Irradiance Thermal Noise Gamma Correction White Balancing Interpolation/ Demosaic t q Noise model q Camera response function (CRF) f: download from Columbia camera response function database (used 196 typical CRFs) Tsin et. al. Statistical calibration of CCD image process. ICCV, 2001 Dependent noise: Independent noise:

Synthesize CCD Noise I Estimate NLF Camera response function: f Dependent noise: Independent noise:

Sample NLFs by Varying the Parameters 0. 02 Camera response function (CRF) f 0. 02 0. 18 Dependent noise: Independent noise: 0. 02 0. 18 0. 04 0. 06

The Prior of NLFs

Likelihood Function q The estimated standard deviation should be probabilistically bigger than and close to the true value I q Bayesian MAP inference

Validation (1): Synthetic Noise q Add synthetic CCD noise, estimate, compare to the ground truth — — — estimated

Validation (2): Measure NLF of a Real Camera q q 29 images were taken under the same settings (the camera is not in the database for training) The real NLF is obtained by computing mean and variance per pixel

Validation (3): Robustness Test q Verify that different images from the same camera give the same estimated NLF (camera not in the database for training)

Application (1): Adaptive Bilateral Filtering q Bilateral filter is an edge-preserving low-pass filter § Spatial sigma and range sigma Input noisy image q Smoothing kernel Denoised image From Durand Dorsey, SIGGRAPH 02 Adaptive bilateral filter § Down-weigh RGB values by signal and noise covariance matrices § The range sigma is set to be a function of the estimated standard deviation of the noise

Test on Low and High Noise low noise Red high noise Green Blue Red Green Blue

Results—Adaptive Bilateral Filtering Standard bilateral filtering Adaptive bilateral filtering low noise high noise

Results—Adaptive Bilateral Filtering Standard bilateral filtering Adaptive bilateral filtering Zoom in high noise

Application (2): Canny Edge Detection low noise Red high noise Green Blue Red Green Blue

Results—Canny Edge Detection Parameters adapted in MATLAB Parameters adapted by estimated noise low noise high noise

Conclusion q Piecewise-smooth image prior model to estimate the upper bound of noise level function (NLF) q Estimate the space of NLF by simulating CCD camera on the existing CRF database q Upper bounds are verified by both synthetic and real experiments q An important step to automate vision algorithms independent of noise

Thank you! Noise Estimation from a Single Image Ce Liu William T. Freeman CSAIL MIT Rick Szeliski Sing Bing Kang Microsoft Research