Digital Photography with Flash and NoFlash Image Pairs

Digital Photography with Flash and No-Flash Image Pairs By: Georg Petschnigg Maneesh Agrawala Hugues Hoppe Richard Szeliski Michael Cohen Kentaro Toyama, Microsoft Corporation Presented by: Yael Amsterdamer Advanced Topics in Visual Computing, Spring 2012

Introduction The Dilemma: to Flash or not to flash? • Natural lighting • Low signal-to-noise ratio (SNR) • Loss of details • Longer exposure – motion blur Digital Photography with Flash and No-Flash Image Pairs • Harsh, unnatural lighting • High SNR • More details • May cause unwanted artifacts (red eye, shadows, specularities) 2

Introduction Why not both? • The idea: use the good features of each photo to create a better image Digital Photography with Flash and No-Flash Image Pairs 3

Introduction Other works on combining images Different exposures Panoramic view Current work • handles 2 images with specific characteristics, • Envisioned for camera firmware Relighting Digital Photography with Flash and No-Flash Image Pairs 4

Introduction Different Applications • Denoising • Detail transfer – Side requirement: handling flash induced shadows and specularities • White Balancing • Continuous Flash Adjustment • Red Eye Removal Digital Photography with Flash and No-Flash Image Pairs 5

Background Camera Adjustment to light • A certain amount of light is needed for a photo • Taking pictures of a dark scene – Increasing exposure time – Opening the aperture – Increasing ISO (sensor sensitivity) • The problem – high ISO noise affects high-frequency details Digital Photography with Flash and No-Flash Image Pairs 6

Background The Input • F – the photo with flash • A – the no-flash (ambient) photo – Composed of 3 RGB intensity values for each pixel – Ap – one pixel in A • Same camera position, focus, aperture – Otherwise, registration is required • Adjusted ISO and exposure time • Undergo non-linear tone-mapping operations Digital Photography with Flash and No-Flash Image Pairs 7

Denoising 1 st Application: Denoising • A classic problem in image processing – The target: analytically reduce SNR • One type of solution: smoothing filters • Bilateral Filter [Tomasi and Manduchi 1998] – Fast, non-iterative – Averaging close-by pixels – Low-pass filter, combined with – Edge-stopping function Digital Photography with Flash and No-Flash Image Pairs 8

Denoising Bilateral filter • However, results still have noise or blur (or both) ambient flash Digital Photography with Flash and No-Flash Image Pairs Bilateral filter 9

Denoising Improvement: Joint Bilateral Filter • In the flash image there are much more details • Why not use F to find edges? The difference Bilateral filter Digital Photography with Flash and No-Flash Image Pairs Joint Bilateral filter 10

Denoising Not all edges are real … • May cause over- or under-blur in joint bilateral filter • We need to eliminate their effect Digital Photography with Flash and No-Flash Image Pairs 11

Denoising Detecting shadows • Observation: the pixels in the flash shadow should be similar to the ambient image • Not identical: – Noise – Inter-reflected flash Pixels for which we will not use F • Compute a shadow mask • Take pixel p if • is manually adjusted • Mask is smoothed and dilated Digital Photography with Flash and No-Flash Image Pairs 12

Denoising Detecting specularities • Take pixels where sensor input is close to maximum (very bright) – Over fixed threshold • Create a specularity mask • Also smoothed and dilated • M – the combination of shadow and specularity masks – Where Mp=1, we use ABase. For other pixels we use ANR. Digital Photography with Flash and No-Flash Image Pairs 13

Detail Transfer There is so much denoising can do • It cannot add details missing in the ambient image • Exist in flash image because of high SNR • We use a quotient image: Reduces the effect of noise in F ANR Bilateral add the details filtered • Multiply with to • Masked in the same way Digital Photography with Flash and No-Flash Image Pairs 14

Detail Transfer Unified process • Denoising + detail transfer + masking shadows and specularities Digital Photography with Flash and No-Flash Image Pairs 15

Detail Transfer More examples Digital Photography with Flash and No-Flash Image Pairs 16

White Balancing The opposite of preserving the atmosphere • The goal: simulate white light in the scene – What color is the light in the scene? • Flash image can help us: The difference between F and A is only the flash light • Assume it is proportional to the surface albedo – Inherent property of the surface – Assume the surface reflects light in the same color (will not work for plastic, glass…) Digital Photography with Flash and No-Flash Image Pairs 17

White Balancing Computing the ambient light • We cancel the color of the surface • The average c of Cp(col) for pixels with certain properties is the ambient light color • Then we scale A to cancel the ambient light Digital Photography with Flash and No-Flash Image Pairs 18

White Balancing Example Estimated ambient light Ambient original Computed color Whitebalanced Digital Photography with Flash and No-Flash Image Pairs 19

Summary Combining flash and no-flash images • Providing complementary details about the scene • Use for denoising and detail transfer • Use for white balancing • Future work – More robustness – Automatization of configuration Digital Photography with Flash and No-Flash Image Pairs 20

Thank you!