EE 7700 High Dynamic Range Imaging Bahadir K

EE 7700 High Dynamic Range Imaging Bahadir K. Gunturk

References n n Slides and papers by Debevec, Ward, Pattaniak, Nayar, Durand, et al… http: //people. csail. mit. edu/fredo/PUBLI/Siggraph 2002/ Bahadir K. Gunturk 2

High Dynamic Range (HDR) Imaging The range of luminances is more than 10^14 candela/m 2 n star light 10 -6 moon light 10 -2 office light day light 100 101 102 104 search light 108 Range of human eye at an instant is around 10^4: 1 (4 log units) Human eye can adapt to see much wider range. Candela (cd) is the unit of luminous intensity (power emitted by a light source in a particular direction, with wavelengths weighted by the sensitivity of the human eye. ) A common candle emits roughly 1 cd. A 100 W incandescent lightbulb emits about 120 cd. Bahadir K. Gunturk 3

Spectral Sensitivity of Human Visual System: Luminosity Function One candela is defined as the luminous intensity of a monochromatic 540 THz light source that has a radiant intensity of 1/683 watts per steradian, or about 1. 464 m. W/sr. The 540 THz frequency corresponds to a wavelength of about 555 nm, which is green light near the peak of the eye's response. A typical candle produces very roughly one candela of luminous intensity. Quantity Luminance Luminous flux Illuminance Derived SI Unit Symbol candela per square meter cd/m 2 lumen cd * sr = lm lux lm/m 2 = lx Photopic (black) and scotopic [1] (green) luminosity functions. The photopic includes the CIE 1931 standard [2] (solid), the Judd-Vos 1978 modified data [3] (dashed), and the Sharpe, Stockman, Jagla & Jägle 2005 data [4] (dotted). The horizontal axis is wavelength in nm. (from Wikipedia) Bahadir K. Gunturk 4

HDR n The range of radiances is more than 10^14 candela/m 2 star light 10 -6 moon light 10 -2 Range of Typical Displays: from ~1 to ~100 cd/m 2 Bahadir K. Gunturk office light day light 100 101 102 104 0 search light 108 255 5

Sensitivity of Eye Cone dominated Gain log Gain rod cone -6 -4 -2 Bahadir K. Gunturk 0 log La 2 4 6 1000 cd/m^2 6

Sensitivity of Eye Rod dominated Gain log Gain rod cone -6 -4 -2 Bahadir K. Gunturk 0 log La 2 4 6 0. 04 cd/m^2 7

Sensitivity of Eye Bahadir K. Gunturk 8

HDR n The range of image capture devices is also low Bahadir K. Gunturk 9

HDR n The range of image capture devices is also low Bahadir K. Gunturk 10

HDR n HDR image rendered to be displayed on a LDR display. Bahadir K. Gunturk 11

HDR Problems: • How to capture an HDR image with LDR cameras? • How to display an HDR image on LDR displays? Bahadir K. Gunturk 12

• Capture multiple images with varying exposure. • Combine them to produce an HDR image. Bahadir K. Gunturk 13

Creating HDR from Multiple Pictures Measured intensity, z t 1 t 2 Irradiance, E (=total power per unit area) t 1 Bahadir K. Gunturk t 2 14

Creating HDR from Multiple Pictures Measured intensity, z z 1 t 2 t 1 t 2 z 1 = t 1 * E z 2 = t 2 * E E Irradiance, E Estimates: E 1=z 1/t 1 E 2=z 2/t 2 Take a weighted sum of E 1 and E 2: w 1 w 2 E=( w 1*E 1 + w 2*E 2 ) / (w 1+w 2) E Bahadir K. Gunturk 15

Creating HDR from Multiple Pictures Measured intensity, z z 1 t 2 t 1 t 2 z 1 = t 1 * E z 2 = t 2 * E E Irradiance, E Estimates: E 1=z 1/t 1 Take a weighted sum of E 1 and E 2: w E=( w(z 1)*E 1 + w(z 2)*E 2 ) / (w(z 1)+w(z 2)) E 2=z 2/t 2 z Bahadir K. Gunturk 255 16

Creating HDR from Multiple Pictures Measured intensity, z z 1 t 2 t 1 t 2 z 1 = t 1 * E z 2 = t 2 * E E Irradiance, E Estimates: E 1=z 1/t 1 Take a weighted sum of E 1 and E 2: w E=( w(z 1)*E 1 + w(z 2)*E 2 ) / (w(z 1)+w(z 2)) E 2=z 2/t 2 z Bahadir K. Gunturk 255 Question: If t 1 and t 2 are not given, how can we estimate them? 17

Creating HDR from Multiple Pictures In general, the camera response is not linear. f z 1 = f ( t 1 * E ) z 2 = f ( t 2 * E ) t 1 t 2 g E 1= g (z 1) / t 1 E 2= g (z 2) / t 2 z w w is sometimes chosen as the derivative of f. (Mann) E=( w(z 1)*E 1 + w(z 2)*E 2 ) / (w(z 1)+w(z 2)) z Questions: How to estimate g and t? One approach is based on polynomial model (Nayar). Bahadir K. Gunturk 18

Radiometric Self Calibration Irradiance Intensity Polynomial model Exposure ratios: Pixel Image number Cost function Solve using If exposure ratios are not known, solve iteratively Bahadir K. Gunturk 19

Tone Mapping Given an HDR image, how are we going to display it in an LDR display? Bahadir K. Gunturk 20

Tone Mapping Given an HDR image, how are we going to display it in an LDR display? Linear Nonlinear Bahadir K. Gunturk 21

Durand & Gunturk Dorsey Bahadir K. 22

Durand & Gunturk Dorsey Bahadir K. 23

Durand & Gunturk Dorsey Bahadir K. 24

Durand & Gunturk Dorsey Bahadir K. 25

Durand & Gunturk Dorsey Bahadir K. 26

Durand & Dorsey Durand & Gunturk Dorsey Bahadir K. Bilateral filter 27

Durand & Gunturk Dorsey Bahadir K. 28

Fattal et al in 1 D log derivative attenuate 2500: 1 7. 5: 1 exp Bahadir K. Gunturk integrate 29

Reinhard et al. L_white is the smallest luminance that will be mapped to pure white (1). Set L_white = L_max to have no “burn-out”. Bahadir K. Gunturk 30

Durand & Gunturk Dorsey Bahadir K. 31

Durand & Gunturk Dorsey Bahadir K. 32

Informal comparison Gradientdomain [Fattal et al. ] Bahadir K. Gunturket al. ] [Fattal Bilateral [Durand et al. ] Photographic [Reinhard et al. ] 33

Spatially Varying Exposures n n n Instead of capturing multiple pictures, allow different amounts of light pass for different pixel positions. Estimate the missing pixels. Combine to obtain an HDR image. 100% 75% 50% 25% Nayar Bahadir K. Gunturk 34

Image Reconstruction: Interpolation Bahadir K. Gunturk 35

Image Reconstruction: Aggregation Bahadir K. Gunturk 36

HDR image examples Bahadir K. Gunturk 37

HDR image examples Bahadir K. Gunturk 38

HDR image examples Bahadir K. Gunturk 39

The Bilateral Filter (BF) n n n The SUSAN filter, which is essentially the bilateral filter, was used for corner/edge detection and denoising in [Smith & Brady 97]. The BF was presented in [Tomasi & Manduchi 98]. [Elad 02] and [Barash 02] show that the BF is related to the weighted least squares estimation and anisotropic diffusion. Fast implementations/approximations have been proposed, e. g. , in [Paris & Durand 06]. In addition to image denoising, the BF is used in tone mapping of HDR images, contrast enhancement, 3 D mesh smoothing, blocking artifact reduction, etc. Bahadir K. Gunturk 40

Bilateral Filtering Intensity (range) proximity Bahadir K. Gunturk Spatial (domain) proximity 41

Bilateral Filtering Input Gaussian Bilateral Bahadir K. Gunturk 42

What are the optimal values of the parameters of the Bilateral Filter? MSE=49. 8 MSE=50. 9 MSE=30. 3 MSE=43. 4 MSE=42. 5 MSE=71. 5 MSE=100. 0 Bahadir K. Gunturk 43