Active Lighting for Appearance Decomposition Todd Zickler DEAS

Active Lighting for Appearance Decomposition Todd Zickler DEAS, Harvard University Appearance Decomposition

Appearance I = f (shape, illumination, reflectance) f -1( I ) = Appearance Decomposition ?

Research Overview COLOR IMAGE FILTERING 3 D RECONSTRUCTION APPEARANCE CAPTURE PHOTOMETRIC INVARIANTS Appearance Decomposition

Getting 3 D Shape: Image-based Reconstruction f I = f (shape, reflectance, illumination) Appearance Decomposition -1( ? I)=

Reflectance: BRDF Bi-directional Reflectance Distribution Function Appearance Decomposition

Conventional 3 D Reconstruction: Restrictive Assumptions LAMBERTIAN: IDEALLY DIFFUSE Appearance Decomposition

Example: Conventional Stereo Il Ir ASSUMPTION: Il = Ir Appearance Decomposition

Example: Conventional Stereo Il Ir ASSUMPTION: Il = Ir Appearance Decomposition

Conventional 3 D Reconstruction: Restrictive Assumptions Shape from shading Variational Stereo [Tsai and Shaw, 1994] [Faugeras and Keriven, 1998] Multiple-window stereo Space Carving [Fusiello et al. , 1997] Appearance Decomposition [Kutulakos and Seitz, 1998]

Reflectance: BRDF Appearance Decomposition

Reflectance: BRDF Appearance Decomposition

Helmholtz Reciprocity [Helmholtz 1925; Minnaert 1941; Nicodemus et al. 1977] Appearance Decomposition

Stereo vs. Helmholtz Stereo STEREO Appearance Decomposition HELMHOLTZ STEREO

Stereo vs. Helmholtz Stereo STEREO Appearance Decomposition HELMHOLTZ STEREO

Stereo vs. Helmholtz Stereo STEREO Appearance Decomposition HELMHOLTZ STEREO

Reciprocal Images Il Ir w Specularities “fixed” to surface w Relation between Il and Ir independent of BRDF Appearance Decomposition

Reciprocity Constraint p v^ l ol Appearance Decomposition p n^ v^ r v^ l or ol = n^ v^ r or

Reciprocity Constraint p v^ l ol Appearance Decomposition p n^ v^ r v^ l or ol ¨ Arbitrary reflectance ¨ Surface normal = n^ v^ r or

Reciprocal Acquisition CAMERA LIGHT SOURCE Appearance Decomposition

Recovered Normals [Zickler et al. 2002] Appearance Decomposition

Recovered Surface [Zickler et al. , ECCV 2002] Appearance Decomposition

In Practice 1. Arbitrary Reflectance 2. Off-the-shelf components 3. Direct surface normals 4. Images aligned with recovered shape 5. Self-calibrating (coming…) Appearance Decomposition

Ongoing Work: Auto-calibration [Zickler et al. , CVPR 2003, CVPR 2006, …] Appearance Decomposition

Research Overview COLOR IMAGE FILTERING 3 D RECONSTRUCTION APPEARANCE CAPTURE PHOTOMETRIC INVARIANTS Appearance Decomposition

Reflectance Decomposition DIFFUSE = SPECULAR + [Phong 1975; Shafer, 1985] Appearance Decomposition

Reflectance Decomposition [Shafer, 1985] Appearance Decomposition

Reflectance Decomposition: Simplifies the Vision Problem Appearance Decomposition = + LAMBERTIAN: IDEALLY DIFFUSE

Reflectance Decomposition: A Difficult Inverse Problem DIFFUSE SPECULAR = + [Bajscy et al. , 1996; Criminisi et al. , 2005; Lee and Bajscy, 1992; Lin et al. , 2002; Lin and Shum, 2001; Miyazaki et al. , 2003; Nayar et al. , 1997; Ragheb and Hancock, 2001; Sato and Ikeutchi, 1994; Tan and Ikeutchi, 2005; Wolfe and Boult, 1991, …] Appearance Decomposition

Known Illuminant: Still Ill-posed B S IRGB D? R Appearance Decomposition G

Known Illuminant: Still Ill-posed B S IRGB D? G R Appearance Decomposition

Observation: Explicit Decomposition not Required B S IRGB r 1 R Appearance Decomposition r 2 J G 1. INVARIANT TO SPECULAR REFLECTIONS 2. BEHAVES ‘LAMBERTIAN’

Observation: Explicit Decomposition not Required B S IRGB r r 1 J 2 R G IRGB || J || [Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005] Appearance Decomposition

Generalization: Mixed Illumination SINGLE ILLUMINANT MIXED ILLUMINATION B B S 1 S S 2 IRGB r 1 R r 2 J G J IRGB r 1 G R [Zickler, Mallick, Kriegman, Belhumeur, CVPR 2006] Appearance Decomposition

Generalization: Mixed Illumination Appearance Decomposition

Example: Binocular Stereo Conventional Grayscale (R+G+B)/3 Specular Invariant, ||J|| (blue illuminant) (blue & yellow illuminants) Recovered depth One image from input stereo pair [Algorithm: Boykov, Veksler and Zabih, CVPR 1998] Appearance Decomposition

(blue & yellow illuminants) Specular Invariant, ||J|| (blue illuminant) Conventional Grayscale (R-+G+B)/3 Example: Optical Flow Appearance Decomposition Ground truth flow [Algorithm: Black and Anandan, 1993]

Example: Photometric Stereo J behaves ‘Lambertian’ Linear function of surface normal [Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005] Appearance Decomposition

Example: Photometric Stereo J behaves ‘Lambertian’ Linear function of surface normal [Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005] Appearance Decomposition

Example: Photometric Stereo J behaves ‘Lambertian’ Linear function of surface normal [Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005] Appearance Decomposition

Example: Photometric Stereo [Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005] Appearance Decomposition

Example: Photometric Stereo [Mallick, Zickler, Kriegman, Belhumeur, CVPR 2005] Appearance Decomposition

Generalized Hue B S IRGB r 1 R Appearance Decomposition ψ r 2 J G

Example: Material-based Segmentation Conventional Grayscale Specular Invariant ||J|| Input image Conventional Hue Generalized Hue y [Zickler, Mallick, Kriegman, Belhumeur, CVPR 2006] Appearance Decomposition

Active Lighting for Image-guided Surgery? Active lighting can provide: 1. Precise shape (surface normals) for a broad class of (non-Lambertian) surfaces 2. Specular and/or shading invariance (e. g. , optical flow, tracking, segmentation) 3. Minimal hardware requirements Endoscopic imagery: 1. Illuminant(s) is/are controlled and known 2. Non-Lambertian surfaces 3. Lack of texture Appearance Decomposition

Acknowledgements Satya Mallick, UCSD Peter Belhumeur, Columbia University David Kriegman, UCSD Sebastian Enrique, Columbia University Ravi Ramamoorthi, Columbia University zickler@eecs. harvard. edu http: //www. eecs. harvard. edu/~zickler Appearance Decomposition