Combining Photometric and Geometric Constraints Yael Moses IDC

Combining Photometric and Geometric Constraints Yael Moses IDC, Herzliya Joint work with Ilan Shimshoni and Michael Lindenbaum, the Technion Y. Moses 1¨

Problem 1: Recover the 3 D shape of a general smooth surface from a set of calibrated images Y. Moses 2¨

Problem 2: Recover the 3 D shape of a smooth bilaterally symmetric object from a single image. Y. Moses 3¨

Shape Recovery ¨ Geometry: Stereo ¨ Photometry: ¨ Shape from shading ¨ Photometric stereo Main problems: Calibrations and Correspondence Y. Moses 4¨

3 D Shape Recovery Photometry: Geometry: ¨Stereo ¨ Shape from shading ¨ Photometric stereo ¨Structure from motion Y. Moses 5¨

Geometric Stereo ¨ 2 different images ¨ Known camera parameters ¨ Known correspondence + + Y. Moses 6¨

Photometric Stereo ¨ 3 D shape recovery: surface normals from two or more images taken from the same viewpoint Y. Moses 7¨

Photometric Stereo Three images Matrix notation Solution: Y. Moses 8¨

Photometric Stereo Main Limitation: ¨ 3 D shape recovery (surface normals) Two or more is images taken Correspondence obtained by a from fixed the viewpoint same viewpoint Y. Moses 9¨

Overview ¨ Combining photometric and geometric stereo: ¨ Symmetric surface, single image ¨ Non symmetric: 3 images ¨ Mono-Geometric stereo ¨ Mono-Photometric stereo ¨ Experimental results. Y. Moses 10¨

The input ¨ Smooth featureless surface ¨ Taken under different viewpoints ¨ Illuminated by different light sources The Problem: Recover the 3 D shape from a set of calibrated images Y. Moses 11¨

Assumptions n ¨ Three or more images n * * ¨ Given correspondence the normals can be computed (e. g. , Lambertian, distant point light source …) ¨ Perspective projection Y. Moses 12¨

Our method Combines photometric and geometric stereo We make use of: ¨ Given Correspondence: ¨Can compute a normal ¨Can compute the 3 D point Y. Moses 13¨

Basic Method Given Correspondence Y. Moses 14¨

First Order Surface Approximation Y. Moses 15¨

First Order Surface Approximation Y. Moses 16¨

First Order Surface Approximation P( ) = (1 - )O 1 + P , N (P( ) - P) = 0 Y. Moses 17¨

First Order Surface Approximation Y. Moses 18¨

New Correspondence Y. Moses 19¨

New Surface Approximation Y. Moses 20¨

Dense Correspondence Y. Moses 21¨

Basic Propagation Y. Moses 22¨

Basic Propagation Y. Moses 23¨

Basic method: First Order ¨ Given correspondence pi and L P and n ¨ Given P and n T ¨ Given P, T and Mi a new correspondence qi Y. Moses 24¨

Extensions ¨ Using more than three images ¨ Propagation: ¨ Using multi-neighbours ¨ Smart propagation ¨ Second error approximation ¨ Error correction: ¨ Based on local continuity ¨ Other assumptions on the surface Y. Moses 25¨

Multi-neighbors Propagation Y. Moses 26¨

Smart Propagation Y. Moses 27¨

Second Order: a Sphere (P-P( ))(N+N )=0 P( ) N P N+N N Y. Moses 28¨

Second Order Approximation Y. Moses 29¨

Second Order Approximation Y. Moses 30¨

Using more than three images ¨ Reduce noise of the photometric stereo ¨ Avoid shadowed pixels ¨ Detect “bad pixels” ¨ Noise ¨ Shadows ¨ Violation of assumptions on the surface Y. Moses 31¨

Smart Propagation Y. Moses 32¨

Error correction The compatibility of the local 3 D shape can be used to correct errors of: ¨ Correspondence ¨ Camera parameters ¨ Illumination parameters Y. Moses 33¨

Score ¨ Continuity: ¨Shape ¨Normals ¨Albedo ¨ The consistency of 3 D points locations and the computed normals: ¨General case: full triangulation ¨Local constraints Y. Moses 34¨

Extensions ¨ Using more than three images ¨ Propagation: ¨ Using multi-neighbours ¨ Smart propagation ¨ Second error approximation ¨ Error correction: ¨ Based on local continuity ¨ Other assumptions on the surface Y. Moses 35¨

Real Images ¨ Camera calibration ¨ Light calibration ¨ Direction ¨ Intensity ¨ Ambient Y. Moses 36¨

Error correction + multi-neighbor 5 Images Y. Moses 37¨

5 pp 5 nn 5 pn 3 pp 3 nn Y. Moses 38¨

Y. Moses 39¨

Y. Moses 40¨

Y. Moses 41¨

Y. Moses 42¨

Y. Moses 43¨

Detected Correspondence Y. Moses 44¨

Error correction + multi-neighbord Multi-neighbors Basic scheme (3 images) Error correction no multi-neighbors Y. Moses 45¨

Synthetic Images New Images Y. Moses 46¨

Ground truth Basic scheme Multi-neighbors Error correction Sec a Y. Moses 47¨

Ground truth Basic scheme Multi-neighbors Error correction Sec b Y. Moses 48¨

Ground truth Basic scheme Multi-neighbors Error correction Sec c Y. Moses 49¨

Ground truth Basic scheme Multi-neighbors Error correction Ground truth Basic scheme Multi-neighbors approx. Error correction Sec d Y. Moses 50¨

Combining Photometry and Geometry Yields a dense correspondence and dense shape recovery of the object in a single path Y. Moses 51¨

Assumptions ¨ Bilaterally Symmetric object ¨ Lambertian surface with constant albedo ¨ Orthographic projection ¨ Neither occlusions nor shadows ¨ Known “epipolar geometry” Y. Moses 52¨

Geometric Stereo ¨ ¨ 2 different images Known camera parameters Known viewpoints Known correspondence 3 D shape recovery Y. Moses 53¨

Computing the Depth from Disparity P Orthographic Projection Z qr Z pr pl ql Y. Moses 54¨

Symmetry and Geometric Stereo Non frontal view of a symmetric object Two different images of the same object Y. Moses 55¨

Symmetry and Geometric Stereo Non frontal view of a symmetric object Two different images of the same object Y. Moses 56¨

Geometry ¨ Weak perspective projection: Around X Around Z Around Y Y. Moses 57¨

Geometry ¨ Projection of Ry: is the only pose parameter Image point Object point Around Y Y. Moses 58¨

Correspondence Assume Yx. Z is the symmetry plane. image z x object Y. Moses 59¨

Mono-Geometric Stereo ¨ 3 D reconstruction: given correspondence and , image x object known unknown z Y. Moses 60¨

Viewpoint Invariant ¨ Given the correspondence and unknown Invariant Y. Moses 61¨

Photometric Stereo ¨ ¨ 2 images Lambertian reflectance Known illuminations Known correspondence 3 D shape recovery (same viewpoint) Y. Moses 62¨

Symmetry and Photometric Stereo Non-frontal illumination of a symmetric object Two different images of the same object Y. Moses 63¨

Notation: Photometry ¨ Corresponding object points: ¨ Illumination: Y. Moses 64¨

Mono-Photometric Stereo ¨ 3 D reconstruction given correspondence and E (up to a twofold ambiguity): known unknown Y. Moses 65¨

Invariance to Illumination ¨ Given correspondence and E unknown ¨ Invariant: Y. Moses 66¨

Mono-Photometric Stereo ¨ 3 D reconstruction E unknown but correspondence is given ¨ Frontal viewpoint with non-frontal illumination. ¨ Use image first derivatives. Y. Moses 67¨

Mono-Photometric Stereo Using image derivatives ¨ 3 global unknowns: E ¨ For each pair: ¨ 5 unknowns zx zy zxx zxy zyy ¨ 6 equations ¨ 3 pairs are sufficient Y. Moses 68¨

Mono-Photometric Stereo Unknown Illumination Y. Moses 69¨

Correspondence ¨ No correspondence => no stereo. ¨ Hard to define correspondence in images of smooth surfaces. ¨ Almost any correspondence is legal when: ¨ Only geometric constraints are considered. ¨ Only photometric constraints are considered. Y. Moses 70¨

Combining Photometry and Geometry ¨ Yields a dense correspondence (dense shape recovery of the object). ¨ Enables recovering of the global parameters. Y. Moses 71¨

Self-Correspondence ¨ A self-correspondence function: Y. Moses 72¨

Dense Correspondence using Propagation Assume correspondence between a pair of points, p 0 l and p 0 r. Y. Moses 73¨

Dense Correspondence using Propagation Y. Moses 74¨

image object x z Y. Moses 75¨

First derivatives of the Correspondence ¨ Assume known E Y. Moses 76¨

Computing and ¨ Object coordinates: Given computing and is trivial ¨ Moving from object to image coordinates depends on the viewing parameter Y. Moses 77¨

¨ Derivatives with respect to the object coordinates: ¨ Derivatives with respect to the image coordinates: Y. Moses 78¨

E image object x z Y. Moses 79¨

General Idea ¨ Given a corresponding pair and E n=(zx, zy, -1)T ¨ Given and n cx and c y ¨ Given cx and c y a new corresponding pair Y. Moses 80¨

Results on Real Images: Given global parameters Y. Moses 81¨

Finding Global Parameters ¨ Assume E and are unknown. ¨ Assume a pair of corresponding points is given. ¨ Two possibilities: ¨ Search for E and directly. ¨ Compute E and from the image second derivatives. Y. Moses 82¨

Integration Constraint: Circular Tour ¨ All roads lead to Rome … ¨ Find and verify correct correspondence ¨ Recover global parameters, E and Y. Moses 83¨

Finding Global Parameters Consider image second derivatives ¨ Due to foreshortening effect: and ¨ We can relate image and object derivatives by Y. Moses 84¨

Testing E and : Image second derivatives For each corresponding pair: and Plus 4 linear equations in 3 unknown. Where Y. Moses 85¨

Counting ¨ 5 unknowns for each pair: zx zy, zxx zxy zyy ¨ 4 global unknowns: E, ¨ For each pair: 6 equations. ¨ For n pairs: 5 n+4 unknowns 6 n equations. 4 pairs are sufficient Y. Moses 86¨

Results on Simulated Data Ground Truth Recovered Shape Y. Moses 87¨

Recovering the Global Parameters Y. Moses 88¨

Degenerate Case ¨ Close to frontal view: problems with geometric-stereo. reconstruction problem ¨ Close to frontal illumination: problems with photometric-stereo. correspondence problem Y. Moses 89¨

Future work ¨ Perspective photometric stereo ¨ Use as a first approximation to global optimization methods ¨ Test on other reflection models ¨ Recovering of the global parameters: ¨ Light ¨ Cameras ¨ Detect the first pair of correspondence Y. Moses 90¨

Future Work ¨ Extend to general 3 images under 3 viewpoints and 3 illuminations. ¨ Extend to non-lambertian surfaces. Y. Moses 91¨

Thanks Y. Moses 92¨

image object x z Y. Moses 93¨

Integration Constraint Y. Moses 94¨

Integration Constraint Y. Moses 95¨

Searching for E ¨ Illumination must satisfy: ¨ ¨ E is further constrained by the image second derivatives. Y. Moses 96¨

Image second derivatives: Where 4 linear equations in 3 unknown Y. Moses 97¨

Image second derivatives For each corresponding pair and E: 4 linear equations in 3 unknown. Where Y. Moses 98¨

Counting ¨ 5 unknowns for each pair: zx, zy, zxx, zxy, zyy ¨ 3 global unknowns: E ¨ For each pair: 6 equations. ¨ For n pairs: 5 n+3 unknowns 6 n equations. 3 pairs are sufficient Y. Moses 99¨

Correspondence Y. Moses 100¨

Variations ¨ Known/unknown distant light source ¨ Known/unknown viewpoint ¨ Symmetric/non-symmetric image ¨ Frontal/non-frontal viewpoint ¨ Frontal/non-frontal illumination Y. Moses 101¨

Correspondence ¨ Epipolar geometry is the only geometric constraint on the correspondence. ¨ Weak photometric constraint on the correspondence. Y. Moses 102¨

Lambertian Surface I= Basic radiometric n 2 n 1 * E E E * P Y. Moses 103¨

Photometric Stereo ¨ First proposed by Woodham, 1980. ¨ Assume that we have two images. . E Y. Moses 104¨