David Fofi 1 Joaquim Salvi 2 El Mustapha

David Fofi 1 Joaquim Salvi 2 El Mustapha Mouaddib 3 1 Le 2 i UMR CNRS 5158 Université de Bourgogne Le Creusot, France d. fofi@iutlecreusot. u-bourgogne. fr 2 VICOROB - IIi. A Universitat de Girona, España qsalvi@eia. udg. es 3 CREA EA 3299 Université de Picardie Jules Verne Amiens, France mouaddib@u-picardie. fr « Uncalibrated Vision based on Structured Light »

0. Outline ……………… 1. Introduction 2. Tools for uncalibrated vision 3. Uncalibrated reconstruction 4. Experimental results 5. Conclusion

I. Introduction ……………… 1. Structured light vision 2. Calibration vs uncalibration

« Structured light vision » …………. . . ……………… IMAGE PLANE PATTERN FRAME ………………. . J. Salvi, J. Batlle, E. Mouaddib, "A robust-coded pattern projection for dynamic measurement of moving scenes", Pattern Recognition Letters, 19, pp. 1055 -1065, 1998. J. Batlle, E. Mouaddib, J. Salvi, "Recent progress in coded structured light to solve the correspondence problem. A survey", Pattern Recognition, 31(7), pp. 963 -982, 1998.

« Calibration vs uncalibration » …………. . . ……………… DRAWBACKS OF HARD-CALIBRATION: • Off-line process (calibration pattern, etc. ) • Has to be repeated each time one of the parameters is modified Working with a camera with automatic focus and aperture is NOT possible. Visual adaptation to the environment is not allowed! A slide or LCD projector needs to be focused. RECONSTRUCTION FROM UNCALIBRATED SENSOR. . .

II. Tools for uncalibrated vision ……………… 1. Test of spatial colinearity 2. Test of coplanarity 3. Stability of the cross-ratio 4. Validity of the affine model

« Test of spatial colinearity » …………. . . ……………… P Q R S Cross-ratio within the pattern and cross-ratio within the image are equals if the points are colinear.

« Test of coplanarity » …………. . . ……………… o' p' s' o p s r' q' r q {o; p, q, r, s}={o'; p', q', r'} Cross-ratio within the pattern and cross-ratio within the image are equals if the point are colinear.

« Stability of the cross-ratio » …………. . . ……………… Error on cross-ratios with a noise from 0 to 0. 5 d (d is the distance between two successive points) Nota: to compare cross-ratios a projective distance is necessary. Method of the random cross-ratios. ………………. . K. Aström, L. Morin, "Random cross-ratios", Research Report n°rt 88 imag-14, LIFIA, 1992.

« Validity of the affine model » …………. . . ……………… n m affine projection m' n' Valid if 0

III. Uncalibrated reconstruction ……………… 1. Projective reconstruction 2. Structured light limitations 3. Euclidean constraints through structured lighting

« Projective reconstruction » …………. . . ……………… Recover the scene structure from n images and m points and. . . Intrinsic parameters Extrinsic parameters Scene geometry Points matching PROJECTIVE RECONSTRUCTION

« Structured light limitations » …………. . . ……………… MOVEMENT OF THE PROJECTOR CAMERA + PROJECTOR THE PATTERN SLIDES ALONG THE OBJECTS MOVEMENT OF THE 3 -D POINTS RECONSTRUCTION FROM TWO VIEWS (i. e. one view and one pattern projection) + HETEROGENEITY OF THE SENSOR INTRINSIC PARAMETERS CANNOT BE ASSUMED CONSTANT = PARAMETERS ESTIMATION APPROACH, CANONICAL REPRESENTATION ………………. . R. Mohr, B. Boufama, P. Brand, “Accurate projective reconstruction”, Proc. of the 2 nd ESPRIT-ARPA-NSF Workshop on Invariance, Azores, pp. 257 -276, 1993. Q. -T. Luong, T. Viéville, "Canonical representations for the geometries of multiple projective views", Proc. of the 3 rd Euro. Conf. on Computer Vision, Stockholm (Sweden), 1994

« The parameters estimation approach » …………. . . ……………… n images composed by m points. . . pij : image point Aj : projection matrix Pj : object point (Uij, Vij) : pixel co-ordinates

« The parameters estimation approach » …………. . . ……………… A unique solution cannot be performed because. . . W is a 4 x 4 invertible matrix… a collineation of the 3 -D space 4 x 4 - 1 (scale factor) = 15 degrees of freedom, thus. . . 5 co-ordinates object points assigned to AN ARBITRARY PROJECTIVE BASIS. A RECONSTRUCTION WITH RESPECT TO A PROJECTIVE FRAME (distances, angles, parallelism are not preserved)

« From projective to Euclidean » …………. . . ……………… Euclidean transformations form a sub-group of projective transformations. . . A collineation W upgrades projective reconstruction to Euclidean one. TRANSLATING EUCLIDEAN KNOWLEDGE OF THE SCENE INTO MATHEMATICAL CONSTRAINTS ON THE ENTRIES OF W. Matching projective points with their corresponding Euclidean points ? YES, BUT. . . Euclidean co-ordinates of points are barely available… … if they are: pattern cross-points have to be projected exactly onto these object points. ………………. . B. Boufama, R. Mohr, F. Veillon, "Euclidean constraints for uncalibrated reconstruction", Proc. of the 4 th Int. Conf. on Computer Vision, Berlin (Germany), pp. 466 -470, 1993.

« Parallelogram constraints » …………. . . ……………… The sensor behaviour is assumed to be affine. . . A C PROJECTED SQUARE B D ONTO A PLANAR SURFACE IMAGE CAPTURE

« Alignment constraints » …………. . . ……………… Vert. plane Pattern Horiz. plane • Points belonging to horizontal or vertical plane. . . • Arbitrary distance between two planes. . . • Cross-point as origin…

« Orthogonality constraints » …………. . . ……………… C' Light stripes Light planes A' B' Planar surfaces C A B Projected lines A'B' ·A'C' = (x. A' - x. B')(x. A' - x. C')+ (y. A' - y. B')(y. A' - y. C')+ (z. A' - z. B')(z. A' -z. C') = 0 otherwise… reduced orthogonality constraint: (x. A' - x. B')(x. A' - x. C')+ (y. A' - y. B')(y. A' - y. C') = 0

« Example » …………. . . ……………… An alignment constraint : x. A' points) We have: Then: = x. B' (relation between unknown Euclidean [x. A' ; y. A' ; z. A' ; t. A']T = W· [x. A ; y. A ; z. A ; t. A]T [x. B' ; y. B' ; z. B' ; t. B']T = W· [x. B ; y. B ; z. B ; t. B]T W 1 i·x. A = W 1 i·x. B (relation between known projective points) … same way for the other constraints… The set of equations is solved by a non-linear optimisation method as Levenberg-Marquardt. 15 independent constraints are necessary (W is a 4 4 matrix defined up to a scale factor)

IV. Experimental results ……………… 1. Colinearity 2. Coplanarity 3. Euclidean reconstruction

« Colinearity » …………. . . ……………… Theoretical (pattern) cross-ratio = 1. 3333 Measured (image) cross-ratio = 1. 3287 Projective error =6. 9 10 -4 Decision = the points are colinear Theoretical cross-ratio = 1. 3333 Measured cross-ratio = 1. 3782 Projective error =6. 2 10 -3 Decision = the points are not colinear

« Coplanarity » …………. . . ……………… Theoretical cross-ratio = 2 Measured cross-ratio = 1. 96 Projective error =2. 2 10 -3 Decision = the points are coplanar Theoretical cross-ratio = 2 Measured cross-ratio = 2. 186 Projective error =5. 9 10 -3 Decision = the points are not coplanar

« Euclidean reconstruction: synthetic data » …………. . . ……………… re-projection of 3 D points onto the image planes (circles: synthetic points, crosses: re-projections)

« Euclidean reconstructions » …………. . . ……………… -20 -40 y -60 -80 -100 -120 -140 -180 -160 -140 -120 x -100 -80 -60

V. Conclusion ………………

• Projective reconstruction from a single pattern projection and a single image capture. • Pattern projection used to retrieve geometrical knowledge of the scene: uncalibrated Euclidean reconstruction. • Structured lighting ensures there is known scene structure. • Structured light provides numerous contraints. • Tests of colinearity and coplanarity can be used to retrieve projective basis (5 points, no 4 of them being coplanar, no 3 of them being colinear).