Computer Vision Range data Marc Pollefeys COMP 256

Computer Vision Range data Marc Pollefeys COMP 256 Some slides and illustrations from J. Ponce, …

Tentative class schedule Computer Vision Aug 26/28 - Introduction Cameras Radiometry Sources & Shadows Color Sep 16/18 Linear filters & edges (hurricane Isabel) Sep 23/25 Pyramids & Texture Multi-View Geometry Stereo Project proposals Tracking (Welch) Optical flow Oct 14/16 - - Oct 21/23 Silhouettes/carving (Fall break) Oct 28/30 - Structure from motion Project update Proj. Sf. M Nov 11/13 Camera calibration Segmentation Nov 18/20 Fitting Prob. segm. &fit. Nov 25/27 Matching templates (Thanksgiving) Matching relations Range data Sep 2/4 Sep 9/11 Sep 30/Oct 2 Oct 7/9 Nov 4/6 Dec 2/4 Dec 9 Final project

Computer Vision Final project • Presentation: Tuesday 2 -5 pm, starts in SN 011, then demos (make your own arrangements, preferably G-lab) • Papers: due Friday (Saturday is ok, but not later!)

Computer Vision RANGE DATA • Active Range Sensors • Segmentation • Elements of Analytical Differential Geometry • Registration and Model Acquisition • Quaternions • Object Recognition Reading: Chapter 21.

Computer Vision Active Range Sensors • Triangulation-based sensors • Time-of-flight sensors • New Technologies Courtesy of D. Huber and M. Hebert.

Computer Vision Structured light • Single grid projection • Binary code • A desktop scanner

Computer Vision Principle deformation+connectivity of pattern 3 D Shape Proesmans and Van Gool, ICPR 96…

Computer Vision Acquisition setup Proesmans and Van Gool, ICPR 96…

Computer Vision Calibration Proesmans and Van Gool, ICPR 96…

Computer Vision Image with projected grid Proesmans and Van Gool, ICPR 96…

Computer Vision Line detectors Proesmans and Van Gool, ICPR 96…

Computer Vision Linking Proesmans and Van Gool, ICPR 96…

Computer Vision Initial grid Proesmans and Van Gool, ICPR 96…

Computer Vision Corrected grid sub-pixel refinement of grid Proesmans and Van Gool, ICPR 96…

Computer Vision Depth computation Proesmans and Van Gool, ICPR 96…

Computer Vision Removing lines Proesmans and Van Gool, ICPR 96…

Computer Vision Texture estimation Proesmans and Van Gool, ICPR 96…

Computer Vision 3 D Reconstruction Proesmans and Van Gool, ICPR 96…

Computer Vision Dionysos Proesmans and Van Gool, ICPR 96…

Computer Vision Theatre mask Proesmans and Van Gool, ICPR 96…

Computer Vision Capital Proesmans and Van Gool, ICPR 96…

Computer Vision Coded planes • Structured light: – Use projector as a camera – Figure out correspondences by coding light pattern – Only need to code 1 D (but not parallel with epipolar lines!) B. Curless

Computer Vision A desktop scanner

Computer Vision

Computer Vision

Computer Vision

Computer Vision

Computer Vision

Computer Vision

Computer Vision

Computer Vision

Computer Vision

Computer Vision More range sensors Delta. Sphere Z-cam

Computer Vision Real-time system Koninckx and Van Gool

Computer Vision Elements of Analytical Differential Geometry • Parametric surface: x : U R 2 E 3 • Unit surface normal: N= 1 (xu x xv) | xu x xv | • First fundamental form: I( t, t ) = Eu’ 2 + 2 Fu’v’+Gv’ 2 • Second fundamental form: II( t, t ) = eu’ 2 + 2 fu’v’+gv’ 2 { E=xu. xu F=xu. xv G=xv. xv { e= – N. xuu f = – N. xuv g= – N. xvv • Normal and Gaussian curvatures: II( t, t ) eg – f 2 t = K= I( t, t ) EG – F 2

Computer Vision Example: Monge Patches h x ( u, v ) = (u, v, h( u, v )) v In this case 1 T • N= ( –h , 1) u v (1+hu 2+hv 2)1/2 • E= 1+hu 2; F = huhv, ; G = 1+hv 2 u –huv –hvv • e= ; f= ; g= 2 2 1/2 (1+hu +hv ) (1+hu 2+hv 2)1/2 And the Gaussian curvature is: huuhvv–huv 2 K= (1+hu 2+hv 2)2 .

Computer Vision Example: Local Surface Parameterization N • u, v axes = principal directions • h axis = surface normal In this case: u v • h(0, 0)=hu(0, 0)=hv(0, 0)=0 • N=(0, 0, 1)T • huv(0, 0)=0, 1= – huu(0, 0), 2= – hvv(0, 0) Taylor expansion of order 2 h(u, v) ¼ – ½ ( 1 u 2 + 2 v 2)

Computer Vision Finding Step and Roof Edges in Range Images

Computer Vision Step Model { z= k 1 x+h when x<0, k 2 x+c+h when x>0. And, since z ’’=0 in x :

Computer Vision Roof Model { z= k 1 x+h when x<0, k 2 x+h when x>0. And has a maximum value inversely proportional to in a point x located at a distance proportional to from the origin.

Computer Vision Computing the Principal Directions and Curvatures • Adaptive smoothing • Finite-difference masks Reprinted from “Describing Surfaces, ” by J. M. Brady, J. Ponce, A. Yuille and H. Asada, Proc. International Symposium on Robotics Research, H. Hanafusa and H. Inoue (eds. ), MIT Press (1985). 1985 MIT.

Computer Vision Scale-Space Matching Reprinted from “Toward a Surface Primal Sketch, ” By J. Ponce and J. M. Brady, in Three-Dimensional Machine Vision, T. Kanade (ed. ), Kluwer Academic Publishers (1987). 1987 Kluwer Academic Publishers.

Computer Vision Segmentation into Planes via Region Growing (Faugeras & Hebert, 1986) Idea: Iteratively merge the pair of planar regions minimizing the average distance to the plane best fitting them. Reprinted from “The Representation, Recognition and Locating of 3 D Objects, ” by O. D. Faugeras and M. Hebert, the International Journal of Robotics Research, 5(3): 27 -52 (1986). 1986 Sage Publications. Reprinted by permission of Sage Publications.

Computer Vision Quaternions q = a+ q is a quaternion, a 2 R is its real part, and 2 R 3 is its imaginary part. Operations on quaternions: • Sum of quaternions: ( a+ ) + ( b+ ) ´ ( a+b ) + ( + ) ( a+ ) ´ ( a+ ) • Multiplication by a scalar: • Quaternion product: ( a+ ) ( b+ ) ´ ( a b – ¢ ) + ( a + b + £ ) • Conjugate: • Norm: q=a+ ) q´a– | q |2 ´ q q = a 2 + | |2 Note: | qq’|= |q| |q’|

Computer Vision Quaternions and Rotations • Let R denote the rotation of angle about the unit vector u. • Define q = cos /2 + sin /2 u. • Then for any vector , R = q q. Reciprocally, if q = a + ( b, c, d )T is a unit quaternion, the corresponding rotation matrix is:

Computer Vision The Iterative Closest Point Registration Algorithm (Besl and Mc. Kay, 1992) Key points: • finding the closest-point pairs (k-d trees, caching); • estimating the rigid transformation (quaternions).

Computer Vision Using Quaternions to Estimate a Rigid Transformation Problem: Find the rotation matrix R and the vector t that minimize n E = | xi’ – R xi – t |2. i=1 n At a minimum: 0 = E/ t = – 2 ( xi’ – R xi – t ). Or. . t = x’ – R x. i=1 If yi = xi –x and yi’ = xi’ –x’, the error is (at a minimum): n E = | y i’ – R y i i=1 Or. . |2 E = A i. T A i n = | y i’ – q y i i=1 with Ai = Linear least squares !! q|2|q|2 n = | y i ’ q – q y i |2 i=1 0 yi. T-yi’T yi’-yi [yi+yi’]£

Computer Vision ICP Registration Results Reprinted from “A Method for Registration of 3 D Shapes, ” by P. J. Besl and N. D. Mc. Kay, IEEE Trans. on Pattern Analysis and Machine Intelligence, 14(2): 238 -256 (1992). 1992 IEEE.

Computer Vision Initial alignment? • Mostly open problem • A possible approach using bitangents (Vanden Wyngaerd and Van Gool)

Computer Vision

Computer Vision

Computer Vision

Computer Vision Fusing Range Images (Curless & Levoy, 1996) Idea: Construct watertight surfaces as level sets of appropriate volumetric density functions. Reprinted from “A Volumetric Method for Building Complex Models from Range Images, ” by B. Curless and M. Levoy, Proc. SIGGRAPH (1996). 1996 ACM, Included here by permission. Courtesy of M. Levoy.

Computer Vision Fusing Range Images (Curless & Levoy, 1996) Idea: Construct watertight surfaces as level sets of appropriate volumetric density functions. Reprinted from “A Volumetric Method for Building Complex Models from Range Images, ” by B. Curless and M. Levoy, Proc. SIGGRAPH (1996). 1996 ACM, Included here by permission. Courtesy of M. Levoy.

Computer Vision Volumetric integration (Curless and Levoy, Siggraph´ 96) range surfaces volume signed distance to surface weight (~accuracy) distance depth sensor surface 1 • use voxel space • new surface as zero-crossing (find using marching cubes) • least-squares estimate (zero derivative=minimum) surface 2 combined estimate

Computer Vision Fusing Range Images (Curless & Levoy, 1996) Idea: Construct watertight surfaces as level sets of appropriate volumetric density functions. Reprinted from “A Volumetric Method for Building Complex Models from Range Images, ” by B. Curless and M. Levoy, Proc. SIGGRAPH (1996). 1996 ACM, Included here by permission. Courtesy of M. Levoy.

Computer Vision From volume to mesh: Marching Cubes • First 2 D, Marching Squares “Marching Cubes: A High Resolution 3 D Surface Construction Algorithm”, William E. Lorensen and Harvey E. Cline, Computer Graphics (Proceedings of SIGGRAPH '87), Vol. 21, No. 4, pp. 163 -169.

Computer Vision From volume to mesh: Marching Cubes “Marching Cubes: A High Resolution 3 D Surface Construction Algorithm”, William E. Lorensen and Harvey E. Cline, Computer Graphics (Proceedings of SIGGRAPH '87), Vol. 21, No. 4, pp. 163 -169.

Computer Vision From volume to mesh: Marching Cubes • Improvement “Marching Cubes: A High Resolution 3 D Surface Construction Algorithm”, William E. Lorensen and Harvey E. Cline, Computer Graphics (Proceedings of SIGGRAPH '87), Vol. 21, No. 4, pp. 163 -169.

Computer Vision The Faugeras-Hebert Plane Matching Algorithm (1986) Key points: • finding initial matches (area comparisons, binning); • estimating the rigid transformation (quaternions).

Computer Vision Finding all the vectors v making an angle between - And + with a vector u.

Computer Vision Using Quaternions to Estimate a Rigid Transformation : n ¢ x – d = 0 ! ’: n’ ¢ x’ – d’ = 0 where n’ = R n and d’ = n’ ¢ t + d. Problem: Find the rotation matrix R and the vector t that minimize n E r = | ni ’ – R ni |2 i=1 and n Et = | di’ – di – ni’¢ t |2. i=1 Linear least squares !!

Computer Vision Recognition Results (Faugeras & Hebert, 1986) Reprinted from “The Representation, Recognition and Locating of 3 D Objects, ” by O. D. Faugeras and M. Hebert, the International Journal of Robotics Research, 5(3): 27 -52 (1986). 1986 Sage Publications. Reprinted by permission of Sage Publications.

Computer Vision Spin Images (Johnson & Hebert, 1998) SP(Q)=(|PQ£ n|, PQ¢ n)

Computer Vision Sample Spin Images Reprinted from “Using Spin Images for Efficient Object Recognition from Cluttered 3 D Scenes, ” by A. E. Johnson and M. Hebert, IEEE Trans. on Pattern Analysis and Machine Intelligence, 21(5): 433 -449 (1999). 1999 IEEE. Matching Criterion:

Computer Vision Recognition Results Reprinted from “Using Spin Images for Efficient Object Recognition from Cluttered 3 D Scenes, ” by A. E. Johnson and M. Hebert, IEEE Trans. on Pattern Analysis and Machine Intelligence, 21(5): 433 -449 (1999). 1999 IEEE.

Computer Vision • What next? • Related courses: – Comp 254 Image Analysis (Spring) – Comp 255 Recent Advances in Image Analysis (Odd Falls) – Comp 290 3 D photography?

Computer Vision The future is bright • Computation is cheap • Lots of pix – cameras are cheap, many pix are digital • Lots of demand for “slicing and dicing” pix – generate models – new movies from old – search • Lots of “hidden value” – can’t do data mining for collections with pix in them • e. g. mortgage papers, cheques, etc. • e. g. filtering

Computer Vision There are lots of cameras! surveillance cameras ~1500/sq. mile in Manhattan

Computer Vision Recent flowering of vision • can do (sort of!) – structure from motion – segmentation – video representation – model building – tracking – face finding • will be able to do (sort of!) – face recognition – inference about people – character recognition – perhaps more

Computer Vision Big open problems • Next step in structure from motion • Really good missing variable formalism • Decent understanding of illumination, materials and shading • Segmentation • Representation for recognition • Efficient management of relations • Recognition processes for lots of objects • A lot of this looks like applied statistics

Computer Vision Next week: Final project presentations