EURON Summer School 2003 From 2 D Images

EURON Summer School 2003 From 2 D Images to 3 D Tangible Models: Reconstruction and Visualization of Martian Rocks Cagatay Basdogan, Ph. D. College of Engineering Koc University

EURON Summer School 2003 The Old Story. . . Sojourner (July 04, 1997)

EURON Summer School 2003 The New Story. . . January 04, 2004

EURON Summer School 2003 Our goal. . .

From 2 D Images to 3 D Models EURON Summer School 2003 Acquisition Left Right Reconstruction Range surfaces 1 4 2 3 rock registration Visualization integration Transmission

EURON Summer School 2003 Limitations. . . Sojourner • CPU: • Memory: • Transmission Rate: • Transmission Delay • Transmission Interval • Transmission Reliability 2 MHz 768 Kb <5 bytes/sec ~ 20 min. ~ 2 hours poor Laptop 2 GHz 256 Mb + 40 Gb 100 Mb/sec < 200 msec anytime good

EURON Summer School 2003 Data Acquisition Range surfaces 3 1 2 Image-Based Rendering Registration Transformed Range surfaces Octree-Based Data Representation and Integration 3 1 Merged Range Data Progressive Transmission Multi-Resolution Transmitted Range Data 3 D Model Iso-surface Extraction 2

EURON Summer School 2003 Data Acquisition JPL – Marsyard (http: //marsyard. jpl. nasa. gov)

EURON Summer School 2003 Input: Range Scans 3 D range surface: coordinates connectivity Left Image Right Image

3 D Reconstruction Range Surface 1 EURON Summer School 2003 1. Registration Range Surface 2 Range Surface 3 2. Integration Top View 3 1 2

3 D Registration EURON Summer School 2003 Pair-wise registration problem: Given 2 overlapping range scans what is the rigid transformation, T, that minimizes the distance between them ? Q P Solution: ICP algorithm (Besl and Kay, 1992) T do Identify corresponding points Compute the optimal T while (E < threshold)

EURON Summer School 2003 3 D Registration : Computation of T R: Rotation Matrix t: translation vector Solution by Horn, 1990: where,

EURON Summer School 2003 3 D Registration: Corresponding Points We now have a closed form solution, how do we get the corresponding points in two scans? Nearest points along the direction of the normal at qi Q P Top View 1 3 2

3 D Integration EURON Summer School 2003 Problem: Given a set of registered range scans, reconstruct a 3 D surface that closely approximates the original shape. Methods: • Delaunay-Based (Amenta et al. , Siggraph’ 98) • Surface-Based (Turk and Levoy, Siggraph’ 94) • Volumetric (Curless and Levoy, Siggraph’ 96) Which one is better ? Computationally Complexity Robustness Memory Usage. . .

EURON Summer School 2003 3 D Integration: Volumetric Approach Input: registered range surfaces Volumetric Integration Output: 3 D Mesh Our implementation: Step 1: Merge the registered range surfaces using an octree Step 2: Extract an isosurface using the Marching Cubes algorithm

EURON Summer School 2003 3 D Integration (Step 1): Merge Range Images Registered rock Mesh 3 Mesh 2 Mesh 1 Merged (no connectivity)

EURON Summer School 2003 3 D Integration (Step 1): Data Representation Using an Octree root Level 0 Level 1 Level 2 Weighted averaging Yemez & Schmitt, 1999

EURON Summer School 2003 Data Reduction Using an Octree Max 500 points/octant Total: 273 points 2 D Example: Quadtree Max 1 point/rectangle Max 100 points/octant Total: 1052 points Max 50 points/octant Total: 1870 points Original Model: 47724 points

EURON Summer School 2003 Octree Representation: Our Implementation root 2 3 4 1 1 1 3 2 1 1 Level 0 Level 1 4 2 3 4 Level 2 2 3 4 Level 3 2 3 4 Reference: = center Path : 1 3 1 1 estimated coordinate and shape 1 2 3 4

EURON Summer School 2003 Octree Encoding Yemez & Schmitt, 1999 6 Our implementation: octree root 1 3 0 2 layer 0 5 4 0 1 1 0 0 0 1 0 layer 1 6 6 0 2 1 3 5 4 6 0 2 1 3 6 5 4 1 3 0 2 5 4 layer 2 1 0 1 1 0 0 0 0 1 0 1 0 0 0 1 0 6 layer 3 4 0 1 2 3 5 4 1000

EURON Summer School 2003 Octrees for 3 D Progressive Data Transmission root Level 0 Level 1 Level 2 215 points . . Level 5 2813 points . . Level 8 5823 points

EURON Summer School 2003 3 D Integration (Step 2): Iso-surface Extraction Merged Data Progressive Transmission Transmitted Data Iso-surface Extraction: Marching Cubes 3 D Mesh v 1 x v 1 y v 1 z v 2 x v 2 y v 2 z … rgb 1 rgb 2. . . n 1 x n 1 y n 1 z n 2 x n 2 y n 2 z. . . Coordinates High Priority Colors Normals Low Priority

EURON Summer School 2003 3 D Integration (Step 2): Marching Cubes 2 -D Example OUT IN -0. 5 +0. 3 iso +0. 8 lin e +0. 4 24 = 16 possible cases

EURON Summer School 2003 Marching Cubes Algorithm : 3 D Case (Lorensen et al. , Siggraph’ 87) 28 = 256 possible cases (a look-up table reduces to 15 cases)

EURON Summer School 2003 3 D Integration: Step 2: Signed Distance Computation -0. 7 isosurfac e +0. 1 Voxel (IN) -0. 3 v 1 +0. 4 v 2 n 1 n 2 Voxel (OUT) Signed Distance: dot(v 1, n 1) > 0 IN, distance = -|v 1| dot(v 2, n 2) < 0 OUT, distance = +|v 2|

EURON Summer School 2003 Can I estimate Normals? Problem: Given a set of P unorganized sample points, estimate the point normals. Algorithm by Hoppe (Siggraph’ 92): Step 1: Tangent Plane Estimation Step 2: Consistent Tangent Plane Orientation

EURON Summer School 2003 N 1. Tangent Plane Estimation N? Sample point N? Neighbor R R ~ (s + r) Noise Density Centeroid Use covariance matrix to compute N SVD : eigenvalues : l 1 > l 2 > l 3 eigenvectors : v 1, v 2 , v 3 N v 3 -v 3

Summer School 2003 2. Consistent Tangent Plane Orientation: Graph EURON Optimization Problem MST: A minimal spanning tree for a connected graph 7 8 9 4 2 11 14 4 8 6 Incorrect Surface Normals 1 10 2 Cost on edge: 1 –Ni. Nj Propagate along directions of low curvature! Ni Corrected Surface Normals Nj Ni Nj

EURON Summer School 2003 point-based rendering with colors

EURON Summer School 2003 3 D Visualization Iso-surface Extraction 3 D Mesh 3 D Texture

EURON Summer School 2003 Autostereoscopic Visualization 3 D Visualization without any eye wear ! Src: Stereographics Inc. Stereoscopic viewing Autostereoscopic viewing

EURON Summer School 2003 Stereoscopic Visualization eye. L eye. R Holographic Optical Element LCDL Stereoscopic viewing LCDR Autostereoscopic viewing

EURON Summer School 2003 Autostereoscopic Displays Relatively new area ! Hale et al. , 1997, Siggraph Perlin et al. , 2000, Siggraph holographic optical element Classification by Hale et al. : • Re-imaging displays • Volumetric displays • Parallax displays • Holograms • Parallax Barrier Displays • Lenticular Sheet Displays • Holographic Stereograms • Electro-Holography

EURON Summer School 2003 Stereo Rendering : Shear Transform Near Distance Far Distance Camera Position Shear Direction Holographic Origin (0, 0, 0) Plate le h Height A idt ngle g An W a) Symmetric Frustum b) Asymmetric Frustum

EURON Summer School 2003 Haptic Interface Shear Transform: v y z r w. L x w. R Autostereoscopic Display Eye pos:

EURON Summer School 2003 Haptic Visualization • Rendering rock textures • Displaying 3 D rock shapes • Tele-science experiments • Guiding user’s movements • Positioning rover instruments HIP

EURON Summer School 2003 Haptic Display of Shape Position Orientation Rock Force Torque Collision Detection Contact Information Collision Response Object Database Rock Geometry Material Properties of Rock

EURON Summer School 2003 Mapping Between Visual and Haptic Workspaces T 3= T 1. T 2 Visual Workspace Haptic Workspace 3 D model of a rock T 2 T 1

EURON Summer School 2003 Synchronization of Cursor Movements HIP FNEW HIP Collision Detection (T 2)-1 T 2 (T 2) F Collision Response Display Visual Cursor

EURON Summer School 2003

EURON Summer School 2003

On-board Computation Unsolved/Untouched Problems EURON Summer School 2003 Resources • 3 D Registration: problems with ICP, global registration • 3 D Integration: robustness, storage requirements • 3 D Transmission: 3 D geometry comp. vs 3 D data comp. • More effective transmission of normals and colors • 3 D Visual and Haptic Texturing • Image-Based rendering • Optimized computation (e. g. efficient data structures such as ADFs) • Missing link between image analysis and 3 d modeling • More efficient graphical rendering (e. g. point-based rendering) • Missing link between real-time 3 D modeling and rover navigation • Unified data structures for transmission of multi-modal data

Acknowledgements: EURON Summer School 2003 Supporting NASA Programs: IPN, ISE, CISMISS, USRP, SBIR, JPL/Caltech-UROP JPL Andres Castano (acquisition, registration) Aaron Keily (encoding) Ed Chow, Jose Salcedo (autostereoscopic visualization) Larry Bergman (human-rover interactions) Industry (Physical Optics Corporation) Steve Cupiac (autostereoscopic visualization) Andrew Kostrzewski, Kirill Kolesnikov (autostereoscopic display; hardware integration) Students Mitch Lum, UW (autostereoscopic and haptic visualization) Elaine Ou, Caltech (vision and touch) University Prof. Marc Levoy, Stanford (3 D range scans of “bunny”, Scanalyze registration software; personal communication) Prof. Brian Curless, UW (3 D photography notes; public domain) Dr. Huges Hoppe, Microsoft (Ph. D. Thesis and 3 D reconstruction code; public domain)

% Mean Error Relative to BBox Diagonal of High Resolution Model EURON Summer School 2003 High Resolution Model (35947 pts) 3. 00 2. 40 1. 80 Averaged Center 1. 20 0. 60 (215 pts) (857 pts) 4 3 5 (4834 pts) 6 7 (5823 pts) 8 Number of Octree Layers

% Mean Error Relative to BBox Diagonal of High Resolution Model EURON Summer School 2003 Using Estimated Normals Using Computed Normals 3. 60 High Resolution Model (35947 pts) 3. 00 2. 40 1. 80 1. 20 0. 60 (220 pts) (931 pts) 4 3 5 (13156 pts) 7 6 (35781 pts) 8 Number of Octree Layers

EURON Summer School 2003 Octrees for 3 D Data Transmission: Encoding 6 0 2 abc 5 4 1 3 bites/octant 3 000 001 010. . 111 0 1 2 a*22 + b*2 + c 7 Ref: Yemez and Schmitt, 1999 1 m Path to a leaf node: 1 m 1 m 34170332 8 layers = 28 X 28 cubes Resolution = 1000 mm / 28 = ~ 4 mm !!!

EURON Summer School 2003 Octrees for Progressive Transmission All data is transmitted at once (Maximum 8 layers): 28 X 28 cubes * 3 bites/cube * 1 byte/8 bits = ~ 6. 3 MB !! Progressive Transmission: Path to a leaf node: 34170332 Layer 1 … Layer 8 If we transmit the difference between layer 4 to 5 : ~ 10 KB ! (not even compressed)

EURON Summer School 2003

EURON Summer School 2003 A Simple 3 D Example: Input Data: Output v 1 x v 1 y v 1 z v 2 x v 2 y v 2 z … v 10 x v 10 y v 10 z n 1 x n 1 y n 1 z n 2 x n 2 y n 2 z … n 10 x n 10 y n 10 z Voxelization

EURON Summer School 2003 Stereoscopic Visualization: Depth Perception 2 D: • Perspective • Occlusion • Lighting, shadows • Relative motion • Texture 3 D: • Binocular disparity • Accommodation • Convergence Stereo Pairs

EURON Summer School 2003 Stereo Rendering Incorrect ! Correct !