RealTime 3 D Model Acquisition Szymon Rusinkiewicz Olaf

  • Slides: 45
Download presentation
Real-Time 3 D Model Acquisition Szymon Rusinkiewicz Olaf Hall-Holt Marc Levoy Princeton University Stanford

Real-Time 3 D Model Acquisition Szymon Rusinkiewicz Olaf Hall-Holt Marc Levoy Princeton University Stanford University

3 D Scanning

3 D Scanning

Possible Research Goals • Low noise • Guaranteed high accuracy • High speed •

Possible Research Goals • Low noise • Guaranteed high accuracy • High speed • Low cost • Automatic operation • No holes

3 D Model Acquisition Pipeline 3 D Scanner

3 D Model Acquisition Pipeline 3 D Scanner

3 D Model Acquisition Pipeline 3 D Scanner View Planning

3 D Model Acquisition Pipeline 3 D Scanner View Planning

3 D Model Acquisition Pipeline 3 D Scanner View Planning Alignment

3 D Model Acquisition Pipeline 3 D Scanner View Planning Alignment

3 D Model Acquisition Pipeline 3 D Scanner View Planning Alignment Merging

3 D Model Acquisition Pipeline 3 D Scanner View Planning Alignment Merging

3 D Model Acquisition Pipeline 3 D Scanner View Planning Alignment Done? Merging

3 D Model Acquisition Pipeline 3 D Scanner View Planning Alignment Done? Merging

3 D Model Acquisition Pipeline 3 D Scanner View Planning Alignment Done? Merging Display

3 D Model Acquisition Pipeline 3 D Scanner View Planning Alignment Done? Merging Display

3 D Model Acquisition Difficulties • Much (often most) time spent on “last 20%”

3 D Model Acquisition Difficulties • Much (often most) time spent on “last 20%” • Pipeline not optimized for hole-filling • Not sufficient just to speed up scanner – must design pipeline for fast feedback

Real-Time 3 D Model Acquisition

Real-Time 3 D Model Acquisition

Real-Time 3 D Model Acquisition Pipeline 3 D Scanner View Planning Alignment Human Done?

Real-Time 3 D Model Acquisition Pipeline 3 D Scanner View Planning Alignment Human Done? Merging Display

Real-Time 3 D Model Acquisition Pipeline 3 D Scanner Alignment View Planning Challenge: Real

Real-Time 3 D Model Acquisition Pipeline 3 D Scanner Alignment View Planning Challenge: Real Time Done? Merging Display

Real-Time 3 D Model Acquisition Pipeline 3 D Scanner View Planning Done? Alignment Part

Real-Time 3 D Model Acquisition Pipeline 3 D Scanner View Planning Done? Alignment Part I: Structured-Light Triangulation Display Merging

Real-Time 3 D Model Acquisition Pipeline 3 D Scanner View Planning Alignment Part II:

Real-Time 3 D Model Acquisition Pipeline 3 D Scanner View Planning Alignment Part II: Fast ICP Done? Merging Display

Real-Time 3 D Model Acquisition Pipeline 3 D Scanner View Planning Alignment Part III:

Real-Time 3 D Model Acquisition Pipeline 3 D Scanner View Planning Alignment Part III: Voxel Grid Done? Merging Display

Triangulation Object Laser Camera • Project laser stripe onto object

Triangulation Object Laser Camera • Project laser stripe onto object

Triangulation Object Laser (x, y) Camera • Depth from ray-plane triangulation

Triangulation Object Laser (x, y) Camera • Depth from ray-plane triangulation

Triangulation • Faster acquisition: project multiple stripes • Correspondence problem: which stripe is which?

Triangulation • Faster acquisition: project multiple stripes • Correspondence problem: which stripe is which?

Continuum of Triangulation Methods Multi-stripe Multi-frame Single-stripe Slow, robust Single-frame Fast, fragile

Continuum of Triangulation Methods Multi-stripe Multi-frame Single-stripe Slow, robust Single-frame Fast, fragile

Time-Coded Light Patterns • Assign each stripe a unique illumination code over time [Posdamer

Time-Coded Light Patterns • Assign each stripe a unique illumination code over time [Posdamer 82] Time Space

Codes for Moving Scenes • Assign time codes to stripe boundaries • Perform frame-to-frame

Codes for Moving Scenes • Assign time codes to stripe boundaries • Perform frame-to-frame tracking of corresponding boundaries – Propagate illumination history Illumination history = (WB), (BW), (WB) [Hall-Holt & Rusinkiewicz, ICCV 2001] Code

Designing a Code • Want many “features” to track: lots of black/white edges at

Designing a Code • Want many “features” to track: lots of black/white edges at each frame • Try to minimize ghosts – WW or BB “boundaries” that can’t be seen directly

Designing a Code 0000 1101 1010 1111 0010 0101 1000 1011 0110 0001 0100

Designing a Code 0000 1101 1010 1111 0010 0101 1000 1011 0110 0001 0100 1110 0111 1100 1001 0011 [Hall-Holt & Rusinkiewicz, ICCV 2001]

Implementation • Pipeline: Project Code Capture Images Find Boundaries Match Boundaries Decode • DLP

Implementation • Pipeline: Project Code Capture Images Find Boundaries Match Boundaries Decode • DLP projector illuminates scene @ 60 Hz. • Synchronized NTSC camera captures video • Pipeline returns range images @ 60 Hz. Compute Range

Real-Time 3 D Model Acquisition Pipeline 3 D Scanner View Planning Alignment Part II:

Real-Time 3 D Model Acquisition Pipeline 3 D Scanner View Planning Alignment Part II: Fast ICP Done? Merging Display

Aligning 3 D Data • ICP (Iterative Closest Points): for each point on one

Aligning 3 D Data • ICP (Iterative Closest Points): for each point on one scan, minimize distance to closest point on other scan…

Aligning 3 D Data • … and iterate to find alignment – Iterated Closest

Aligning 3 D Data • … and iterate to find alignment – Iterated Closest Points (ICP) [Besl & Mc. Kay 92]

ICP in the Real-Time Pipeline • Potential problem with ICP: local minima – In

ICP in the Real-Time Pipeline • Potential problem with ICP: local minima – In this pipeline, scans close together – Very likely to converge to correct (global) minimum • Basic ICP algorithm too slow (~ seconds) – Point-to-plane minimization – Projection-based matching – With these tweaks, running time ~ milliseconds [Rusinkiewicz & Levoy, 3 DIM 2001]

Real-Time 3 D Model Acquisition Pipeline 3 D Scanner View Planning Alignment Part III:

Real-Time 3 D Model Acquisition Pipeline 3 D Scanner View Planning Alignment Part III: Voxel Grid Done? Merging Display

Merging and Rendering • Goal: visualize the model well enough to be able to

Merging and Rendering • Goal: visualize the model well enough to be able to see holes • Cannot display all the scanned data – accumulates linearly with time • Standard high-quality merging methods: processing time ~ 1 minute per scan

Merging and Rendering

Merging and Rendering

Merging and Rendering

Merging and Rendering

Merging and Rendering

Merging and Rendering

Merging and Rendering +

Merging and Rendering +

Merging and Rendering • Point rendering, using accumulated normals for lighting

Merging and Rendering • Point rendering, using accumulated normals for lighting

Example: Photograph 18 cm.

Example: Photograph 18 cm.

Result

Result

Postprocessing • Real-time display – Quality/speed tradeoff – Goal: let user evaluate coverage, fill

Postprocessing • Real-time display – Quality/speed tradeoff – Goal: let user evaluate coverage, fill holes • Offline postprocessing for high-quality models – Global registration – High-quality merging (e. g. , using VRIP [Curless 96])

Postprocessed Model

Postprocessed Model

Recapturing Alignment

Recapturing Alignment

Summary • 3 D model acquisition pipeline optimized for obtaining complete, hole-free models •

Summary • 3 D model acquisition pipeline optimized for obtaining complete, hole-free models • Use human’s time most efficiently • Pieces of pipeline selected for real-time use: – Structured-light scanner for moving objects – Fast ICP variant – Simple grid-based merging, point rendering

Limitations • Prototype noisier than commercial systems – Could be made equivalent with careful

Limitations • Prototype noisier than commercial systems – Could be made equivalent with careful engineering – Ultimate limitations on quality: focus, texture • Scan-to-scan ICP not perfect alignment drift – Due to noise, miscalibration, degenerate geometry – Reduced, but not eliminated, by “anchor scans” – Possibly combine ICP with separate trackers

Future Work • Faster scanning – Better stripe boundary tracking – Multiple cameras, projectors

Future Work • Faster scanning – Better stripe boundary tracking – Multiple cameras, projectors – High-speed cameras, projectors • Application in different contexts – Cart- or shoulder-mounted for digitizing rooms – Infrared for imperceptibility

Acknowledgments • Collaborators: – Li-Wei He – James Davis – Lucas Pereira – Sean

Acknowledgments • Collaborators: – Li-Wei He – James Davis – Lucas Pereira – Sean Anderson • Sponsors: – Sony – Intel – Interval