Example Based 3 D Shape Completion Mark Pauly

Example Based 3 D Shape Completion Mark Pauly 1, 2, Niloy J. Mitra 1, Joachim Giesen 2, Markus Gross 2, Leonidas J. Guibas 1 1 Stanford University 2 ETH, Zurich

Shape Completion • Incomplete raw scans • Imperfect range scanned data • Complex objects with occluded regions • Misalignment of multiple-views depth image scans • Specular highlights Ill-posed problem Use prior knowledge !! Example Based 3 D Shape Completion

Diffusion (Davis, Marschner, Garr, Levoy, 2002) Example Based 3 D Shape Completion

Context Based Solution (Sharf, Alexa, Cohen-or, 2004) Example Based 3 D Shape Completion

Template based Solution (Allen, Curless, Popovic, 2003; Kraevoy and Sheffer, 2005) Example Based 3 D Shape Completion

Our Solution • Use 3 D model database to provide geometric priors for shape completion • Apply non-rigid transforms on the models • More deformation less likely completion • Consistently combine geometric information from multiple context models • Final result comes with confidence values Example Based 3 D Shape Completion

Shape Completion Pipeline Example Based 3 D Shape Completion

Data Classification Local analysis • quality of fit • uniformity of sample distribution High Scored Point Cloud Low Example Based 3 D Shape Completion • confidence value assigned to each point

Shape Completion Pipeline Example Based 3 D Shape Completion

Database Retrieval 1. 93 1. 71 Example Based 3 D Shape Completion 1. 46 1. 27 1. 0

Shape Completion Pipeline Example Based 3 D Shape Completion

Non-rigid Alignment Similar to the approaches proposed by: • Allen, Curless and Popovic, 2003. • Sumner and Popovic, 2004. Example Based 3 D Shape Completion

Non-rigid Alignment Deformation Model • Piecewise linear. Each vertex of the mesh assigned an independent displacement vector. Optimize for smallest Shape Matching Penalty • Distortion Measure • Geometric Error Feature Correspondence Example Based 3 D Shape Completion Derived in the continuous setting to allow consistent comparison between different context models.

Warped Models Low Context Model Warped Model Matching Penalty High Example Based 3 D Shape Completion

Non-rigid Transform Example Based 3 D Shape Completion

Shape Completion Pipeline Example Based 3 D Shape Completion

Initial Segmentation Input Data Example Based 3 D Shape Completion Warped Context Model

Patch Growing Initial Segmentation Example Based 3 D Shape Completion Final Segmentation

Video Example Based 3 D Shape Completion

Shape Completion Pipeline Example Based 3 D Shape Completion

Blending Final Segments Example Based 3 D Shape Completion

Result Example Based 3 D Shape Completion

Shape Completion Pipeline Example Based 3 D Shape Completion

Giraffe Example Context Models Deformed Models Segmentation Example Based 3 D Shape Completion

Giraffe Example Context Models Deformed Models Final Model Example Based 3 D Shape Completion

Evaluation Input Data Context Model Example Based 3 D Shape Completion Final Model Evaluation

Enriching the Database Example Based 3 D Shape Completion

Additional Constraints Physical Model Acquired Data No Constraints Example Based 3 D Shape Completion Context Model Symmetry Constraints

Galleria dell’ Accademia Example Based 3 D Shape Completion

Future Directions • • • Improve the retrieval stage. Automatic feature point detection. Use of more semantic information. Apply learning techniques to shape completion. Completion of additional attributes like surface texture, color. Example Based 3 D Shape Completion

Acknowledgements • NSF grants CARGO-0138456, ITR-0205671, FRG-0454543, ARO grant DAAD 19 -03 -1 -033. • Stanford Graduate fellowship. • Mario Botsch, David Koller, Doo Young Kwon, Marc Levoy, Filip Sadlo, Vin da Silva, and Bob Sumner. Example Based 3 D Shape Completion

Questions Example Based 3 D Shape Completion

Example Based 3 D Shape Completion

Correspondence High Invalid Valid Input Data Example Based 3 D Shape Completion Warped Model Low

Video Example Based 3 D Shape Completion

Related Works • Example-based image completion Drori et al. 03 [Drori et al. 2003; Jia and tang 2003; Sun et al. 2003] • Texture synthesis [Efros and Leung 1999; Efros and Freeman 2001; Wei and Levoy 2000; Ying et al. 2001] Wei and Levoy 00 • Smooth surface completion [Curless and Levoy 1996; Davis et al. 2002; Ilic and Fua 2003; Verdera et al. 2003; Liepa 2003] Liepa 03 • Model-based surface reconstruction [Savchenko et al. 2002] • Curve analogies [Hertzmann et al. 2002] Example Based 3 D Shape Completion Hertzmann et al. 02