Example Based 3 D Shape Completion Mark Pauly
Example Based 3 D Shape Completion Mark Pauly 1, 2, Niloy J. Mitra 1, Joachim Giesen 2, Markus Gross 2, Leonidas J. Guibas 1 1 Stanford University 2 ETH, Zurich
Shape Completion • Incomplete raw scans • Imperfect range scanned data • Complex objects with occluded regions • Misalignment of multiple-views depth image scans • Specular highlights Ill-posed problem Use prior knowledge !! Example Based 3 D Shape Completion
Diffusion (Davis, Marschner, Garr, Levoy, 2002) Example Based 3 D Shape Completion
Context Based Solution (Sharf, Alexa, Cohen-or, 2004) Example Based 3 D Shape Completion
Template based Solution (Allen, Curless, Popovic, 2003; Kraevoy and Sheffer, 2005) Example Based 3 D Shape Completion
Our Solution • Use 3 D model database to provide geometric priors for shape completion • Apply non-rigid transforms on the models • More deformation less likely completion • Consistently combine geometric information from multiple context models • Final result comes with confidence values Example Based 3 D Shape Completion
Shape Completion Pipeline Example Based 3 D Shape Completion
Shape Completion Pipeline Example Based 3 D Shape Completion
Data Classification Local analysis • quality of fit • uniformity of sample distribution High Scored Point Cloud Low Example Based 3 D Shape Completion • confidence value assigned to each point
Shape Completion Pipeline Example Based 3 D Shape Completion
Database Retrieval 1. 93 1. 71 Example Based 3 D Shape Completion 1. 46 1. 27 1. 0
Shape Completion Pipeline Example Based 3 D Shape Completion
Non-rigid Alignment Similar to the approaches proposed by: • Allen, Curless and Popovic, 2003. • Sumner and Popovic, 2004. Example Based 3 D Shape Completion
Non-rigid Alignment Deformation Model • Piecewise linear. Each vertex of the mesh assigned an independent displacement vector. Optimize for smallest Shape Matching Penalty • Distortion Measure • Geometric Error Feature Correspondence Example Based 3 D Shape Completion Derived in the continuous setting to allow consistent comparison between different context models.
Warped Models Low Context Model Warped Model Matching Penalty High Example Based 3 D Shape Completion
Non-rigid Transform Example Based 3 D Shape Completion
Shape Completion Pipeline Example Based 3 D Shape Completion
Initial Segmentation Input Data Example Based 3 D Shape Completion Warped Context Model
Patch Growing Initial Segmentation Example Based 3 D Shape Completion Final Segmentation
Video Example Based 3 D Shape Completion
Shape Completion Pipeline Example Based 3 D Shape Completion
Blending Final Segments Example Based 3 D Shape Completion
Result Example Based 3 D Shape Completion
Shape Completion Pipeline Example Based 3 D Shape Completion
Shape Completion Pipeline Example Based 3 D Shape Completion
Giraffe Example Context Models Deformed Models Segmentation Example Based 3 D Shape Completion
Giraffe Example Context Models Deformed Models Final Model Example Based 3 D Shape Completion
Evaluation Input Data Context Model Example Based 3 D Shape Completion Final Model Evaluation
Enriching the Database Example Based 3 D Shape Completion
Additional Constraints Physical Model Acquired Data No Constraints Example Based 3 D Shape Completion Context Model Symmetry Constraints
Galleria dell’ Accademia Example Based 3 D Shape Completion
Galleria dell’ Accademia Example Based 3 D Shape Completion
Galleria dell’ Accademia Example Based 3 D Shape Completion
Galleria dell’ Accademia Example Based 3 D Shape Completion
Future Directions • • • Improve the retrieval stage. Automatic feature point detection. Use of more semantic information. Apply learning techniques to shape completion. Completion of additional attributes like surface texture, color. Example Based 3 D Shape Completion
Acknowledgements • NSF grants CARGO-0138456, ITR-0205671, FRG-0454543, ARO grant DAAD 19 -03 -1 -033. • Stanford Graduate fellowship. • Mario Botsch, David Koller, Doo Young Kwon, Marc Levoy, Filip Sadlo, Vin da Silva, and Bob Sumner. Example Based 3 D Shape Completion
Questions Example Based 3 D Shape Completion
Example Based 3 D Shape Completion
Correspondence High Invalid Valid Input Data Example Based 3 D Shape Completion Warped Model Low
Video Example Based 3 D Shape Completion
Related Works • Example-based image completion Drori et al. 03 [Drori et al. 2003; Jia and tang 2003; Sun et al. 2003] • Texture synthesis [Efros and Leung 1999; Efros and Freeman 2001; Wei and Levoy 2000; Ying et al. 2001] Wei and Levoy 00 • Smooth surface completion [Curless and Levoy 1996; Davis et al. 2002; Ilic and Fua 2003; Verdera et al. 2003; Liepa 2003] Liepa 03 • Model-based surface reconstruction [Savchenko et al. 2002] • Curve analogies [Hertzmann et al. 2002] Example Based 3 D Shape Completion Hertzmann et al. 02
- Slides: 41