ImageBased Visual Hulls Wojciech Matusik Chris Buehler Leonard

Image-Based Visual Hulls Wojciech Matusik Chris Buehler Leonard Mc. Millan Massachusetts Institute of Technology Laboratory for Computer Science Ramesh Raskar University of North Carolina at Chapel Hill Steven J. Gortler Harvard University

Motivation Real-time acquisition and rendering of dynamic scenes

Previous Work • Virtualized Reality (Rander’ 97, Kanade’ 97, Narayanan’ 98) • Visual Hull (Laurentini’ 94) • Volume Carving (Potmesil’ 87, Szeliski’ 93, Seitz’ 97) • CSG Rendering (Goldfeather’ 86, Rappoport’ 97) • Image-Based Rendering (Mc. Millan’ 95, Debevec’ 96, Debevec’ 98)

Contributions • View-dependent image-based visual hull representation • Efficient algorithm for sampling the visual hull • Efficient algorithm computing visibility • A real-time system

What is a Visual Hull?

Why use a Visual Hull? • Can be computed robustly • Can be computed efficiently background + foreground background - foreground =

Rendering Visual Hulls Reference 1 Desired Reference 2

Build then Sample Reference 1 Desired Reference 2

Sample Directly Reference 1 Desired Reference 2

Image-Based Computation Reference 1 Desired Reference 2

Observation • Incremental computation along scanlines Desired Reference

Binning • Sort silhouette edges into bins Epipole

Binning • Sort silhouette edges into bins Bin 1 Epipole

Binning • Sort silhouette edges into bins Bin 1 Bin 2 Epipole

Binning • Sort silhouette edges into bins Bin 1 Bin 2 Epipole Bin 3

Binning • Sort silhouette edges into bins Bin 1 Bin 2 Epipole Bin 3 Bin 4

Binning • Sort silhouette edges into bins Bin 1 Bin 2 Epipole Bin 3 Bin 4 Bin 5

Scanning Bin 1 Epipole

Scanning Bin 2 Epipole

Scanning Epipole Bin 4

Scanning Epipole Bin 5

Coarse-to-Fine Sampling

IBVH Results • Approximately constant computation per pixel per camera • Parallelizes • Consistent with input silhouettes

Video of IBVH

Shading Algorithm • A view-dependent strategy

Visibility Algorithm

Visibility in 2 D Reference view Desired view

Visibility in 2 D Front-most Points Reference view Desired view

Visibility in 2 D Visible Reference view Desired view

Visibility in 2 D Coverage Mask Reference view Desired view

Visibility in 2 D Visible Coverage Mask Reference view Desired view

Visibility in 2 D Not Visible Coverage Mask Reference view Desired view

Visibility in 2 D Coverage Mask Reference view Desired view

Visibility in 2 D Visible Coverage Mask Reference view Desired view

Visibility in 2 D Coverage Mask Reference view Desired view

Visibility in 2 D Not Visible Coverage Mask Reference view Desired view

Shaded Visual Hulls

System Camera Client Server (4 x 500 Mhz) Camera Client

System Trigger Signal Camera Client Server (4 x 500 Mhz) Camera Client

System Camera Client Server (4 x 500 Mhz) Camera Client

System Camera Client Compressed video Server (4 x 500 Mhz) Camera Client

System Camera Client Server (4 x 500 Mhz) Camera Client Intersection Camera Client

System Camera Client Server (4 x 500 Mhz) Visibility Camera Client

System Camera Client Server (4 x 500 Mhz) Shading Camera Client

More IBVH Results

Future Work • 3 D teleconferencing • Virtual sets • Post-production camera effects • Mixed reality

Summary • Visual hulls with texture can provide a compelling real-time visualizations • Visual hulls can be computed accurately and efficiently in image space • View dependent shading with visibility

Acknowledgements • DARPA ITO Grant F 30602 -971 -0283 • A generous grant from Intel Corporation • NSF Career Awards 9875859 & 9703399 • Tom Buehler & Kari Anne Kjølass • Thanks to all members of the MIT Computer Graphics Group