IBM IBR The study of imagebased modeling and

IBM / IBR “The study of image-based modeling and rendering is the study of sampled representations of geometry. ” Ó 2002 Marc Levoy

Shortcutting the vision/graphics pipeline real world vision pipeline geometry image-based rendering graphics pipeline views (from M. Cohen) Ó 2002 Marc Levoy

Apple Quick. Time VR [Chen, Siggraph ’ 95] • outward-looking – panoramic views taken at regularly spaced points • inward-looking – views taken at points on the surface of a sphere Ó 2002 Marc Levoy

View interpolation from a single view 1. Render object 2. Convert Z-buffer to range image 3. Re-render from new viewpoint 4. Use depths to resolve overlaps Q. How to fill in holes? Ó 2002 Marc Levoy

View interpolation from multiple views 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Re-render from new viewpoint 4. Use depths to resolve overlaps 5. Use multiple views to fill in holes Ó 2002 Marc Levoy

Post-rendering 3 D warping [Mark et al. , I 3 D 97] • render at low frame rate • interpolate to real-time frame rate – – interpolate observer viewpoint using B-Spline convert reference images to polygon meshes warp meshes to interpolated viewpoint composite by Z-buffer comparison and conditional write Ó 2002 Marc Levoy

Results • rendered at 5 fps, interpolated to 30 fps • live system requires reliable motion prediction – tradeoff between accuracy and latency • fails on specular objects Ó 2002 Marc Levoy

Light field rendering [Levoy & Hanrahan, Siggraph ’ 96] • must stay outside convex hull of the object • like rebinning in computed tomography Ó 2002 Marc Levoy

A light field is an array of images Ó 2002 Marc Levoy

Stanford multi-camera array (Horowitz, Levoy, Hanrahan) • cheap imagers + cheap optics + fast networking + plentiful computation = high-performance imaging using an array of low-cost cameras Ó 2002 Marc Levoy

Applications for the array • key issue is the spacing and arrangement of the cameras Ó 2002 Marc Levoy

Cameras tightly packed: high-X imaging • high-resolution – by abutting the cameras’ fields of view • high speed – by staggering their triggering times • high dynamic range – mosaic of shutter speeds, apertures, density filters • high depth of field – mosaic of differently focused lenses Ó 2002 Marc Levoy

Cameras widely spaced: a video-rate light field camera • compression of (video) light fields – key is estimation of rough geometry • autostereoscopic display of light fields – lens arrays, bumpy mirrored sheets • shape from light fields – Do more images help? Ó 2002 Marc Levoy