Imagebased modeling IBM and imagebased rendering IBR CS
![Image-based modeling (IBM) and image-based rendering (IBR) CS 248 - Introduction to Computer Graphics Image-based modeling (IBM) and image-based rendering (IBR) CS 248 - Introduction to Computer Graphics](https://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-1.jpg)
Image-based modeling (IBM) and image-based rendering (IBR) CS 248 - Introduction to Computer Graphics Autumn quarter, 2005 Slides for December 8 lecture Ó 2005 Marc Levoy
![The graphics pipeline modeling animation rendering Ó 2005 Marc Levoy The graphics pipeline modeling animation rendering Ó 2005 Marc Levoy](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-2.jpg)
The graphics pipeline modeling animation rendering Ó 2005 Marc Levoy
![The graphics pipeline the traditional pipeline modeling animation rendering motion capture image-based rendering the The graphics pipeline the traditional pipeline modeling animation rendering motion capture image-based rendering the](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-3.jpg)
The graphics pipeline the traditional pipeline modeling animation rendering motion capture image-based rendering the new pipeline? 3 D scanning Ó 2005 Marc Levoy
![IBM / IBR “The study of image-based modeling and rendering is the study of IBM / IBR “The study of image-based modeling and rendering is the study of](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-4.jpg)
IBM / IBR “The study of image-based modeling and rendering is the study of sampled representations of geometry. ” Ó 2005 Marc Levoy
![Image-based representations: the classics 3 D more geometry – model + texture/reflectance map – Image-based representations: the classics 3 D more geometry – model + texture/reflectance map –](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-5.jpg)
Image-based representations: the classics 3 D more geometry – model + texture/reflectance map – model + displacement map – volume rendering [Blinn 78] [Cook 84] [Levoy 87, Drebin 88] 2 D + Z – range images – disparity maps [Binford 73] [vision literature] 2. 5 D – sprites [vis-sim, games] n 2 D – epipolar plane images – movie maps [Bolles 87] – environment maps, a. k. a. panoramas [19 th century] [Lippman 78] 2 D less geometry Ó 2005 Marc Levoy
![Recent additions more geometry full model – view-dependent textures [Debevec 96] – surface light Recent additions more geometry full model – view-dependent textures [Debevec 96] – surface light](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-6.jpg)
Recent additions more geometry full model – view-dependent textures [Debevec 96] – surface light fields – Lumigraphs [Wood 00] [Gortler 96] sets of range images – view interpolation – layered depth images – relief textures [Chen 93, Mc. Millan 95, Mark 97] [Shade 98] [Oliveira 00] feature correspondences – plenoptic editing [Seitz 98, Dorsey 01] camera pose less geometry – image caching – sprites + warps – light fields [Schaufler 96, Shade 96] [Lengyel 97] [Levoy 96] no model – outward-looking QTVR [Chen 95] Ó 2005 Marc Levoy
![Rangefinding technologies • passive – shape from stereo – shape from focus – shape Rangefinding technologies • passive – shape from stereo – shape from focus – shape](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-7.jpg)
Rangefinding technologies • passive – shape from stereo – shape from focus – shape from motion, etc. • active – texture-assisted shape-from-X – triangulation using structured-light – time-of-flight Ó 2004 Marc Levoy
![Laser triangulation rangefinding Ó 2004 Marc Levoy Laser triangulation rangefinding Ó 2004 Marc Levoy](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-8.jpg)
Laser triangulation rangefinding Ó 2004 Marc Levoy
![single scan of St. Matthew 1 mm single scan of St. Matthew 1 mm](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-9.jpg)
single scan of St. Matthew 1 mm
![Post-processing pipeline • steps 1. aligning the scans 2. combining aligned scans 3. filling Post-processing pipeline • steps 1. aligning the scans 2. combining aligned scans 3. filling](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-10.jpg)
Post-processing pipeline • steps 1. aligning the scans 2. combining aligned scans 3. filling holes Ó 2004 Marc Levoy
![Digitizing the statues of Michelangelo using laser scanners • • • 480 individually aimed Digitizing the statues of Michelangelo using laser scanners • • • 480 individually aimed](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-11.jpg)
Digitizing the statues of Michelangelo using laser scanners • • • 480 individually aimed scans 2 billion polygons 7, 000 color images 30 nights of scanning 22 people Ó 2004 Marc Levoy
![](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-12.jpg)
![](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-13.jpg)
![Replica of Michelangelo’s David (20 cm tall) Ó 2004 Marc Levoy Replica of Michelangelo’s David (20 cm tall) Ó 2004 Marc Levoy](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-14.jpg)
Replica of Michelangelo’s David (20 cm tall) Ó 2004 Marc Levoy
![Solving the jigsaw puzzle of the Forma Urbis Romae Solving the jigsaw puzzle of the Forma Urbis Romae](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-15.jpg)
Solving the jigsaw puzzle of the Forma Urbis Romae
![The puzzle as it now stands Ó 2005 Marc Levoy The puzzle as it now stands Ó 2005 Marc Levoy](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-16.jpg)
The puzzle as it now stands Ó 2005 Marc Levoy
![Clues for solving the puzzle • • • incised lines incision characteristics marble veining Clues for solving the puzzle • • • incised lines incision characteristics marble veining](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-17.jpg)
Clues for solving the puzzle • • • incised lines incision characteristics marble veining fragment thickness shapes of fractured surfaces rough / smooth bottom surface straight sides, indicating slab boundaries location and shapes of clamp holes the wall: slab layout, clamp holes, stucco archaeological evidence Ó 2005 Marc Levoy
![Matching incised lines fragment 156 fragment 167 fragment 134 Matching incised lines fragment 156 fragment 167 fragment 134](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-18.jpg)
Matching incised lines fragment 156 fragment 167 fragment 134
![fragment 156 fragment 167 fragment 134 fragment 156 fragment 167 fragment 134](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-19.jpg)
fragment 156 fragment 167 fragment 134
![Geometry-based versus image-based rendering conceptual world model construction real world rendering geometry-based rendering flythrough Geometry-based versus image-based rendering conceptual world model construction real world rendering geometry-based rendering flythrough](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-20.jpg)
Geometry-based versus image-based rendering conceptual world model construction real world rendering geometry-based rendering flythrough of scene image acquisition images computer vision image-based rendering flythrough of scene Ó 2004 Marc Levoy
![Shortcutting the vision/graphics pipeline real world vision pipeline geometry image-based rendering graphics pipeline views Shortcutting the vision/graphics pipeline real world vision pipeline geometry image-based rendering graphics pipeline views](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-21.jpg)
Shortcutting the vision/graphics pipeline real world vision pipeline geometry image-based rendering graphics pipeline views (from M. Cohen) Ó 2004 Marc Levoy
![Apple Quick. Time VR [Chen, Siggraph ’ 95] • outward-looking – panoramic views taken Apple Quick. Time VR [Chen, Siggraph ’ 95] • outward-looking – panoramic views taken](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-22.jpg)
Apple Quick. Time VR [Chen, Siggraph ’ 95] • outward-looking – panoramic views taken at regularly spaced points • inward-looking – views taken at points on the surface of a sphere Ó 2004 Marc Levoy
![View interpolation from a single view 1. Render object 2. Convert Z-buffer to range View interpolation from a single view 1. Render object 2. Convert Z-buffer to range](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-23.jpg)
View interpolation from a single view 1. Render object 2. Convert Z-buffer to range image 3. Tesselate to create polygon mesh 4. Re-render from new viewpoint 5. Use depths to resolve overlaps Q. How to fill in holes? Ó 2004 Marc Levoy
![View interpolation from multiple views 1. Render object from multiple viewpoints 2. Convert Z-buffers View interpolation from multiple views 1. Render object from multiple viewpoints 2. Convert Z-buffers](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-24.jpg)
View interpolation from multiple views 1. Render object from multiple viewpoints 2. Convert Z-buffers to range images 3. Tesselate to create multiple meshes 4. Re-render from new viewpoint 5. Use depths to resolve overlaps 6. Use multiple views to fill in holes Ó 2004 Marc Levoy
![Post-rendering 3 D warping [Mark et al. , I 3 D 97] • render Post-rendering 3 D warping [Mark et al. , I 3 D 97] • render](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-25.jpg)
Post-rendering 3 D warping [Mark et al. , I 3 D 97] • render at low frame rate • interpolate to real-time frame rate – – interpolate observer viewpoint using B-Spline convert reference images to polygon meshes warp meshes to interpolated viewpoint composite by Z-buffer comparison and conditional write Ó 2004 Marc Levoy
![Results • rendered at 5 fps, interpolated to 30 fps • live system requires Results • rendered at 5 fps, interpolated to 30 fps • live system requires](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-26.jpg)
Results • rendered at 5 fps, interpolated to 30 fps • live system requires reliable motion prediction – tradeoff between accuracy and latency • fails on specular objects Ó 2004 Marc Levoy
![Image caching [Shade et al. , SIGGRAPH 1996] • precompute BSP tree of scene Image caching [Shade et al. , SIGGRAPH 1996] • precompute BSP tree of scene](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-27.jpg)
Image caching [Shade et al. , SIGGRAPH 1996] • precompute BSP tree of scene (2 D in this case) • for first observer position – draw nearby nodes (yellow) as geometry – render distant nodes (red) to RGB images (black) – composite images together • as observer moves – if disparity exceeds a threshold, rerender image Ó 2004 Marc Levoy
![Light field rendering [Levoy & Hanrahan, SIGGRAPH 1996] • must stay outside convex hull Light field rendering [Levoy & Hanrahan, SIGGRAPH 1996] • must stay outside convex hull](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-28.jpg)
Light field rendering [Levoy & Hanrahan, SIGGRAPH 1996] • must stay outside convex hull of the object • like rebinning in computed tomography Ó 2004 Marc Levoy
![The plenoptic function Radiance as a function of position and direction in a static The plenoptic function Radiance as a function of position and direction in a static](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-29.jpg)
The plenoptic function Radiance as a function of position and direction in a static scene with fixed illumination • for general scenes 5 D function L ( x, y, z, , f ) • in free space 4 D function “ the (scalar) light field” Ó 2004 Marc Levoy
![The free-space assumption • applications for free-space light fields – flying around a compact The free-space assumption • applications for free-space light fields – flying around a compact](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-30.jpg)
The free-space assumption • applications for free-space light fields – flying around a compact object – flying through an uncluttered environment Ó 2004 Marc Levoy
![Some candidate parameterizations Point-on-plane + direction y L ( x, y, , f ) Some candidate parameterizations Point-on-plane + direction y L ( x, y, , f )](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-31.jpg)
Some candidate parameterizations Point-on-plane + direction y L ( x, y, , f ) x • convenient for measuring luminaires
![More parameterizations Chords of a sphere L ( 1, f 1, 2, f 2 More parameterizations Chords of a sphere L ( 1, f 1, 2, f 2](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-32.jpg)
More parameterizations Chords of a sphere L ( 1, f 1, 2, f 2 ) • convenient for spherical gantry • facilitates uniform sampling
![Two planes (“light slab”) L ( u, v, s, t ) • uses projective Two planes (“light slab”) L ( u, v, s, t ) • uses projective](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-33.jpg)
Two planes (“light slab”) L ( u, v, s, t ) • uses projective geometry – fast incremental display algorithms
![Creating a light field • off-axis (sheared) perspective views Ó 2004 Marc Levoy Creating a light field • off-axis (sheared) perspective views Ó 2004 Marc Levoy](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-34.jpg)
Creating a light field • off-axis (sheared) perspective views Ó 2004 Marc Levoy
![A light field is an array of images Ó 2004 Marc Levoy A light field is an array of images Ó 2004 Marc Levoy](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-35.jpg)
A light field is an array of images Ó 2004 Marc Levoy
![Displaying a light field foreach x, y compute u, v, s, t I(x, y) Displaying a light field foreach x, y compute u, v, s, t I(x, y)](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-36.jpg)
Displaying a light field foreach x, y compute u, v, s, t I(x, y) = L(u, v, s, t) Ó 2004 Marc Levoy
![Devices for capturing light fields: Stanford Multi-Camera Array • cameras closely packed – high-X Devices for capturing light fields: Stanford Multi-Camera Array • cameras closely packed – high-X](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-37.jpg)
Devices for capturing light fields: Stanford Multi-Camera Array • cameras closely packed – high-X imaging – synthetic aperture photography • cameras widely spaced – video light fields – new computer vision algorithms Ó 2004 Marc Levoy
![The BRDF kaleidoscope [Han et al. , SIGGRAPH 2003] • discrete number of views The BRDF kaleidoscope [Han et al. , SIGGRAPH 2003] • discrete number of views](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-38.jpg)
The BRDF kaleidoscope [Han et al. , SIGGRAPH 2003] • discrete number of views • hard to capture grazing angles • uniformity? Ó 2004 Marc Levoy
![Light field morphing [Zhang et al. , SIGGRAPH 2002] UI for specifying feature polygons Light field morphing [Zhang et al. , SIGGRAPH 2002] UI for specifying feature polygons](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-39.jpg)
Light field morphing [Zhang et al. , SIGGRAPH 2002] UI for specifying feature polygons and their correspondences sample morph • feature correspondences = 3 D model Ó 2004 Marc Levoy
![Autostereoscopic display of light fields [Isaksen et al. , SIGGRAPH 2000] • • image Autostereoscopic display of light fields [Isaksen et al. , SIGGRAPH 2000] • • image](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-40.jpg)
Autostereoscopic display of light fields [Isaksen et al. , SIGGRAPH 2000] • • image is at focal distance of lenslet collimated rays spatial resolution ~ # of lenslets in the array angular resolution ~ # of pixels behind each lenslet each eye sees a different sets of pixels stereo Ó 2004 Marc Levoy
![End-to-end 3 D television [Matusik et al. , SIGGRAPH 2005] • • 16 cameras, End-to-end 3 D television [Matusik et al. , SIGGRAPH 2005] • • 16 cameras,](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-41.jpg)
End-to-end 3 D television [Matusik et al. , SIGGRAPH 2005] • • 16 cameras, 16 video projectors, lenticular lens array spatial resolution ~ # of pixels in a camera and projector angular resolution ~ # of cameras and projectors horizontal parallax only Ó 2004 Marc Levoy
![Why didn’t IBR take over the world? • warping and rendering range images is Why didn’t IBR take over the world? • warping and rendering range images is](http://slidetodoc.com/presentation_image_h/de1dcc73cb753f77c04e98a566e0c6e6/image-42.jpg)
Why didn’t IBR take over the world? • warping and rendering range images is slow – pixel-sized triangles are inefficient – just as many pixels need to be touched as in normal rendering • arms race against improvements in 3 D rendering – – level of detail (LOD) culling techniques hierarchical Z-buffer etc. • visual artifacts are objectionable – not small and homogeneous like 3 D rendering artifacts Ó 2004 Marc Levoy
- Slides: 42