Image Based Modeling and Rendering PI Malik Capture

Image Based Modeling and Rendering (PI: Malik) • Capture photorealistic models of real world scenes that can support interaction – Vary viewpoint – Vary lighting – Vary scene configuration • This objective was achieved by combining and inventing new techniques in computer vision and graphics.

Image-based Modeling and Rendering • 1 st Generation---- vary viewpoint but not lighting – Acquire photographs – Recover geometry (explicit or implicit) – Texture map

Image-based Modeling and Rendering • Photographs are not Reflectance Maps ! Illumination Radiance Reflectance • 2 nd Generation---- vary viewpoint and lighting for non-diffuse scenes – Recover geometry – Recover reflectance properties – Render using light transport simulation

Image-based Modeling • 2 nd Generation---- vary viewpoint and lighting – Recover geometry & reflectance properties – Render using light transport simulation or local shading Original Lighting & Viewpoint Novel Lighting & Viewpoint

Inverse Global Illumination (Yu et al, SIGGRAPH 99) Reflectance Properties Radiance Maps Geometry Light Sources

Real vs. Synthetic

Real vs. Synthetic

• Key contributions in Inverse Illumination A digital. Global camera can undertake all the data acquisition tasks involved. • Both specular and high resolution diffuse reflectance properties can be recovered from photographs. • Reflectance recovery can re-render non-diffuse real scenes under novel illumination as well as from novel viewpoints.

Image-based Modeling and Rendering • 3 rd Generation--Vary spatial configurations in addition to viewpoint and lighting Novel Viewpoint & Configuration

Framework • Input – Multiple range scans of a scene – Multiple photographs of the same scene • Output – Geometric meshes of each object in the scene – Registered texture maps for objects

Segmentation Results [Yu, Ferencz and Malik TVCG ’ 01]

Models of Individual Objects

Texture-Mapping and Object Manipulation

Transitions • Façade has inspired several image-based modeling packages including Canoma from Metacreations. • Technique for registering range and radiance images being used by Cyra, a leading time of flight laser scanner company.

Continuing Challenges • Finding correspondences between widely separated camera views automatically • Models of reflectance and texture for natural materials and objects