Imagebased rendering Michael F Cohen Microsoft Research Computer

Image-based rendering Michael F. Cohen Microsoft Research

Computer Graphics Output Image Synthetic Camera Model

Computer Vision Output Model Real Scene Real Cameras

Combined Output Image Synthetic Camera Model Real Scene Real Cameras

But, vision technology falls short Output Image Synthetic Camera Model Real Scene Real Cameras

… and so does graphics. Output Image Synthetic Camera Model Real Scene Real Cameras

Image Based Rendering Output Image Synthetic Real Scene Camera Images+Model Real Cameras -or. Expensive Image Synthesis

Ray q Constant radiance • time is fixed q 5 D • 3 D position • 2 D direction

All Rays q Plenoptic Function • all possible images • too much stuff!

Line q Infinite line q 4 D • 2 D direction • 2 D position

Ray q Discretize q Distance between 2 rays • Which is closer together?

Image q What is an image? q All rays through a point • Panorama?

Image q 2 D • position of rays has been fixed • direction remains

Image q Image plane q 2 D • position

Object q Light leaving towards “eye” q 2 D • just dual of image

Object q All light leaving object

Object q 4 D • 2 D position • 2 D direction

Object q All images

Lumigraph q How to • organize • capture • render

Lumigraph - Organization 2 D position q 2 D direction q q s

Lumigraph - Organization 2 D position q s q 2 plane parameterization u

Lumigraph - Organization 2 D position q s, t t u, v s, t v u, v q 2 plane parameterization s u

Lumigraph - Organization Hold s, t constant q Let u, v vary q An image q s, t u, v

Lumigraph - Organization q Discretization • higher res near object • if diffuse • captures texture • lower res away • captures directions s, t u, v

Lumigraph - Capture q Idea 1 • Move camera carefully over s, t plane • Gantry • see Lightfield paper s, t u, v

Lumigraph - Capture q Idea 2 • Move camera anywhere • Rebinning • see Lumigraph paper s, t u, v

Lumigraph - Rendering q For each output pixel • determine s, t, u, v • either • find closest discrete RGB • interpolate near values s, t u, v

Lumigraph - Rendering q For each output pixel • determine s, t, u, v • either • use closest discrete RGB • interpolate near values s u

Lumigraph - Rendering q Nearest • closest s • closest u • draw it q Blend 16 nearest • quadrilinear interpolation s u

High-Quality Video View Interpolation Using a Layered Representation Larry Zitnick Sing Bing Kang Matt Uyttendaele Simon Winder Rick Szeliski Interactive Visual Media Group Microsoft Research

Current practice free viewpoint video Many cameras vs. Motion Jitter

Video view interpolation Fewer cameras and Smooth Motion Automatic Real-time rendering

Prior work: IBR (static) Plenoptic Modeling Mc. Millan & Bishop, SIGGRAPH ‘ 95 Light Field Rendering Levoy & Hanrahan, SIGGRAPH ‘ 96 The Lumigraph Gortler et al. , SIGGRAPH ‘ 96 Concentric Mosaics Shum & He, SIGGRAPH ‘ 99

Prior work: IBR (dynamic) Stanford Multi-Camera Array Project Virtualized Reality. TM Kanade et al. , IEEE Multimedia ‘ 97 Image-Based Visual Hulls Matusik et al. , SIGGRAPH ‘ 00 Dynamic Light Fields Goldlucke et al. , VMV ‘ 02 Free-viewpoint Video of Humans Carranza et al. , SIGGRAPH ‘ 03 3 D TV Matusik & Pfister, SIGGRAPH ‘ 04

System overview Video Capture OFFLINE Stereo Representation Compression File ONLINE Selective Decompression Render

cameras hard disks concentrators controlling laptop

Calibration Zhengyou Zhang, 2000

Input videos

Key to view interpolation: Geometry Stereo Geometry Image 1 Image 2 Camera 1 Camera 2 Virtual Camera

Image correspondence Image 1 Image 2 Leg Correct Wall Good Incorrect Bad Match Score

Local matching Image 1 Image 2 Low texture

Global regularization A Create MRF (Markov Random Field): Image 1 Image 2 E B P A F C Q A S D color. A ≈ color. B → z. A ≈ z. B Each segment is a node R T U z. A ≈ z. Pof, zstates Number Q, z S = number of depth levels

Iteratively solve MRF

Depth through time

Matting Background Surfacematting Interpolated view without Foreground Surface Background Strip Width Foreground Bayesian Matting Chuang et al. 2001 Camera Background Alpha Foreground

Rendering with matting No Matting

Representation Main Background Boundary Strip Width Foreground Main Layer: Boundary Layer: Color Alpha Depth

“Massive Arabesque” videoclip