Creating images the 2 D way JeanFranois Lalonde

Creating images the 2 -D way Jean-François Lalonde April 20, 2010

Creating images (3 -D)

Creating images (2 -D + 3 -D)

Inserting objects into images

Inserting objects into images(2 -D + 3 -D) [Debevec, ’ 98]

Inserting objects in images Highly detailed geometry Highly detailed materials Very expensive Realistic renderings Expensive and impractical [Debevec, ’ 98]

Alternative: Clip art Easy, intuitive, cheap Not realistic

Creating images (2 -D) Photo-realistic Image-based rendering Cartoon Photo Clip Art ? Expensive and impractical Cheap and intuitive

“Photoshop-ing” Composite by David Dewey

Inserting objects into images

Challenges Insert THIS object: impossible! object orientation scene illumination

The use of data Insert SOME object: much easier!

The Google model Database Query Sort the objects Results

2 -D image vs 3 -D scene

Outline Phase I: Database annotation Name: person Subgroup: person, standing Height: 1. 5 m Local context: in shadow Illumination: sunny, bright day, no cloud Segmentation quality: excellent, >40 points Upsampling blur: low Phase II: Object insertion

Data source: Label. Me Online (http: //labelme. csail. mit. edu), usercontributed 170, 000 objects in 40, 000 images Polygons and names [Russell et al. , 2005

Data organization Top-level categories (chosen manually, 16 total) Second-level categories (from annotations or clustering)

Annotating the objects

Camera parameters Assume flat ground plane all objects on ground camera roll is negligible (consider pitch only) Camera parameters: height and orientation

Camera parameters Human height distribution 1. 7 +/- 0. 085 m (National Center for Health Statistics) Car height distribution 1. 5 +/- 0. 19 m (automatically learned)

Object heights Database image Pixel heights Real heights

Estimated object heights Car Man Woman 1. 5 m Parking meter Fire hydrant 1. 0 m 0. 5 m Object Estimated average height (m) Car 1. 51 Man 1. 80 Woman 1. 67 Parking meter 1. 36 Fire hydrant 0. 87

Geometry is not enough

Illumination context Exact environment map is impossible Approximations [Khan et al. , ‘ 06] Database image Environment map rough approximation

Illumination context Database image Automatic Photo Popup Hoiem et al. , SIGGRAPH ‘ 05 P(pixel|class) CIE L*a*b* histograms

Illumination nearest-neighbors

Other criteria: local context

Other criteria: segmentation Label. Me contributors not always reliable Segmentation quality 38 points / polygon 4 points / polygon

Other criteria: blur Resolution: avoid up-sampling x 3 up-sampling

Recap Phase I: Database annotation Object properties (used for sorting the database) Label Cluster 3 -D height Camera Illumination context Local context Segmentation Phase II: Object insertion Blur

Let’s insert an object! Poor user-provided segmentations Noticeable seams

Seams Input Destination image Result Visible seam! [Perez et al. , 2003]

Poisson blending: idea Input Destination Enforce boundary color (seamless result) Enforce same gradient than input Result [Perez et al. , 2003]

Why gradients? 1 -D example bright dark Regular blending

1 -D example Original signals Derivatives Blending derivatives Reintegration results

1 -D example Intensity domain Gradient domain ?

2 -D: not so easy +1 4 5 -2 -3 0 2 +2 Non integrable: sum over a loop ≠ 0 Actually happens all the time in practice

2 -D: some notation Finite differences

2 -D: a (possible) solution ? Least-squares solution:

2 -D: a (popular) solution Solution: Poisson equation Popular because: Solution is obtained by solving a linear system of equations Can be solved (somewhat) efficiently ‘’ in matlab FFT Multi-grid solvers (approximate, but really

Results & limitations Image editing Some limitations Images need to be very well aligned Differences in background “bleed through” Images from [Perez et al. , 2003]

Poisson blending: improvements Drag-and-Drop Pasting [Jia et al. , 2006] Color bleeding User-selected boundary Poisson blending Refined boundary Poisson blending Images from [Jia et al. , 2006]

Still not right! Not so sensitive to shadow direction [Cavanagh, 2005]

Shadow transfer Database image Shadow estimate Refined shadow Object alone Shadow alone Object with shadow + =

User interface

Street accident

Bridge

Painting

Alley

Failure cases Porous objects Shadow transfer

Failure cases Best matching objects

Pros & cons 3 -D Pros Cons More control Complex! (camera, geometry, lighting. . . ) 2 -D + 3 -D Realistic 2 -D Realistic (photoshop) 2 -D (automatic) Easy, intuitive Complex! Need access to scene Complex! Need to sort through images “Generic” objects

Thanks! jlalonde@cs. cmu. edu