Using Photographs to Enhance Videos of a Static

Using Photographs to Enhance Videos of a Static Scene Pravin Bhat 1, C. Lawrence Zitnick 2, Noah Snavely 1, Aseem Agarwala 3, Maneesh Agrawala 4, Michael Cohen 1, 2, Brian Curless 1, Sing Bing Kang 2 University of Washington 1, Microsoft Research Redmond 2 University of California 3, Adobe Systems 4 EGSR 2007

An overview of Spacetime Fusion

Motivation • Low quality video Input Video

Motivation • Low quality video • Reconstructed video Input Video Reconstructed Video

Motivation • Low quality video • Reconstructed video – Reconstructed from photos – Good spatial reconstruction – Bad temporal reconstruction Input Video Reconstructed Video

Motivation • Spacetime Fusion result Input Video Spacetime Fusion Result

Motivation • Spacetime Fusion result – Spatial properties of reconstruction – Temporal properties of input video Input Video Spacetime Fusion Result

Spacetime Fusion • Define a 3 D gradient field

Spacetime Fusion • Define a 3 D gradient field – Spatial gradients from reconstruction – Temporal gradients from input video

Spacetime Fusion • Define a 3 D gradient field – Spatial gradients from reconstruction – Temporal gradients from input video – Key Idea • Temporal gradients defined between motion compensated temporal neighbors

Spacetime Fusion • Define a 3 D gradient field – Spatial gradients from reconstruction – Temporal gradients from input video – Key Idea • Temporal gradients defined between motion compensated temporal neighbors Video frame: t - 1

Spacetime Fusion • Define a 3 D gradient field – Spatial gradients from reconstruction – Temporal gradients from input video – Key Idea • Temporal gradients defined between motion compensated temporal neighbors Gt Video frame: t - 1 Gt(x, y, t) = V(x, y, t) - V(x, y, t - 1)

Spacetime Fusion • Define a 3 D gradient field – Spatial gradients from reconstruction – Temporal gradients from input video – Key Idea • Temporal gradients defined between motion compensated temporal neighbors Gt Video frame: t - 1 Gt(x, y, t) = V(x, y, t) - V(x - u, y - v, t - 1)

Spacetime Fusion • Define a 3 D gradient field – Spatial gradients from reconstruction – Temporal gradients from input video – Key Idea • Temporal gradients defined between motion compensated temporal neighbors • Increases compatibility between temporal gradients and spatial gradients

Spacetime Fusion • Define a 3 D gradient field – Spatial gradients from reconstruction – Temporal gradients from input video – Key Idea • Temporal gradients defined between motion compensated temporal neighbors • Increases compatibility between temporal gradients and spatial gradients • Integrate the 3 D gradient field

Spacetime Fusion • Integrating the gradient field Solve linear system: Av = b

Spacetime Fusion • Integrating the gradient field Solve linear system: Av = b Constraints: vx, y, t – vx-1, y, t = Gx(x, y, t) vx, y, t – vx, y-1, t = Gy(x, y, t) vx, y, t – vx-u, y-v, t = Gt(x, y, t)

Applications

Enhanced Exposure

Edit Propagation Input Video

Edit Propagation User Edits

Edit Propagation User Edits

Edit Propagation User Edits

Edit Propagation User Edits

Edit Propagation User Edits

Edit Propagation User Edits

Edit Propagation Edited Video

Super-Resolution

Conclusion • Spacetime fusion

Conclusion • Spacetime fusion – Combines spatial and temporal gradients from two different sources

Conclusion • Spacetime fusion – Combines spatial and temporal gradients from two different sources – Requires motion vectors for temporal source • stereo (static scenes) • flow (dynamic scenes)

Conclusion • Spacetime fusion – Combines spatial and temporal gradients from two different sources – Requires motion vectors for temporal source • stereo (static scenes) • flow (dynamic scenes) – Major applications • Enforcing temporal coherence • Transferring lighting information