Texture Optimization for Example based Synthesis Vivek Kwatra

Texture Optimization for Example -based Synthesis Vivek Kwatra 1, 2 Irfan Essa 1 Aaron Bobick 1 Nipun Kwatra 1 1 GVU Center/College of Computing, Georgia 2 Computer Science Dept. , UNC Chapel Hill Tech

Motivation + Input Texture = Input Flow Field Output Sequence

Example-based Texture Synthesis

Example-based Texture Synthesis 1. Dynamic texturing (animation/video) 2. Greater control: external knobs, e. g. , flow, shape, illumination 3. Explicit texture quality evaluation and refinement

Contributions I • Global Optimization for texture synthesis – Global texture quality metric – Iterative multi-scale optimization

Contributions II • Flow-guided texture animation – Flow incorporated into optimization – Applications: fluid rendering, flow visualization + Input Texture = Input Flow Field Output Sequence

Related Work (a selected sample) Local vs. Global MRF Annealing [Paget’ 98] Texture Optimization [Kwatra’ 05] Histogram Matching [Heeger’ 95] Graphcut Textures [Kwatra’ 03] Non-parametric sampling Image Quilting [Efros’ 99, Wei’ 00] [Efros’ 01] Pixel vs. Patch

Related Work (a selected sample) Better Quality Local vs. Global MRF Annealing [Paget’ 98] Texture Optimization [Kwatra’ 05] Histogram Matching [Heeger’ 95] Graphcut Textures [Kwatra’ 03] Non-parametric sampling Image Quilting [Efros’ 99, Wei’ 00] [Efros’ 01] Pixel vs. Patch

Related Work (a selected sample) Better Quality Local vs. Global MRF Annealing [Paget’ 98] Texture Optimization [Kwatra’ 05] Histogram Matching Greater Controllability [Heeger’ 95] Graphcut Textures [Kwatra’ 03] Non-parametric sampling Image Quilting [Efros’ 99, Wei’ 00] [Efros’ 01] Pixel vs. Patch

Related Work • Flow-guided Synthesis – Advected Textures [Neyret’ 03] – Flow-based Video Editing [Bhat’ 04] • Optimization – Image-based Priors [Fitzgibbon’ 03] – Space-time Video Completion [Wexler’ 04]

Flow-guided Texture Animation • Flow consistency – Perceived motion similar to flow Source Texture Similarity • Texture similarity – Shape, size, orientation of texture elements similar to source Flow Consistency Flowing Target Flow

Naive Approach • Ignore Texture Similarity Source Texture Similarity • Warp each frame via flow field Target Frames X 0 Warp X 1 Flow Consistency Flowing Target Flow Xn

Naive Approach Texture structure not maintained Source Texture Similarity Flow Consistency Flowing Target Flow

Instead of… Warp Target Frames X 0 Warp X 1 Xn

Our Approach Warp + Correct Target Frames Synthesized Frame X– X Warp Correct W Warped Frame

“Correct” Operation Link to original goals Target Frames X– X Warp W Correct

“Correct” Operation Link to original goals Target Frames X– X Correct Flow Consistency Warp W

“Correct” Operation Link to original goals Source Texture Similarity Target Frames X– X Correct Flow Consistency Warp W

Optimization of Energy / Cost Energy = Flow Energy + Texture Energy Source Texture Similarity Target Frames X– X Correct Flow Consistency Warp W

Flow Energy Target Frames X– X Warp Flow Energy W

Texture Energy • Tricky! – Pixel-by-pixel comparison of source and target not possible • Compare texture “elements” – Local pixel neighborhoods • Want each target neighborhood to be similar to some source neighborhood

Texture Energy Z (source) X (target frame)

Energy for Single Neighborhood Z (source) p (pixel) X (target frame)

Energy for Single Neighborhood Z (source) Xp (neighborhood) X (target frame)

Energy for Single Neighborhood (nearest neighbor) Zp Z (source) Xp (neighborhood) X (target frame)

Energy for Single Neighborhood (nearest neighbor) Zp Z Texture Energy (single neighborhood) Xp (neighborhood) X

Texture Energy for Entire Image Z X

Texture Energy for Entire Image Z X = individual neighborhood energy

Optimization • Optimize Total Energy of target frame: • Initialize: X W (target frame warped frame) • Iteratively improve target frame

Optimization: Intuition Zp Texture Force Xp Source Target Frame Flow Force Wp Warped Frame

Optimization: Intuition Zp Blend Xp Source Target Frame Wp Warped Frame

Overlapping Neighborhoods Zp Zq Blend Source Xq Xp Target Frame Wp Warped Frame

We are not done yet! Zp Nearest neighbor may change Xp Source Target Frame Wp Warped Frame

Iterative Algorithm • Step 1 – Find Nearest Source Neighborhoods • Step 2 – Blend Source & Warped Neighborhoods • Repeat until convergence

Iterative Algorithm • Step 1 – Find Nearest Source Neighborhoods • Minimize E( X ) w. r. t {Zp} • Approximate nearest-neighbors – Hierarchical Clustering [Johnson’ 67] – Tree-Structured Vector Quantization [Wei’ 00]

Iterative Algorithm • Step 2 – Blend Source & Warped Neighborhoods • Minimize E( X ) w. r. t X • Set Solve Linear System

Robust Optimization • Quadratic Texture Energy Et sensitive to outlier neighborhoods • Non-quadratic Et more robust – Et( X ) = p k. Xp – Zpkr, 0<r<2

Robust Optimization • Quadratic Texture Energy Et sensitive to outlier neighborhoods • Non-quadratic Et more robust – Et( X ) = p k. Xp – Zpkr, 0<r<2 • Solve weighted quadratic at each iteration Et( X ) = p k. Xp – Zpkr-2. k. Xp – Zpk 2 = p p . k. Xp – Zpk 2

Minimize Texture Energy Alone • Novel Static Texture Synthesis Algorithm Source Texture Similarity X– X Correct Flow Consistency W

Texture Energy Graph • Random Initialization • Multiple – Resolution Levels – Neighborhood Sizes • Progressively refined output

Results: Texture Synthesis

Results: Texture Synthesis Input Wei-Levoy [Wei’ 00] Image Quilting [Efros’ 01] Graph-cuts [Kwatra’ 03] Texture Optimization [Kwatra’ 05]

Results: Texture Synthesis Image Quilting Graph-cuts Texture Optimization [Efros’ 01] [Kwatra’ 03] [Kwatra’ 05]

Results: Flow-guided Synthesis

Results: Timing • Static texture synthesis – 5 -10 iterations per resolution level Resolution One iteration Total time 64 x 64 128 x 128 256 x 256 2. 5 sec 10 sec 25 sec 20 sec 2 min 8. 5 min • Flowing texture synthesis – 20 -60 seconds per frame

Summary • Global Optimization for controllable texture synthesis • Explicit evaluation of texture quality • Iterative progressive refinement of texture • Flow-guided texture animation within optimization framework + Input Texture = Input Flow Field Output Sequence

Limitations and Future Work • Optimization finds local minima – Blurry / misaligned texture elements • Texture scale should be compatible with flow field • Extensions – Controllable video textures – Other control criteria like shape, illumination

Acknowledgements • Eugene Zhang for flow design code • Greg Turk, Gabriel Brostow, Delphine Nain, Ravi Ruddarraju, Gaurav Chanda, Stephanie Brubaker, Mitch Parry, Pei Yen • Computational Perception Lab, Geometry Group @ Georgia Tech • CMU Graphics Lab Web page: http: //www. cc. gatech. edu/cpl/projects/textureoptimization/