Texture Optimization for Example based Synthesis Vivek Kwatra
Texture Optimization for Example -based Synthesis Vivek Kwatra 1, 2 Irfan Essa 1 Aaron Bobick 1 Nipun Kwatra 1 1 GVU Center/College of Computing, Georgia 2 Computer Science Dept. , UNC Chapel Hill Tech
Motivation + Input Texture = Input Flow Field Output Sequence
Example-based Texture Synthesis
Example-based Texture Synthesis 1. Dynamic texturing (animation/video) 2. Greater control: external knobs, e. g. , flow, shape, illumination 3. Explicit texture quality evaluation and refinement
Contributions I • Global Optimization for texture synthesis – Global texture quality metric – Iterative multi-scale optimization
Contributions II • Flow-guided texture animation – Flow incorporated into optimization – Applications: fluid rendering, flow visualization + Input Texture = Input Flow Field Output Sequence
Related Work (a selected sample) Local vs. Global MRF Annealing [Paget’ 98] Texture Optimization [Kwatra’ 05] Histogram Matching [Heeger’ 95] Graphcut Textures [Kwatra’ 03] Non-parametric sampling Image Quilting [Efros’ 99, Wei’ 00] [Efros’ 01] Pixel vs. Patch
Related Work (a selected sample) Better Quality Local vs. Global MRF Annealing [Paget’ 98] Texture Optimization [Kwatra’ 05] Histogram Matching [Heeger’ 95] Graphcut Textures [Kwatra’ 03] Non-parametric sampling Image Quilting [Efros’ 99, Wei’ 00] [Efros’ 01] Pixel vs. Patch
Related Work (a selected sample) Better Quality Local vs. Global MRF Annealing [Paget’ 98] Texture Optimization [Kwatra’ 05] Histogram Matching Greater Controllability [Heeger’ 95] Graphcut Textures [Kwatra’ 03] Non-parametric sampling Image Quilting [Efros’ 99, Wei’ 00] [Efros’ 01] Pixel vs. Patch
Related Work • Flow-guided Synthesis – Advected Textures [Neyret’ 03] – Flow-based Video Editing [Bhat’ 04] • Optimization – Image-based Priors [Fitzgibbon’ 03] – Space-time Video Completion [Wexler’ 04]
Flow-guided Texture Animation • Flow consistency – Perceived motion similar to flow Source Texture Similarity • Texture similarity – Shape, size, orientation of texture elements similar to source Flow Consistency Flowing Target Flow
Naive Approach • Ignore Texture Similarity Source Texture Similarity • Warp each frame via flow field Target Frames X 0 Warp X 1 Flow Consistency Flowing Target Flow Xn
Naive Approach Texture structure not maintained Source Texture Similarity Flow Consistency Flowing Target Flow
Instead of… Warp Target Frames X 0 Warp X 1 Xn
Instead of… Warp Target Frames X 0 Warp X 1 Xn
Our Approach Warp + Correct Target Frames Synthesized Frame X– X Warp Correct W Warped Frame
“Correct” Operation Link to original goals Target Frames X– X Warp W Correct
“Correct” Operation Link to original goals Target Frames X– X Correct Flow Consistency Warp W
“Correct” Operation Link to original goals Source Texture Similarity Target Frames X– X Correct Flow Consistency Warp W
Optimization of Energy / Cost Energy = Flow Energy + Texture Energy Source Texture Similarity Target Frames X– X Correct Flow Consistency Warp W
Flow Energy Target Frames X– X Warp Flow Energy W
Texture Energy • Tricky! – Pixel-by-pixel comparison of source and target not possible • Compare texture “elements” – Local pixel neighborhoods • Want each target neighborhood to be similar to some source neighborhood
Texture Energy Z (source) X (target frame)
Energy for Single Neighborhood Z (source) p (pixel) X (target frame)
Energy for Single Neighborhood Z (source) Xp (neighborhood) X (target frame)
Energy for Single Neighborhood Z (source) Xp (neighborhood) X (target frame)
Energy for Single Neighborhood (nearest neighbor) Zp Z (source) Xp (neighborhood) X (target frame)
Energy for Single Neighborhood (nearest neighbor) Zp Z Texture Energy (single neighborhood) Xp (neighborhood) X
Texture Energy for Entire Image Z X
Texture Energy for Entire Image Z X = individual neighborhood energy
Optimization • Optimize Total Energy of target frame: • Initialize: X W (target frame warped frame) • Iteratively improve target frame
Optimization: Intuition Zp Texture Force Xp Source Target Frame Flow Force Wp Warped Frame
Optimization: Intuition Zp Blend Xp Source Target Frame Wp Warped Frame
Overlapping Neighborhoods Zp Zq Blend Source Xq Xp Target Frame Wp Warped Frame
We are not done yet! Zp Nearest neighbor may change Xp Source Target Frame Wp Warped Frame
Iterative Algorithm • Step 1 – Find Nearest Source Neighborhoods • Step 2 – Blend Source & Warped Neighborhoods • Repeat until convergence
Iterative Algorithm • Step 1 – Find Nearest Source Neighborhoods • Minimize E( X ) w. r. t {Zp} • Approximate nearest-neighbors – Hierarchical Clustering [Johnson’ 67] – Tree-Structured Vector Quantization [Wei’ 00]
Iterative Algorithm • Step 2 – Blend Source & Warped Neighborhoods • Minimize E( X ) w. r. t X • Set Solve Linear System
Robust Optimization • Quadratic Texture Energy Et sensitive to outlier neighborhoods • Non-quadratic Et more robust – Et( X ) = p k. Xp – Zpkr, 0<r<2
Robust Optimization • Quadratic Texture Energy Et sensitive to outlier neighborhoods • Non-quadratic Et more robust – Et( X ) = p k. Xp – Zpkr, 0<r<2 • Solve weighted quadratic at each iteration Et( X ) = p k. Xp – Zpkr-2. k. Xp – Zpk 2 = p p . k. Xp – Zpk 2
Minimize Texture Energy Alone • Novel Static Texture Synthesis Algorithm Source Texture Similarity X– X Correct Flow Consistency W
Texture Energy Graph • Random Initialization • Multiple – Resolution Levels – Neighborhood Sizes • Progressively refined output
Texture Energy Graph • Random Initialization • Multiple – Resolution Levels – Neighborhood Sizes • Progressively refined output
Results: Texture Synthesis
Results: Texture Synthesis Input Wei-Levoy [Wei’ 00] Image Quilting [Efros’ 01] Graph-cuts [Kwatra’ 03] Texture Optimization [Kwatra’ 05]
Results: Texture Synthesis Image Quilting Graph-cuts Texture Optimization [Efros’ 01] [Kwatra’ 03] [Kwatra’ 05]
Results: Flow-guided Synthesis
Results: Timing • Static texture synthesis – 5 -10 iterations per resolution level Resolution One iteration Total time 64 x 64 128 x 128 256 x 256 2. 5 sec 10 sec 25 sec 20 sec 2 min 8. 5 min • Flowing texture synthesis – 20 -60 seconds per frame
Summary • Global Optimization for controllable texture synthesis • Explicit evaluation of texture quality • Iterative progressive refinement of texture • Flow-guided texture animation within optimization framework + Input Texture = Input Flow Field Output Sequence
Limitations and Future Work • Optimization finds local minima – Blurry / misaligned texture elements • Texture scale should be compatible with flow field • Extensions – Controllable video textures – Other control criteria like shape, illumination
Acknowledgements • Eugene Zhang for flow design code • Greg Turk, Gabriel Brostow, Delphine Nain, Ravi Ruddarraju, Gaurav Chanda, Stephanie Brubaker, Mitch Parry, Pei Yen • Computational Perception Lab, Geometry Group @ Georgia Tech • CMU Graphics Lab Web page: http: //www. cc. gatech. edu/cpl/projects/textureoptimization/
- Slides: 52