Hybrid Texture Synthesis Andrew Nealen Marc Alexa Discrete

Hybrid Texture Synthesis Andrew Nealen Marc Alexa Discrete Geometric Modeling Group (DGM) Technische Universität Darmstadt Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

2 D Texture Synthesis nxm Input Texture Nx. M Output Texture ► The goal: Synthesize an output texture which is perceptually similar to the input texture. Also ensure that the result contains sufficient variation. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Our Inspiration ► Existing (and Impressive) Technology • Pixel-Based: ─ ─ Non-parametric Sampling [Efros and Leung 1999] Tree-structured Vector Quantization [Wei and Levoy 2000] Synthesizing Natural Textures [Ashikhmin 2001] Image Analogies [Hertzmann et al. 2001] • Patch-Based: ─ Patch-Based Sampling [Guo et al. 2001] ─ Image Quilting [Efros and Freeman 2001] ─ Hierarchical Pattern Mapping [Soler et al. 2002] Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Our Inspiration ► Possible • • Drawbacks ? Loss of global structure Loss of scale Boundary mismatch Blurring Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Our Inspiration ► Possible • • Drawbacks ? Loss of global structure Loss of scale Boundary mismatch Blurring Wei/Levoy Algorithm: Can occur when using structured Textures with rich histograms. This is due to the L 2 norm. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Our Inspiration ► Possible • • Drawbacks ? Loss of global structure Loss of scale Boundary mismatch Blurring Patch-Based Sampling Algorithm: Can occur when texture features are not well-aligned across patch boundaries Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Our Inspiration ► Possible • • Drawbacks ? Loss of global structure Loss of scale Boundary mismatch Blurring ► Is it possible to reduce all deficiencies for an even larger class of textures ? Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Pixel-Based Texture Synthesis Input Output Step n in Algorithm Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Pixel-Based Texture Synthesis Input Output Build L-shaped Neighborhood Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Pixel-Based Texture Synthesis Input Output Find Matching Neighborhood(s) in Input Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Pixel-Based Texture Synthesis Input Output Select Best Neighborhood from all Candidates Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Pixel-Based Texture Synthesis Input Output Copy Selected Pixel to Output Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Pixel-Based Texture Synthesis Input Output Repeat for all Pixels in this Row. . . Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Pixel-Based Texture Synthesis Input Output . . . and all other Output Rows Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Pixel-Based Texture Synthesis Main difference between Efros/Leung (EL), Wei/Levoy (WL) and Ashikhmin (AS): Per. Pixel Search Strategy. ► Ashikhmin’s insight: “WL tends to produce ► textures with smoothed out edges in some cases. ” Ashikhmin’s solution: consider only coherent pixels in the search, which encourages region growing. ► Problem: inconsistencies can still occur for highly-structured textures. ► Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Patch-Based Texture Synthesis Input Output Step n in Algorithm Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Patch-Based Texture Synthesis Input Output Patch to be Synthesized in this Step Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Patch-Based Texture Synthesis Input Output Build Neighborhood (Image Mask) Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Patch-Based Texture Synthesis Input Output Find Matching Neighborhood(s) in Input Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Patch-Based Texture Synthesis Input Output Select Best Neighborhood from all Candidates Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Patch-Based Texture Synthesis Input Output Copy Selected Patch to Output Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Patch-Based Texture Synthesis Input Output Apply some Overlap Repair Strategy. . . Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Patch-Based Texture Synthesis Input Output . . . And we are done with this patch Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Patch-Based Texture Synthesis Input Output Repeat for all Patches in Output Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Patch-Based Texture Synthesis Generalization of Pixel-Based Synthesis ► ► ► Main (visual) difference between Image Quilting (IQ) and Patch-based Sampling (PBS): Overlap Repair Strategy. IQ performs a minimum-error-boundarycut (MEBC) through the error surface (least cost path; Dijkstra). PBS feathers the overlap region (alphablending). Pro: patch-based algorithms are good at maintaining global structure. Problems: boundary mismatch (MEBC, top) and blurring (PBS, bottom). Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Hybrid Texture Synthesis (HTS) Algorithm ► Hybrid, two-fold approach to texture synthesis • Adaptive patch sampling [Soler et al. 2002] ─ Start with a uniform quadrilateral grid of patches (the output) ─ Recursively split a patch if the best fit taken from the input texture is not good enough (a user-defined tradeoff max) • Overlap re-synthesis (novel overlap repair strategy) ─ Mark invalid pixels in the overlap region ─ Validity map is defined by the error surface E(x)=(xsrc – xdst)2 and a user-defined tolerance parameter max ─ Define an ordering for the invalid pixels and re-synthesize them using a per-pixel strategy Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Adaptive Patch Sampling Principle Synthesize White Patch p > max Split Patch p < max Essentially: Hierarchical Pattern Mapping applied to the plane [Soler et al. 2002] ► Analogy: Assembling a quadtree puzzle ► Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Adaptive Patch Sampling Trade-off max = 0. 01 max = 1. 0 ► Trade-off max = 0. 04 between preserving global structure and avoiding detail artifacts Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

In Search of Good Patches Synthesizing a single patch Synthesize Black Patch i Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

In Search of Good Patches Extracting the image mask Synthesize Black Patch i ► Grow patch by overlap (toroidally) Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

In Search of Good Patches Extracting the image mask Synthesize Black Patch i Image Mask (Ii) ► Extract Binary Support (Ji) image mask (Ii) and binary support (Ji), and circularly shift to upper left corner Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

In Search of Good Patches Computing the error image Input Texture (T) Image Mask (Ii) ► Compute Binary Support (Ji) error Ei(x 0) between Ii and T for each circular shift x 0=( x, y) of the input texture T Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

In Search of Good Patches Computing the error image Input Texture (T) Error Image (Ei) Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

In Search of Good Patches Selecting a Patch Input Texture (T) Error Image (E) Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Selected Patch (P)

In Search of Good Patches Selecting a Patch Input Texture (T) Error Image (E) ► Note: Selected Patch (P) error image can be computed efficiently in the Fourier domain. Complexity: O(n log n) per patch. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Overlap Re-synthesis Overlap error and pixel invalidation Patch and Image Mask Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Selected Patch (P)

Overlap Re-synthesis Overlap error and pixel invalidation Image Mask (I) Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Selected Patch (P)

Overlap Re-synthesis Overlap error and pixel invalidation Image Mask (I) Error Surface (S) Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Selected Patch (P)

Overlap Re-synthesis Overlap error and pixel invalidation Patch and Image Mask Error Surface (S) P with Invalid Pixels (blue) user-defined max is the pixel error tolerance. Setting to 1 results in pure feathering. ► Trade-off: Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Overlap Re-synthesis Compositing P OVER Ri-1 Intermediate Result Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 Patch with Invalid Pixels

Overlap Re-synthesis Compositing Composited Result Patch with Invalid Pixels Simple scanline re-synthesis: possibly insufficient causal neighborhood within valid pixel region ► Solution: alternative ordering (Pixel Traversal Map) ► Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Overlap Re-synthesis Pixel Traversal Map (Ordering) Composited Result ► Pixel Traversal Map (M) Patch with Invalid Pixels Pixel Traversal Map: repeated morphological dilation of the binary support for valid pixels with a euclidian disk of radius 1 (city-block distance transform). Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Overlap Re-synthesis Pixel Traversal Map (Ordering) Composited Result ► Pixel Traversal Map (M) Patch with Invalid Pixels Traversal Map: step 1 Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Overlap Re-synthesis Pixel Traversal Map (Ordering) Composited Result ► Pixel Traversal Map (M) Patch with Invalid Pixels Traversal Map: step 2. . . Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Overlap Re-synthesis Pixel Traversal Map (Ordering) Composited Result ► Pixel Traversal Map (M) Patch with Invalid Pixels Traversal Map: . . . step n Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Overlap Re-synthesis Pixel Traversal Map (Ordering) Composited Result Pixel Traversal Map (M) Patch with Invalid Pixels ► Art Restorer Analogy: stepwise restoration of the hole from the boundary of the existing image. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Overlap Re-synthesis Per-Pixel Re-synthesis Intermediate Result ► Synthesize Pixel Traversal Map (M) red pixel from valid (or already resynthesized) pixels Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Overlap Re-synthesis Per-Pixel Re-synthesis Image Mask (Ij) Binary Support (Jj) Intermediate Result ► Extract image mask (Ij) and binary support (Jj) for best-pixel search in the input texture. Complexity: O(n log n) per pixel. Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Overlap Re-synthesis Per-Pixel Re-synthesis Intermediate Result Pixel Traversal Map (M) Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Overlap Re-synthesis Per-Pixel Re-synthesis Intermediate Result Shifted Pixel Traversal Map (M) Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Overlap Re-synthesis Per-Pixel Re-synthesis Intermediate Result with Traversal Map Pixel Traversal Map (M) Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Overlap Re-synthesis Per-Pixel Re-synthesis Overlap Re-synthesis Step 1 Pixel Traversal Map (M) Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Overlap Re-synthesis Per-Pixel Re-synthesis Overlap Re-synthesis Step 2 Pixel Traversal Map (M) Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Overlap Re-synthesis Per-Pixel Re-synthesis Overlap Re-synthesis Step 3 Pixel Traversal Map (M) Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Overlap Re-synthesis Per-Pixel Re-synthesis Overlap Re-synthesis Result Pixel Traversal Map (M) Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Results Overlap Repair Comparisons No Repair IQ PBS Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003 HTS

Results Synthesis Comparisons Input IQ Efros/Leung Wei/Levoy PBS HTS Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Results Synthesis Comparisons Input PBS HTS Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Results Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Conclusions and Future Work ► Improve Computational Complexity • Pixel neighborhoods of patch and pixel stage are not known a priori, so precomputation is not straightforward • We can solve this in the pixel stage by employing kcoherence search [Tong et al. 2002] [Ashikhmin 2001] ► Improve Error Metric • Still using the L 2 norm due to its simplicity • Develop a metric which takes feature mismatch into account • Texton approach looks promising [Zhang et al. 2003] Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Questions ? ► Contact Information Andrew Nealen andy@nealen. com Marc Alexa alexa@informatik. tu-darmstadt. de http: //www. dgm. informatik. tu-darmstadt. de Matlab code and extended Thesis: http: //www. nealen. com Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003