Detail Preserving Shape Deformation in Image Editing SIGGRAPH

Detail Preserving Shape Deformation in Image Editing SIGGRAPH 2007 Hui Fang and John C. Hart

Abstract � We propose an image editing system ◦ Preserve its detail and orientation by resynthesizing texture from the source ◦ Patch-based texture synthesis that aligns texture features with image features

Introduction �A novel image editing system that allows a user to select and move one or more image feature curves ◦ Replacing any texture stretched by the deformation with texture resynthesized �Anisotropic feature-aligned texture synthesis step to preserve texture detail �Distortion to the texture coordinates for each patch to align the target image features �Graph. Cut textures [Kwatra et al. 2003]

Introduction �A new method that distorts the coordinates of patch ◦ Image Analogies [Hertzmann et al. 2001] can synthesize a texture to adhere to a given feature line �Yields more high-frequency noise unlike modern patchbased synthesis ◦ Image Quilting [Efros and Freeman 2001] could fill different silhouettes with a texture �Boundary patches appeared to repeat ◦ Feature matching and deformation for texture synthesis [Wu and Yu 2004] distorted neighboring patches to connect their feature lines �Not as global as what us did

Overview � Deformation ◦ Draw feature curves in the source image, and then move them to their desired destination positions � Curvilinear Coordinates ◦ Define curvilinear coordinates using curve tangent vectors & Euler integration � Textured Patch Generation ◦ A pair of curvilinear coordinate is generated ◦ Texture synthesis over the destination grid from source � Image Synthesis ◦ Finalize the synthesis via Graph. Cut

Deformation p'i(t) pi(t) D(p'f) = pf – p'f D(∂I’) = 0

Deformation Original Deformed

Curvilinear Coordinates p'i(t) T'

Curvilinear Coordinates � Since the parametrization of each feature curve is arbitrary, one can encounter global orientation inconsistencies ◦ Calculate separate tangent field for each curve then use only the field which is the closest � We integrate these diffused tangents to construct a local curvilinear coordinate system extending from any chosen “origin” pixel

Curvilinear Coordinates p'i(t) k j

Curvilinear Coordinates � Time-step ɛ=1 ◦ 30 ~ 40 pixels along spines (j direction) ◦ 15 ~ 30 pixels wide ribs (k direction) ◦ Two pixels short of nearby feature curve to prevent overlapping � Smooth the coordinates with several Laplacian iterations ◦ λ = 0. 7 ◦ Removes singularities and self-intersections that can occur ◦ Does not completely solve the problem (Not very noticeable)

Curvilinear Coordinates

Textured Patch Generation � Source origin q 0, 0 = D(q'0, 0) � Bilinear image filter to find the color at the source � Unit-radius cone filter centered at each destination to accumulate the synthesized texture ◦ Small reduction in the resolution of the resynthesized texture detail

Image Synthesis � Use Graph. Cut [Kwatra et al. 2003] ◦ Generate patches individually, using a priority queue to generate first patches whose origin pixel is closest to the feature curve and adjacent to a previously synthesized patch ◦ Generate a pool of candidate textured patches synthesized from source patches grown from origins randomly chosen from an 11× 11 pixel region surrounding the point D(q'0, 0) ◦ Choose one with the least overlapping difference with previously synthesized neighboring patches

Image Synthesis � Selected patch merges into destination via Graph. Cut � Use Poission Image Editing when the seam produces by Graph. Cut is unsatisfactory

Scale Adaptive Retexturing � The deformation field D can potentially compress a large source area into a small target area ◦ Cause blocky artifacts and seams ◦ Occur when the origin pixels of neighboring patches in the target map to positions in the source with different texture characteristics � Can be overcome by altering the texture synthesis sampling

Scale Adaptive Retexturing

Scale Adaptive Retexturing � We detect these potential problems with a (real) compression field C' ◦ Clamp the compression field to values in [1, 3] to limit its effect ◦ The “spine” length and “rib” breadth of patches are reduced by C'(x, y)

Scale Adaptive Retexturing

Results � Accelerated the construction of source feature curves by using portions of the segmentation boundary produced by Lazy Snapping [Li et al. 2004] ◦ Feature curves do not need to match feature contours exactly, as deformed features were often aligned by the texture search � Used the ordinary Laplacian deformation for interactive preview ◦ Denoted some feature curves as “passive” to aid texture orientation

Results � Filtering used for curvilinear grid resampling removes some of the high frequency detail ◦ Could be recovered by sharpening with histogram interpolation and matching [Matusik et al. 2005]

Results

Results

Results

Failure case

Results � Sharp image changes (like shading changes) should identified by feature curves ◦ Lack of feature curves will cause unrealistic discontinuities in the result � Poisson artifacts image editing hides some of these ◦ by softly blending the misaligned features

Results Measured on a 3. 40 GHz Pentium 4 CPU (31 x 31 search domain for beach)

Conclusion � Stretched texture details can be adequately recovered by a local retexturing around userdefined feature curves � Assumes that the orientation of texture detail of an image is related to the orientation of nearby feature curves � Matting can be used to eliminate unwanted artifacts (Fig. 5) � In practice the success of this approach depends primarily on the selection of the feature curves ◦ The most promising direction of future work in this topic would be to add the automatic detection and organization of image feature curves