Numerical geometry of nonrigid shapes Nonrigid Correspondence Calculus

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 1 Of bodies changed to various forms, I sing. Ovid, Metamorphoses Non-Rigid Correspondence and Calculus of Shapes Alexander Bronstein, Michael Bronstein © 2008 All rights reserved. Web: tosca. cs. technion. ac. il

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 2 Does a “natural” correspondence exist?

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 3 Correspondence ‘‘ makes sense ‘‘ ‘‘ ‘‘ accurate Aesthetic Semantic ‘‘ beautiful ‘‘ Geometric

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 4 Correspondence n Correspondence is not a well-defined problem! Chances to solve it with geometric tools are slim. If objects are sufficiently similar, we have better chances. Correspondence between nonrigid deformations of the same object.

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 5 1 D motivation: correspondence between curves n Two curves , n Arclength parametrization n Unique up to initial point. n Reparametrize and to canonical parametrization. n Find correspondence between intervals n Correspondence between and

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 6 The curse of higher dimension n We relied on existence of a “canonical” arclength parametrization. n Was possible due to existence of total ordering of points in 1 D. n Surfaces (2 D objects) do not have a total ordering. n Hence, no analogy of arclength parametrization for surfaces. n We can still find an invariant parametrization.

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 7 Invariant parametrization

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 8 Invariant parametrization Ingredients: n Parametrization domain . n Group of deformations . n Shape . n Parametrization procedure, constructing given the shape . Desideratum: commutativity of the parametrization procedure with the deformation: How to construct such an invariant parametrization procedure?

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 9 Canonical forms, bis n Compute minimal distortion embeddings n Define intrinsic parametrizations n Find rigid motion between parametrizations n Define correspondence between shapes

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 10 Canonical forms n Embedding into the plane is not distortionless. n Invariance of parametrization holds only approximately n Generally, there exists no rigid motion bringing and into perfect correspondence. n Relax assumptions on bijection. : allow to be any

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 11 Image processing insight n Given two grayscale images and , find the optical flow (a. k. a. disparity map, motion field, etc. ) minimizing the error n Local image misalignment n Given two shapes , find n parametrized by and minimizing measures mismatch between and .

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 12 Image processing approach n Problem: functional n Make it intrinsic replacing n depends on parametrization with has also to be parametrization-independent. n Example: normal misalignment n Problem: not isometry-invariant. n Make an intrinsic quantity, e. g. , n Not limited to geometric quantities. n May include photometric information. . .

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 13 Healthy solution to ill-posed problems n Minimization problem is ill-posed! n Add a regularization term n Tikhonov n Total variation

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 14 Intrinsic regularization n Regularizer has to be parametrization-invariant. n Frobenius norm n is replaced by the Hilbert-Schmidt norm is an intrinsic quantity in parametrization coordinates is correspondence between shapes. is the intrinsic gradient on . is the norm in the tangent space of .

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 15 Physical insight n Cauchy-Green deformation tensor n Square of local change of distance due to elastic deformation. n measures average distance deformation. n = elastic energy (a. k. a. Dirichlet energy) of thin rubber sheet pressed against a mold .

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 16 Some harmony n Dirichlet energy n We have been looking for a regularizer… …but found a good measure for shape mismatch! n is an intrinsic quantity. n Minimizing gives a minimum deformation correspondence. n Minimizer is a harmonic map of to .

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 17 Minimum distortion correspondence n Define a general energy functional n is intrinsic, hence can be expressed in terms of the metric n Correspondence problem becomes n GMDS with generalized stress.

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 18 Minimum distortion correspondence

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 19 Generalized stress n “Harmonic stress”: gives the norm of the Cauchy-Green tensor n Our good old L 2 stress gives “as isometric as possible” correspondence.

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 20 Minimum distortion correspondence n Defined up to intrinsic symmetry of and .

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 21 Partial correspondence

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 22 ® MATLAB intermezzo Minimum distortion correspondence

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 23 Texture transfer Reference Transferred texture TIME

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 24 Virtual body painting

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 25 Texture substitution I’m Alice. I’m Bob. I’m Alice’s texture on Bob’s geometry

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 26 Calculus of shapes n Given two shapes and , and a correspondence . n We can define a convex combination of the two shapes as a new shape, where the extrinsic location of each point is given by Alternatively n Define deformation field into and express transforming. n We can create new shapes by adding or subtracting other shapes. n We have a calculus of shapes.

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 27 Calculus of shapes in shape space Extrapolation Interpolation

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 28 Temporal super-resolution TIME

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 29 Motion-compensated interpolation

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 30 Metamorphing 100% Alice 75% Alice 25% Bob 50% Alice 50% Bob 75% Alice 50% Bob 100% Bob

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 31 Face caricaturization EXAGGERATED EXPRESSION 0 1 1. 5

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 32 The quest for trajectory n In our definition of linear trajectory between corresponding points was used. n If and are extrinsically similar, this gives good result. n Generally, there is no guarantee that is a valid shape: n Not a manifold n Self-intersecting n Even if shape is valid, it is not necessarily isometric to .

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 33 The quest for trajectory n Given two shapes and , and a correspondence , we want to find intermediate shapes n For each point , define a trajectory . for such that The big question: n How to select trajectories? n No self-intersections of intermediate meshes. n No distortion of intrinsic geometry in intermediate meshes.

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 34 Deformation field n Define deformation field n Tangent to the trajectory n In order for intermediate shapes to be isometric to must hold for all and . ,

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 35 The Killing field n Killing field: deformation field preserving the metric. n Satisfies for all and n May not exist, even if and are isometric! n Remember: not every nonrigid shape is continuously bendable… Wilhelm Karl Joseph Killing (1847 -1923)

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 36 Metric for deformation fields n As isometric as possible deformation field. n Define inner product between deformation fields of n Induces a norm n Problem: vanishes for n Solution: add stiffening term: being a rigid motion.

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 37 As isometric as possible morph n We have a Riemannian metric on the space of shapes. n Find a minimal geodesic connecting between n Boundary conditions n Minimum deviation from Killing field along the path. n As isometric as possible morph. and . .

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 38 Summary and suggested reading n Invariant surface parametrization G. Zigelman, R. Kimmel, and N. Kiryati, Texture mapping using surface flattening via multi-dimensional scaling, IEEE TVCG 9 (2002), no. 2, 198– 207. n An image processing insight to correspondence B. K. P. Horn and B. G. Schunck, Determining optical flow, Artificial Intelligence 17 (1981), no. 1 -3, 185– 203. n Harmonic embeddings N. Litke, M. Droske, M. Rumpf, and P. Schroder, An image processing approach to surface matching. n Minimum distortion correspondence n Calculus of shapes A. M. Bronstein, M. M. Bronstein, R. Kimmel, Calculus of non-rigid surfaces for geometry and texture manipulation, IEEE TVCG 13 (2007), no. 5, 903– 913.

Numerical geometry of non-rigid shapes Nonrigid Correspondence & Calculus of Shapes 39 Summary and suggested reading n Morphing M. Alexa, Recent advances in mesh morphing, Computer Graphics Forum 21 (2002), no. 2, 173– 196. V. Surazhsky and C. Gotsman, Controllable morphing of compatible planar triangulations, ACM Trans. Graphics 20 (2001), no. 4, 203– 231. M. Kilian, N. J. Mitra, and H. Pottmann, Geometric modeling in shape space, ACM Trans. Graphics 26 (2007), no. 3.