Applications Shmuel Peleg and Joshua Herman Panoramic Mosaics

Applications Shmuel Peleg and Joshua Herman, “Panoramic Mosaics by Manifold Projection”, Computer Vision and Pattern Recognition (CVPR), 1997 Wolfgang Heidrich and Hans-Peter Seidel, “View-independent environment maps”, SIGGRAPH / Eurographics Workshop on Graphics Hardware, 1998 Matthew Brand, “Charting a Manifold”, Mitsubishi tech report, 2003 K. Grochow, S. Martin, A. Hertzmann, and Z. Popovic Style-based Inverse Kinematics, Siggraph 2004 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Applications using manifolds • Many problems can be phrased in manifold terminology • Provides an alternative way of viewing the problem • Can also provide some formalism 2 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Applications • Surface fitting • Consistent parameterization • Image-based rendering • Environment mapping • Animation 3 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Surface fitting • CT, MRI data, noisy, densely sampled in widely-spaced contours • Boundary conditions less problematic Fitting Manifold Surfaces To 3 D Point Clouds Cindy Grimm, David Laidlaw, and Joseph Crisco J. of Biomedical Engineering Feb. 2002 4 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Consistent parameterization • Build manifold once • Fit to multiple bone point data sets 5 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Aligning parameterization • Identify similar points on meshes • Pin parameterization 6 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Application: Panoramas • Problem statement: • Given images from a known camera movement • Rotation about camera axis • “Push-broom” pan (assumes negligible depth) • “Glue” images together into a single image Rover, nasa. gov 7 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Camera rotation • Final image can be rendered on a cylinder • No parallax • Each image samples some number of pixels on cylinder (manifold) image Peleg and Herman 8 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Push broom/vertical slit camera • Translation of camera • Image slit perpendicular to camera motion • Need not travel in straight line • Depth differences negligible • Parallax • Manifold is part of ground plane viewed by camera Peleg and Herman Direction of travel Ground plane 9 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Practical problem • How to line up individual images to create one seamless image? • Manifold: Final image (3 D function RGB on 2 D manifold) • Charts: Individual images (2 D charts) • Overlap regions/transition functions: Unknown • Assume translation • (Account for optical effects of camera) • Note: Only works for these two camera motions 10 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

General solution • Define a format for the transition function • E. g. , translation in x, y • Define an error metric that measures how well two overlap regions agree • E. g. , pixel difference • Optimize over free parameters in transition function • E. g. , x, y shift between all pairs of overlapping images 11 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Solving for overlaps, transitions • Find translation that minimizes pixel differences • Find y that minimizes || I 0(s) – I 1(y(s))|| • y(s) = s + Ds, where Ds is unknown 0 1 Ds 12 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Final image • Transition functions align images (abstract manifold) • Final image colors? (RGB function on manifold) • Blend and embedding functions for each chart • Embedding function: Original image • Blending function: How much to use of each overlapping image • Usually favor very short blend regions 13 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Application: Environment mapping • Place scene/model inside sphere • Light intensity/color found by intersecting normal with sphere • 1 -1 mapping between normal direction and sphere • Every point on sphere assigned light intensity/color • Implementation • Store colors in one (or more) texture maps (2 D) 15 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Parameterization • Surface normal (point on sphere) to point in texture map • Atlas/local parameterization • Desirable properties • Even sampling of sphere • Adaptive • Partition • Overlap (mip mapping, continuity) • Simple to compute • Amenable to GPU implementation 16 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Approach 1 • Single texture map • Not unique (poles) • Poor sampling • Simple to compute 17 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Approach II • Cube mapping • Six charts • Discontinuities at edges • Sampling better at center of faces than edges • Simple (plane) computation • Which plane? 18 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Approach III • Parabolic mapping • Chart functions use parabolic function • Better sampling • Slightly more computation • Less-noticeable seams 19 Heidrich and Hans-Peter Seidel Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Proposal • Use chart approach • Allows for adaptive sampling (more detail where needed) • Chart sizes uniform: Tile texture map • Include overlap • Minimal extra texture map • Mip-mapping/down sampling 20 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Application: Animation • Human configuration space lies on a manifold of dimension n embedded in m dimensional space, where n << m • Articulated skeleton: over 40 degrees of freedom (shoulders, knees, hips, etc. , each 13 degrees of rotation) • Individual motions (reaching, walking) certainly lie on lower dimension manifolds • End-point of reach plus time • Shape of manifold of all possible human motions? • Who knows? K. Grochow, S. Martin, A. Hertzmann, and Z. Popovic 21 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Overview • Manifold learning • Data samples (e. g. , motion capture, key frames) • Interpolation equals manifold construction • Editing equals manifold editing • High-level review of manifold learning techniques • What kind of manifolds can you learn/construct from data? • 2 D animation example • Manifolds in animation 22 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

2 D illustration • Two joint angles • Circle X Circle manifold (torus) • Animation • Repetitive motion • Joint angle plot • Circle manifold • Animation is a 1 D manifold embedded in 2 D 23 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

2 D illustration • Two joint angles • Circle X Circle manifold (torus) • Animation • Repetitive motion • Joint angle plot • Circle manifold • Animation is a 1 D manifold embedded in 2 D 24 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Manifold learning • Input: Sample points in Rm • E. g. , Motion capture sequence, each pose is a data point, m is number of dof of joints • 2 D example: q, f for each pose • Assume data lies on a manifold of dimension n • Constraints on manifold shape/geometry (e. g. , linear, no self-intersections) • Goal: Parameterize manifold • Single chart 25 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Manifold learning, linear techniques • Principal components analysis (PCA), Independent components analysis (ICA) • Assumes shape of manifold in space is linear/planar • Learn vector(s) that span the hyper-plane 26 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Manifold learning, deformed linear • Support vector machines (SVM) • Deform space points are in first • Deformation is “nice” • Warps points to new places • “Guess” which deformation will work • Learn linear/planar function Deformation 27 Linear parameterization Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Manifold learning, local shape • Isomap, Local linear embedding (LLE), Semi-definite embedding (SDE) • Define a distance metric between points • Assumptions: • K closest neighbors are also closest K neighbors on the manifold • Disk around point in manifold would contain all K neighbors, but not others • Know dimension n of manifold • Possible to deform K neighbors into n dimensions while maintaining (relative) distances • Output: Single chart/parameterization of entire data set 28 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Isomap, LLE, SDE cont. • Non-obvious failure modes • Circular/repetitive data sets • Self-intersections Modified : 1 D embedding Raw result: 2 D embedding 29 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Manifold construction as learning • Use K neighbors to define chart domains (Uc) • Charts are “squished” Gaussians • Center, tangent vectors • Find transition functions (affine transformations) • Transformation takes tangent vectors into Rn • Aligns free vectors with neighbors Uc Matthew Brand, “Charting a Manifold”, Mitsubishi tech report, 2003 30 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Uses of animation manifold • General idea: • Construct (implicitly or explicitly) a manifold representing valid human poses • Create a new animation sequence • Foot must touch here, reach here, etc. • Not sufficient to constrain all degrees of freedom (dof) • Project on to manifold to fill in remaining dof K. Grochow, S. L. Martin, A. Hertzmann, Z. Popovic, Style-Based Inverse Kinematics, Siggraph 2004 31 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Re-sequencing as embedded manifold • Goal: Given existing sequence (samples), add more/change samples • Assumptions: • Samples come from some smooth manifold • Some form of interpolation gives new samples on manifold • Current approaches: Interpolation between neighboring samples in sequence for given new time 32 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Re-phrasing problem • Manifold learning or sequence timing provides parameterization/abstract manifold • Embed manifold with smooth function • Parameterization • Use function fitting • Re-sequencing: Evaluate embedding function E(M) 33 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Caveats • What makes animation data difficult? • “Distance” loses meaning in >> 10 dimensions • Every point equally far away • Can’t enumerate • Noise • Error in capture process • Skeleton only approximates human motion • Joint angle representation • Don’t explicitly deal with manifold, parameterization 34 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html

Summary • Manifolds provide a formalism for breaking a problem into manageable pieces • Charts provide local parameterization • Planar domains • Overlaps: Natural mechanism for moving between parameterizations • Blend functions instead of geometric constraints • No boundary condition problems • Explicitly encapsulating/representing manifold is beneficial • Cleaner algorithm specifications 35 Siggraph 2005, 8/1/2005 www. cs. wustl. edu/~cmg/Siggraph 2005/siggraph. html