Modeling the Shape of People from 3 D

Modeling the Shape of People from 3 D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne Koller, Sebastian Thrun

The Dataset n Scans n n 4 views, ~125 k polygons ~65 k points each Problems n n Missing surface Drastic pose changes

Space of Human Shapes movie [scape movie]

The Modeling Pipeline

Talk outline n Scan registration n n Rigid Nonrigid Correlated correspondence Decomposition into approximately rigid parts Learning a deformable human model

Talk outline n 3 D Scan registration n n Rigid Nonrigid Correlated correspondence Decomposition into approximately rigid parts Learning a deformable human model

Rigid Scan Registration n Aligning point clouds X and Z n n n Match each point in Z to its closest point in X Compute the best rigid transformation T which aligns the matching pairs Transform the points of cloud X by T Repeat Iterative Closest Point (ICP) algorithm [Besl et al ’ 92]

ICP matching step n Objective (using n obtains n Substituting the above for s yields a system linear in R Choose a parameterization of R (quaternions q) and solve n Exact solution n n Setting Overall, ICP converges to a local minimum of the energy )

Nonrigid Registration Example n Deformable template model X n n Set of deformable springs Nonrigid registration n n Deforms X to match a scan Z Minimizes the deformation of X

Generative Model Deformation / Transformation Q Model mesh X Data Generation / Correspondences C Transformed mesh X’ Data mesh Z Goal: Given meshes X and Z, recover transformation Q and correspondences C

Data mesh generation Transformed Model Data Correspondence ck specifies which point in X’ generates point zk: X’ Z

Nonrigid ICP failure example n Nonrigid ICP attempts to maximize X n Z Local Minima n Poor transformation and correspondence estimates reinforce each other

Correlated correspondences n Nonrigid-ICP decorrelates the correspondences n Correspondences are conditionally independent given Q n n c 1 c 2 n If Q is unknown, correspondences are correlated E. g. nearby points in Z should be mapped to nearby points in X The search space is exponential for the general registration problem

Correlated Correspondence Z n n X n Compute an embedding of mesh Z into model X The domain of ci is the set of possible (discrete) matches for zi on the surface of X The solution is a consistent assignment to all correspondence variables C

Correlated Correspondence Z n n n X Edge is generated from some edge in X Prefer values of c 1, c 2 according to the deformation of X they induce Doing this for all edges in Z results in a Markov network over C (c 1) c (c 1, c 2) 1 (c 1, c 3) c (c 2) c 2 (c 2, c 3) 3 (c 3)

Pairwise Markov Networks i yi ij yj {1, …, K} n Inference is Markov Nets is generally intractable n Good approximate algorithms exist (belief propagation)

CC Potentials (c 1) c (c 1, c 2) 1 (c 1, c 3) c (c 2) c 2 (c 2, c 3) 3 (c 3) n n Deformation potentials Spin image* potentials n n Quantify the similarity of the local surface around two matching points Geodesic Potentials *[Johnson+Hebert ’ 97]

Geodesic potentials Z n X Nearby points in Z must be nearby in X n Constraint between each pair of adjacent points zk, zl Z n X Distant points in Z must be distant in X n Constraint between each pair of distant points zk, zl (farther than 5 r) r resolution of mesh X

Local surface signatures n Use spin-images [Johnson ’ 97] n n n 2 D Histogram of distances from an oriented reference point Rotationally-invariant / Robust under clutter and occlusion / Compressible (PCA) Potential f(ck = i) encodes how well the signature of point zk matches the signature of point xi in the model: n

Results: Human poses dataset Model Cyberware scans Registrations Art. Model II • 4 markers were used on each scan to avoid the need for multiple initializations of Loopy-BP • Art. Model I found using 20 registered scans • Art. Model II found using 44 registered scans

CC movie [movie]

Talk Outline n Scan registration n n Rigid Nonrigid Correlated correspondence Recovering the articulated skeleton Learning a deformable human model

Recovering the skeleton Input: models, correspondences Output: rigid parts, skeleton

Probabilistic Generative Model a Part labels Points … 1 … x 1 Transformed Model Instance a x … 1 Points aj = Part (xj) T N y Point labels Transformations N b 1 y N b. K z 1 … z K bk = Model. Point (zk)

Part Properties n Parts are preferably contiguous regions n n Adjacent points on the surface should have similar labels Enforce this with a Markov network: If i, j are connected in model mesh 1 > > 0. 5 a a 1 a 2 3

Iterative Optimization n Hard EM n n Given transformations to get , perform min-cut inference Given labels , solve for rigid transformations

Results: Puppet articulation

Results: Arm articulation

Parts movie [parts. gif]

Talk Outline n Scan registration n n Rigid Nonrigid Correlated correspondence Recovering the articulated skeleton Learning a deformable human model

Pose variation Modeling Human Shape Body-shape variation

Representation of pose deformation Rigid articulated deformation Pose deformation Given estimates of R, Q, synthesizing the shape is straightforward :

Learning non-rigid pose deformation n n For each polygon, predict entries of nearest 2 joints (represented as twists Linear regression parameters : from rotations of ).

Learning body-shape deformation n Include also change in shape due to different people: n Do PCA over body-shape matrices :

Shape completion n Process: n n n Add a few markers (~6 -8) Run CC algorithm to get >100 markers Optimize for pose/body -shape parameters and non-rigid surface

Mocap shape completion

Recap n Algorithms for registration between two surfaces n ICP algorithm iterates between finding corresponding points between two scans and computing the transformation between them n n n CC explicitly encodes correlations between correspondence variables n n n Rigid variant (6 unknown DOF) Nonrigid variant (6 Number of polygons DOF) Essential when we have poor initial transformation estimate Exponential complexity, approximate algorithm Modeling humans n Decouple effects of pose and different physique on shape deformation

The end

Geodesic Potentials: close Z n X Nearby points in Z must be nearby in X n Constraint between each pair of adjacent points zk, zl

Geodesic Potentials: far Z n X Distant points in Z must be distant in X n Constraint between each pair of distant points zk, zl