Video Motion Capture Christoph Bregler Jitendra Malik UC

Video Motion Capture Christoph Bregler Jitendra Malik UC Berkley 1997 3/5/2002 Phillip Saltzman

Overview • Challenges • Review • Method • Results • Conclusions 3/5/2002 Phillip Saltzman

Challenges • High Accuracy • Frequent Inter-part Occlusion • Low Contrast 3/5/2002 Phillip Saltzman

Review 3/5/2002 Phillip Saltzman

Review • Motion capture on synthetic images – O’Rouke and Balder, 1980 • 1 DOF marker free tracking – Hogg, 1983. Rohr, 1993 • Higher DOF full body tracking – Gravrila and Davis, 1995 3/5/2002 Phillip Saltzman

Review About the previous work – All in controlled environments with high contrast and clear edge bounries – Most use skintight suits or markers – Camera calibration needed 3/5/2002 Phillip Saltzman

Method 3/5/2002 Phillip Saltzman

Method Basic Assumptions – From frame to frame, all intensity pixel intensity changes are local: – u is motion model and is written as a matrix equation: 3/5/2002 Phillip Saltzman

Method Finding Gradients – Gradient form of the first equation: – Find a least squares solution to f – Warp image I(t+1) based on f – Find new gradients – Repeat to minimize 3/5/2002 Phillip Saltzman

Method Motion as twists – Standard pose matrix to move from object space to camera space (3 D) – Scaled orthographic projection moves to image space – Requires knowing something about the 3 D model of the image. Approximated as ellipsoids. 3/5/2002 Phillip Saltzman

Method Motion as twists – Any motion can be represented as a rotation about an axis, and a translation about that axis – For example, to make this motion: 3/5/2002 Phillip Saltzman

Method Motion as twists You make this motion: 3/5/2002 Phillip Saltzman

Method Motion as twists – Twists can be represented as small vector or matrix – Can be made to a pose by – Encode the motion of a pixel between two frames 3/5/2002 Phillip Saltzman

Method Motion as twists – Linear algebra manipulation allows using the twist vector to write a motion equation for each pixel – Those equations are put in a vector and used to find a global f parameter for that object 3/5/2002 Phillip Saltzman

Method Kinematic chains – Body parts represented as multiple connected objects – Each object can be found by the top pose and an angle and twist for each object down the chain – More linear algebra is used to find a f for each body part 3/5/2002 Phillip Saltzman

Method Multiple cameras – Adds accuracy because change of fully occluded parts drop with each view – Normal motion equation is: – H is system of equations for each pixel – f is global parameter vector for each object – z is initial position of the pixel 3/5/2002 Phillip Saltzman

Method Multiple cameras – Adding synchronized cameras: – H becomes a matrix where each column represents a view – The f vector gets a term W for each view that represents the pose seen from that view – z becomes a vector with an initial position for each view. 3/5/2002 Phillip Saltzman

Method Support maps – Limits pixel search to area defined by map for speed – Value for each pixel in range [0, 1], where 1 means pixel is in the region – Method for finding starts as an elliptical guess, but refining it is not described 3/5/2002 Phillip Saltzman

Method Algorithm review Input: Image I(t), I(t+1), pose and IK angles Output: Pose and IK angles for I(t+1) Find 3 D points for each pixel in image Compute support map for each segment Set poses and IK angles for I(t+1) = I(t) Iterate: Compute gradients Estimate f Update poses and IK angles Warp image based on the pose and support map 3/5/2002 Phillip Saltzman

Method Initialization – Algorithm depends on known positions for the first frame – For multiple views, each first frame must be initialized – User clicks joint positions, and 3 D estimations and joint angles are computed – Values like symmetry can be enforced 3/5/2002 Phillip Saltzman

Results 3/5/2002 Phillip Saltzman

Results In Lab Movie – Single angle – 53 frames with decent results – Upper leg hard to track, so IK chain compensates with lower leg and torso 3/5/2002 Phillip Saltzman

Results Oblique Lab Movie – – – 3/5/2002 Oblique angle Tracking over 45 frames Algorithm could track change in scale due to perspective changes Phillip Saltzman

Results Digital Muybridge – – 3/5/2002 Oldest known “movie” High noise and low contrast Low framerate Multiple views Phillip Saltzman

Conclusions Future Work/Shortcomings – May break with large movements – Fixed camera only – Did not show tracking of back limbs – No timing data – Few results 3/5/2002 Phillip Saltzman