image and video editing Based on human body

image and video editing Based on human body posture Kai Wang

Introduction Source Image Target Pose Synthesized Image Synthesizing Images of Humans in Unseen Poses Guha Balakrishnan，Amy Zhao，Adrian V. Dalca，Fredo Durand， John Guttag 2

Introduction Everybody Dance Now Caroline Chan, Shiry Ginosar, Tinghui Zhou, Alexei A. Efros 3

Introduction Photo Wake-Up: 3 D Character Animation from a Single Photo Chung-Yi Weng 1 Brian Curless, Ira Kemelmacher-Shlizerman University of Washington Facebook, Inc 4

Related Work • The closest related work either does not recover fully 3 D models or is built on video input. • Most single-image person animation has focused on primarily 2 D or pseudo-3 D animation A. Hornung, E. Dekkers, and L. Kobbelt. Character animation from 2 d pictures and 3 d motion data. ACM Transactions on Graphics (TOG), 26(1): 1, 2007. • Most methods for 3 D body shape estimation focus on semi-nude body. 5

Related Work Base on: • Smpl • Automatic Pose Estimation of 3 D Human • Mask R-CNN • Patch. Match 6

Overview 7

Method 8

person detection and segmentation We use Mask R-CNN , an image segmentation algorithm. It classifies different objects in the image and identifies their boundaries 9

2 D pose estimation We estimate the pose through picture. Then use it to build a smpl model. 10

Method 11

SMPL model SMPL is a skinned vertexbased model that accurately represents a wide variety of body shapes in natural human poses and is compatible with existing graphics pipelines(Maya, Unity). M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black. SMPL: A skinned multiperson linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6): 248: 1– 248: 16, Oct. 2015. LBS DQBS SMPL 12

Mesh Warping, Rigging, & Skinning • To construct a 3 D mesh with skeletal rigging, we first fit a SMPL model to the 2 D input pose, which additionally recovers camera parameters. • We then project this mesh into the camera view to form a silhouette mask. • The projection additionally gives us a depth map , a normal map and a skinning map. 13

Solve f(x) we solve for a smooth inverse warp, f(x)： To construct the inverse warp, f(x), many smooth warping functions are possible; we choose one based on meanvalue coordinates Suppose we have a correspondence function that identifies points on the input silhouette boundary with points on the SMPL silhouette boundary. We now seek a mapping 14

Constructing the complete mesh We reconstruct front and back meshes in the standard way: back-project depths into 3 D and construct two triangles for each 2 x 2 neighborhood. We assign corresponding skinning weights to each vertex. Stitching the front and back meshes together is straightforward as they correspond at the boundary. 15

Method 16

Patch. Match A method of texture synthesis, Patch. Match uses other areas in the image to restore the edge area. 17

using motion capture sequences 18

Results and Discussion As shown in(b), the fitted, semi-nude SMPL model does not correctly handle subject silhouettes. The results are shown in (c). Their method does not fit the silhouette well; e. g. , smooth SMPL parts don’t become complex. 19

DOWNLOADS at http: //vcc. szu. edu. cn Thank You!