Learning 3 D mesh segmentation and labeling Presented

Learning 3 D mesh segmentation and labeling Presented by Ayrat Mutygullin 23 May 2017 Evangelos Kalogerakis Aaron Hertzmann Karan Singh 1

Goal: mesh segmentation and labeling Labeled Mesh Input Mesh Head Neck Torso Leg Tail Training Meshes Ear 2

Related work: mesh segmentation [Mangan and Whitaker 1999, Shlafman et al. 2002, Katz and Tal 2003, Liu and Zhang 2004, Katz et al. 2005, Simari et al. 2006, Attene et al. 2006, Lin et al. 2007, Kraevoy et al. 2007, Pekelny and Gotsman 2008, Golovinskiy and Funkhouser 2008, Li et al. 2008, Lavoue and Wolf 2008, Huang et al. 2009, Shapira et al. 2010] Surveys: [Attene et al. 2006, Shamir 2008, Chen et al. 2009] 3

Related work: mesh segmentation Shape Diameter [Shapira et al. 10] Randomized Cuts [Golovinskiy and Funkhouser 08] Random Walks [Lai et al. 08] Normalized Cuts [Golovinskiy and Funkhouser 08] 4

Is it possible to perform human level segmentation? [X. Chen et al. SIGGRAPH 09] 5

Is it possible to perform human level segmentation? [X. Chen et al. SIGGRAPH 09] 6

Related work: computer vision for segmentation and labeling Textonboost [Shotton et al. ECCV 06] 7

Related work: mesh segmentation & labeling Consistent segmentation of 3 D meshes [Golovinskiy and Funkhouser 09] Multi-objective segmentation and labeling [Simari et al. 09] 8

Learning mesh segmentation and labeling Learn from examples Significantly better results than state-of-the-art No manual parameter tuning Can learn different styles of segmentation Several applications of part labeling 9

Labeling problem statement Head c 3 c 1 c 2 Neck Torso Leg c 4 Tail Ear C = { head, neck, torso, leg, tail, ear } 10

Conditional Random Field for Labeling Head Neck Torso Leg Tail Labeled Mesh Input Mesh Ear Unary term 11

Conditional Random Field for Labeling Head Neck Torso Leg Tail Labeled Mesh Input Mesh Face features Ear

Conditional Random Field for Labeling Head Neck Torso Leg Tail Labeled Mesh Input Mesh Face Area Ear

Conditional Random Field for Labeling Head Neck Torso Leg Tail Input Mesh Labeled Mesh Pairwise Term Ear

Conditional Random Field for Labeling Head Neck Torso Leg Tail Input Mesh Labeled Mesh Ear Edge Features

Conditional Random Field for Labeling Head Neck Torso Leg Tail Input Mesh Labeled Mesh Edge Length Ear

Conditional Random Field for Labeling Head Neck Torso Leg Tail Labeled Mesh Input Mesh Unary term Ear

Feature vector x surface curvature singular values from PCA shape diameter distances from medial surface average geodesic distances shape contexts spin images contextual label features 18

Learning a classifier x 2 Head Neck Torso Leg Tail Ear x 1 19

Learning a classifier We use the Jointboost classifier [Torralba et al. 2007] x 2 Head ? Neck Torso Leg Tail Ear x 1

Unary term 21

Unary Term Most-likely labels Classifier entropy 22

Our approach Head Neck Torso Leg Tail Input Mesh Labeled Mesh Ear Pairwise Term 23

Pairwise Term Geometry-dependent term 24

Pairwise Term Head Neck Label compatibility term Ear Tors Leg Tail o Head Neck Ear Tors o Leg Tail 25

Full CRF result Head Neck Torso Leg Tail Ear Unary term classifier Full CRF result 26

Dataset used in experiments We label 380 meshes from the Princeton Segmentation Benchmark[Chen et al. 2009] Antenn a Head Thorax Leg Abdome n Each of the 19 categories is treated separately 27

Quantitative Evaluation Labeling • 6% error by surface area • No previous automatic method Segmentation • Our result: 9. 5% Rand Index error • State-of-the art: 16% [Golovinskiy and Funkhouser 08] • With 6 training meshes: 12% • With 3 training meshes: 15% 28

Labeling results 29

Segmentation Comparisons Shape Diameter [Shapira et al. 10] Randomized Cuts [Golovinskiy and Funkhouser 08] Our approach 30

Segmentation Comparisons Shape Diameter [Shapira et al. 10] Randomized Cuts [Golovinskiy and Funkhouser 08] Our approach 31

Learning different segmentation styles Head Neck Torso Leg Tail Ear Training Meshes Head Test Meshes Front Torso Middle Torso Back Torso Front Leg Back Leg Tail 32

Generalization to different categories Head Wing Body Tail Head Neck Torso Leg 33

Failure cases Face Hair Neck Handle Nose Cup Torso Leg 34

Limitations Adjacent segments with the same label are merged Head Tors o Upper arm Lower arm Hand Upper leg Lower leg Foot 35

Limitations Results depend on having sufficient training data Handle Cup Top Spout 19 training meshes 36

Limitations Many features are sensitive to topology Head Tors o Upper arm Lower arm Hand Upper leg Lower leg Foot 37

Thank you! 38

Question 1 - Density of meshes. Did you have to remesh? - No, they used just Benchmark meshes, And i think they are very nice and easy to use without remeshing.

Question 2 • - If you would work with scanned meshes or something, you have to worry about density of samplings of this meshes, right? • - Ok, so for that reason we use multy-scaled features, so essentially we compute for example curvature in relative size of object, so during the feature selection that Joinboost does, the scale of this object would be also automatically selected to use this mapping, so its adapted for different density meshes. But for point clouds and some complex architecture models, we have to explore more features I guess.

Question 3 Slide 18. - Some of this descriptors have already been used for segmentation individually? - In our case we put all this features in a big feature vector and we do this because different such a features might be relevant for different parts and different segmentation styles.

Question 4 - What is the ω and μ in pairwise term ? - This classifier helps detect boundaries better than using only dihedral angles. The second term penalizes boundaries between faces with high exterior dihedral angle ω. The μ term penalizes boundary length and is helpful for preventing jaggy boundaries and for removing small, isolated segments. A small constant ε is added to avoid computing log 0.

Question 5 - What do Joint. Boost classifier? - Joint. Boost is a boosting algorithm that has many appealing properties: it performs automatic feature selection and can handle large numbers of input features for multiclassification, it has a fast sequential learning algorithm, and it produces output probabilities suitable for com- bination with other terms in the CRF model. Joint. Boost is designed to share features among classes, which greatly reduces generalization error for multiclass recognition when classes overlap in feature space.