Object Segmentation Presented by Sherin Aly 1 What
Object Segmentation Presented by Sherin Aly 1
What is a ‘Good Segmentation’?
http: //www. eecs. berkeley. edu/Research/Projects/CS/vision/groupin g/resources. html
Learning a classification model for segmentation Xiaofeng Ren and Jitendra Malik 4
methodology • Two-classification model • Over segmentation as preprocessing • They use classical Gestalt cues – Contour, texture, brightness and continuation • A linear classifier is used for training 5
Good Vs Bad segmentation a) Image from Corel Imagebase b) superimposed with a human marked segmentation c) Same image with Bad segmentation 6
How do we distinguish good segmentations from bad segmentations? 7
How? • Use “Classical Gestalt cues” – proximity, similarity and good continuation • Instead of Ad-hoc features combination decision about 8
Gestalt Principles of Grouping In order to interpret what we receive through our senses, we attempt to organize this information into certain groups. http: //allpsych. com/psychology 101/perception. html 9
Methodology • • • Preprocessing Feature extraction Feature evaluation Training Optimization Find good segmentaion 10
Preprocessing • Local • Coherent • Preserve structure • Contour • texture Superpixel map K=200 Reconstruction of human segmentation from Superpixels a contour-based measure is used to quantify this approximation 11
Tolerance 1, 2, and 3 The percentage of human marked boundaries covered by the superpixel maps 12
Feature Extraction 1. inter-region texture similarity 2. intra-region texture similarity 3. inter-region brightness similarity 4. intra-region brightness similarity 5. inter-region contour energy 6. intra-region contour energy 7. curvilinear continuity 13
Feature Extraction 1. inter-region texture similarity 2. intra-region texture similarity 3. inter-region brightness similarity 4. intra-region brightness similarity 5. inter-region contour energy 6. intra-region contour energy 7. curvilinear continuity 14
Feature Extraction 1. inter-region texture similarity 2. intra-region texture similarity 3. inter-region brightness similarity 4. intra-region brightness similarity 5. inter-region contour energy 6. intra-region contour energy 7. curvilinear continuity 15
Power of Gestalt cues = 16
Training the classifier • simple logistic regression classifier, Empirical distribution of pairs of features 17
Precision is the fraction of detections which are true positives. Recall is the fraction of true positives which are detected 18
Conclusion • There simple linear classifier had promising results on a variety of natural images. • boundary contour is the most informative grouping cue, and it is in essence discriminative. 19
Pros & Cons • Cons – The larger spatial support that superpixels provide, allowing more global features to be computed than on pixels alone. – The use of superpixels improves the computational efficiency – Super. Pixels technique is very applicable • Pros – Might fall in Local Minima 20
Combining Top-down and Bottom-up Segmentation Eran Borenstein Eitan Sharon Shimon Ullman 21
Motivation • Bottom-Up segmentation – Rely on continuity principle – Capture image properties “texture, grey level uniformity and contour continuity” – Segmentation based on similarities between image regions • How can we capture prior knowledge of a specific object (class)? – Answer: Top-Down Segmentation – use prior knowledge about an object Credit: Joseph Djugash
Bottom-Up Segmentation Credit: Joseph Djugash Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements ”.
Normalized-Cut Measure Credit: Joseph Djugash Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements ”.
Top-Down approach Input Credit: Joseph Djugash Fragments Matching Cover
Another step towards the middle Top-Down Bottom-Up Credit: Joseph Djugash
Some Definitions & Constraints • Measure of saliency h(Γi), hi є [0, 1) • A configuration vector s contains labels si (1/1) of all the segments (Si) in the tree • The label si can be different from its parent’s label s i – • Cost function for a given s Defines the weighted edge between Si & Si – Top-down term Bottom-up term
Classification Costs • The terminal segments of the tree determine the final classification • The top-down term is defined as: • The saliency of a segment should restrict its label (based on its parent’s label) • The bottom-up term is defined as:
Confidence Map • Evaluating the confidence of a region: • Causes of Uncertainty of Classification – Bottom-up uncertainty – regions where there is no salient bottom-up segment matching the top-down classification – Top-down uncertainty – regions where the topdown classification is ambiguous (highly variable shape regions) • The type of uncertainty and the confidence values can be used to select appropriate additional processing to improve segmentation
Results • Calculate average distance between a given segmentation contour and a benchmark contour. • Removing from the average all contour points having a confidence measure less than 0. 1. • The resulting confidence map efficiently separated regions of high and low consistency. • The combined scheme improved the top-down contour by over 67% on average. • This improvement was even larger in object parts with highly variable shape. 31
The initial classification map T(x, y) Results (cont. ) Buttom up • top-down process may produce a figure-ground approximation that does not follow the image discontinuities. • Salient bottom-up segments can correct these errors and delineate precise region boundaries
Results III (cont. )
Results III (cont. ) the top-down completely misses a part of the object. The confidence map may be helpful in identifying such cases,
Results III (cont. ) bottom-up segmentation may be insufficient in detecting the figure-ground contour, and the top-down process completes the missing information
Results III (cont. )
Results III (cont. ) Salient bottom-up segments can correct these errors and delineate precise region boundaries
Conclusion • Buttom-up and top-down merits • Provide reliable confidence map • It take into account all discontinuities at all scales But: • If the object is assigned a given category, the specific features cannot be adopted for other categories 38
Constrained Parametric Min-Cuts for Automatic Object Segmentation Joao Carreira Cristian Sminchisescu 39
Traditional Segmentation: Finding Homogeneous Regions g. Pb-owt-ucm: P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. PAMI 2010. 40
Bottom-up Object Segmentation Conventional Bottom-up Segmentation Proposed approach 1. Split multiple times 2. Retain object-like segmentations High redundancy Credit: J. Carreira
Bottom-up Object Segmentation A single multi-region segmentation or a hierarchy 42 Credit: J. Carreira
Proposed Bottom-up Object Segmentation Credit: J. Carreira robust set of overlapping figure-ground segmentations single-shot multiregion segmentation superpixels 43 Segments with object-like regularities
Constrained Parametric Min-Cuts for Automatic Object Segmentation Figure ground segmentation by growing regions around seeds parametric max-flow solver Ranking 44 Credit: J. Carreira
Constrained Parametric Min-Cuts for Automatic Object Segmentation 45 Credit: J. Carreira
Initialization • Foreground – Regular 5 x 5 grid geometry – Centroids of large N-Cuts regions – Centroids of superpixels closest to grid positions • Background – – Full image boundary Horizontal boundaries Vertical boundaries All boundaries excluding the bottom one Performance broadly invariant to different initializations
Generating a segment pool: constrained min-cut object min cut hard constraint background 47 Credit: J. Carreira
Generating a Segment Pool: Constrained Parametric Min-Cuts 48 Credit: J. Carreira
Generating a Segment Pool: Constrained Parametric Min-Cuts 49 Credit: J. Carreira
Generating a Segment Pool: Constrained Parametric Min-Cuts 50 Credit: J. Carreira
Generating a Segment Pool: Constrained Parametric Min-Cuts Can solve for all values of object bias in the same time complexity of solving a single min-cut using a parametric max-flow solver 51 Credit: J. Carreira
Fast Rejection Large set of initial segmentations (~5500) High Energy Low Energy ~2000 segments with the lowest energy Credit: Sasi. Kanth Bendapudi Yogeshwar Nagaraj Cluster segments based on spatial overlap (at least 0. 95) Lowest energy member of each cluster (~154 in PASCAL VOC)
Constrained Parametric Min-Cuts for Automatic Object Segmentation Credit: J. Carreira • ranks all the sampled object segmentations • discard all but a small subset of confident 53 ones.
Ranking object hypotheses mid-level, category independent features Boundary – normalized boundary energy Region – location, perimeter, area, Euler number, orientation, contrast with background Gestalt – convexity, smoothness Good Bad High boundary energy Low boundary energy Smooth. Non smooth. Euler number = 0 High Euler number 54 Credit: J. Carreira
Segment Ranking • Model data using a host of features – Graph partition properties – Region properties – Gestalt properties • Apply Features Normalization • Train regressor with the largest overlap ground-truth segment using Random Forests • Diversify similar rankings using Maximal Marginal Relevance (MMR)
Graph Partition Properties • Cut – Sum of affinities along segment boundary • Ratio Cut – Sum along boundary divided by the number • Normalized Cut – Sum of cut and affinity in foreground and background • Unbalanced N-cut – N-cut divided by foreground affinity • Thresholded boundary fraction of a cut
Region Properties • • Area Perimeter Relative Centroid Bounding Box properties • Fitting Ellipse properties • Eccentricity • Orientation • Convex Area • Euler Number • Diameter of Circle with the same area of the segment • Percentage of bounding box covered • Absolute distance to the center of the image
Gestalt Properties • • Inter-region texton similarity Intra-region texton similarity Inter-region brightness similarity Intra-region brightness similarity Inter-region contour energy Intra-region contour energy Curvilinear continuity Convexity – Ratio of foreground area to convex hull area
Feature Importance for the Random Forest regressor
Feature Importance
How to Model Segment Quality ? Best overlap with a ground truth object computed by intersection-over-union. 64 Credit: J. Carreira
Diversifying the Ranking Segment Ranking using Maximum Marginal Relevance Best two hypotheses Middle two hypotheses Worst two hypotheses Original Diversified 66
Performance Credit: Sasi. Kanth Bendapudi Yogeshwar Nagaraj
Ranking 79 Credit: J. Carreira
Running Demos • Methodologies employed – Kmeans using: • • • Texture RGB Texture + RGB + HSV Texture + Lab + HSV 80
Running Demos • Data set used – Microsoft Research Cambridge Object Recognition Image Database, version 1. 0. – Used: 7 classes with 23 per class • • Animal-grass Trees-sky-grass Buildings-sky-grass Airplanes-sky-grass Animal-grass Faces-BG Car-wall-ground 81
Experiment Results Features Texture + RGB RGB +HSV Texture+Lab+ HSV Animal-grass 72. 7% 74. 1% 72. 3% 72. 6% 74. 1% Trees-sky-grass 37. 1% 40. 7% 38. 2% 37. 1% Buildings-skygrass 44. 6% 42. 8% 51. 9% 45. 4% 44. 7% Airplanes-skygrass 58. 8% 54. 6% 59. 7% 58. 7% Animal-grass 64. 8% 69. 3% 71% 64. 9% Faces-BG 100% 100% Car-wall-ground 67. 2% 68. 4% 64. 9% 67. 2% Mean 63. 6% 63. 5% 65. 3% 64. 6% 63. 8% 82
Microsoft Research Cambridge Object Recognition Image Database, v Experiment Results Features One iteration Elapsed time is Texture + RGB 7. 42 secs Overall Elabsed 19. 9 time for mins experiment RGB +HSV Texture+Lab+ HSV 12. 26 secs. 1. 62 secs 1. 5 secs 7. 84 sec 32. 9 mins 4. 4 mins 4 min 21 mins 83
84
85
86
87
Acknowledgment • Dr. Devi Parikh • Dr. Joao Carreira 88
- Slides: 72