Panoptic Segmentation Without Bounding Box Proposals Week 9

Panoptic Segmentation Without Bounding Box Proposals Week 9 Audrey Wiseman, Kevin Duarte

The Project • A panoptic segmentation network that uses centers to locate instances • Panoptic Deeplab

Work So Far • Implemented Panoptic Deeplab • Semantic Segmentation Branch • Added Instance Segmentation • Generates segmentations, instance centers, and center offsets Ground Truths Predictions Segmentations Instance Centers Center Offsets

This Week: New Post-Processing Instance Maps Offsets “Vote Map” Set of Centers

Visual Defects Defect 1: Edge cases between two instances Defect 2: A single instance shown as multiple instances

Defect 1 • Found the bounding box for each instance • Computed ratio of the number of pixels to the size of bounding box • If bounding box ratio is below a given threshold, the instance is set to next closest instance Defect 1: Edge cases between two instances

Defect 2 • Find the mean offset for each instance • Should ideally be zero • If mean offset is above a given threshold, the instance is set to next closest instance Defect 2: A single instance shown as multiple instances

Current Results Semantic Segmentation Results Class Io. U Category Io. U Score 67. 7% 84. 9%

Current Results (cont. ) Instance Segmentation Results Postprocessing Method Old Postprocessing New Postprocessing m. AP Score 16% 7. 1% New Postprocessing w/ Bounding Box Heuristic 12. 3% New Postprocessing w/ Mean Offset Heuristic New Postprocessing – identify instances by class TBD

Next Step- Capsules Segmentations . . … …. . … Capsules 2 Capsules 1 Image Encoder Decoder Object selection 1 2 … … … Instance Capsules ………. N Class Capsules Offsets