Visualizing the Invisible Occluded Vehicle Segmentation and Recovery

Visualizing the Invisible: Occluded Vehicle Segmentation and Recovery ICCV 19 Xiaosheng Yan 1, Feigege Wang 1, Wenxi Liu 1∗, Yuanlong Yu 1∗, Shengfeng He 2, Jia Pan 3 1 Fuzhou University∗; 2 South China University of Technology; 3 The University of Hong Kong

Task

Two steps 1. segmentation and amodal completion 2. appearance recovery

Network of segmentation and completion

Network of segmentation and completion visible part segmentation

Network of segmentation and completion amodal completion (Generator)

Two kind of Discriminators Object Discriminator: • Real: Gt mask, Sampled Mask • Fake: Recovered Mask classify whether its input masks are real vehicle masks or not

Two kind of Discriminators Instance Discriminator: • Real: Gt mask • Fake: Sampled Mask, Recovered Mask classify whether the input mask is the segmentation of the vehicle

Two steps 1. segmentation and amodal completion 2. appearance recovery

Appearance Recovery Network At training time, up and bottom two branches; at test time, only up branch

Appearance Recovery Network up branch input: origin image+visible mask+recovered amodal mask visible mask: indicate the visible part which will be reserved amodal mask: estimate the whole vehicle shape and fill in the invisible part

Appearance Recovery Network bottom branch input: origin image without amodal mask region+recovered amodal mask enforce the network to inpaint the whole vehicle based on image context

Experiments Dataset • from Shape. Net, select 401 different classes of vehicles • for each vehicle, screen-shot each rendered image from 80 different viewpoints • 32, 080 mask to form the auxiliary 3 D model pool

Ablation Study 1: for amodal mask Dstd vs 2 kind of D Dstd: gt vs recoverd mask

Ablation Study 2: for appearance Only up branch vs 2 branches

Compare with other methods amodal mask

Compare with other methods appearance

Application: Occluded vehicle tracking • average pixel error (APE) and average overlap (AO)