3 D Object Detection using point cloud 1
3 D Object Detection —— using point cloud 1 程博文 3 D Object Detection 2019. 10. 21
3 D Data Source • Monocular camera(RGB image) • Binocular/Stereo camera(RGB-D image) • Li. DAR sensor(point cloud data) 2 程博文 3 D Object Detection 2019. 10. 21
3 D Data Source • Monocular camera(RGB image) • Binocular/Stereo camera(RGB-D image) • LIDAR sensor(point cloud data) • Projecting • Voxelization • Handcrafted feature extraction • Feature learning directly 3 程博文 3 D Object Detection 2019. 10. 21
3 D Data Source 4 程博文 3 D Object Detection 2019. 10. 21
3 D Data Source • Monocular camera(RGB image) • Binocular/Stereo camera(RGB-D image) • LIDAR Sensor(point cloud data) • Projecting • Voxelization • Handcrafted feature extraction • Feature learning directly 5 程博文 3 D Object Detection 2019. 10. 21
3 D Data Source • Feature learning directly on point cloud • Point. Net++ • Frustum Point. Net • Point. RCNN • Vote. Net 6 程博文 3 D Object Detection 2019. 10. 21
Point. Net • Feature learning directly on point cloud • Point. Net++ • Frustum Point. Net • Point. RCNN • Vote. Net 7 程博文 3 D Object Detection 2019. 10. 21
Point. Net CVPR 2017 8 程博文 3 D Object Detection 2019. 10. 21
Point. Net Challenges • Permutation invariance • Geometric transformation invariance 9 程博文 3 D Object Detection 2019. 10. 21
Point. Net Challenges • Permutation invariance 1. Sort input into canonical order 2. Use RNN with all permutations 3. Use a simple symmetric function 10 程博文 3 D Object Detection 2019. 10. 21
Point. Net 11 程博文 3 D Object Detection 2019. 10. 21
Theorem 1 12 程博文 3 D Object Detection 2019. 10. 21
Theorem 1 13 程博文 3 D Object Detection 2019. 10. 21
Theorem 2 14 程博文 3 D Object Detection 2019. 10. 21
Theorem 2 Bottleneck dimension of f Critical point set of S 15 程博文 3 D Object Detection 2019. 10. 21
Point. Net Challenges • Permutation invariance • Geometric transformation invariance 16 程博文 3 D Object Detection 2019. 10. 21
Point. Net • Input alignment 17 程博文 3 D Object Detection 2019. 10. 21
Point. Net • Input alignment Affine transformation matrix 18 程博文 3 D Object Detection 2019. 10. 21
Point. Net • Feature alignment 19 程博文 3 D Object Detection 2019. 10. 21
Point. Net • Feature alignment Orthogonal matrix(prior) 20 程博文 3 D Object Detection 2019. 10. 21
Challenges’ solution Challenges • Permutation invariance • Simple symmetric function • Geometric transformation invariance • Alignment using matrix multiplication 21 程博文 3 D Object Detection 2019. 10. 21
Point. Net Classification Network 22 程博文 3 D Object Detection 2019. 10. 21
Point. Net Classification Network 23 程博文 3 D Object Detection 2019. 10. 21
Point. Net Classification Network 24 程博文 3 D Object Detection 2019. 10. 21
Point. Net Classification Network 25 程博文 3 D Object Detection 2019. 10. 21
Point. Net Classification Network 26 程博文 3 D Object Detection 2019. 10. 21
Point. Net Classification Network 27 程博文 3 D Object Detection 2019. 10. 21
Point. Net Classification Network 28 程博文 3 D Object Detection 2019. 10. 21
Extension to Point. Net Segmentation Network 29 程博文 3 D Object Detection 2019. 10. 21
Extension to Point. Net Segmentation Network 30 程博文 3 D Object Detection 2019. 10. 21
Results on Object Classification 31 程博文 3 D Object Detection 2019. 10. 21
Visualizing Global Point Cloud Features 32 程博文 3 D Object Detection 2019. 10. 21
Point. Net++ NIPS 2017 33 程博文 3 D Object Detection 2019. 10. 21
Problems of Point. Net • Lack of local context • Non-uniform sampling density 34 程博文 3 D Object Detection 2019. 10. 21
Problems of Point. Net • Lack of local context • Non-uniform sampling density 35 程博文 3 D Object Detection 2019. 10. 21
Hierarchical Point Set Feature Learning • Lack of local context • Hierarchical point set feature learning 36 程博文 3 D Object Detection 2019. 10. 21
Hierarchical Point Set Feature Learning • Sampling&Grouping 37 程博文 3 D Object Detection 2019. 10. 21
Hierarchical Point Set Feature Learning • Point. Net 38 程博文 3 D Object Detection 2019. 10. 21
Hierarchical Point Set Feature Learning • Recursively apply pointnet 39 程博文 3 D Object Detection 2019. 10. 21
Point. Net layer vs. Convolution layer 40 程博文 3 D Object Detection 2019. 10. 21
Point. Net++ for Classification&Segmentation 41 程博文 3 D Object Detection 2019. 10. 21
Point. Net++ for Classification&Segmentation 42 程博文 3 D Object Detection 2019. 10. 21
Problems • Lack of local context • Non-uniform sampling density 43 程博文 3 D Object Detection 2019. 10. 21
Non-uniform Sampling Density • Non-uniform sampling density • Multi-scale grouping • Multi-resolution grouping 44 程博文 3 D Object Detection 2019. 10. 21
Problems’ solution • Lack of local context • Hierarchical point set feature learning • Non-uniform sampling density • Multi-scale grouping • Multi-resolution grouping 45 程博文 3 D Object Detection 2019. 10. 21
Robust learning under varying sampling density SSG：Single Scale Grouping 46 程博文 3 D Object Detection 2019. 10. 21
Point. Net++ Results: Classification 47 程博文 3 D Object Detection 2019. 10. 21
Point. Net++ Results: Non-Euclidean Space 48 程博文 3 D Object Detection 2019. 10. 21
Frustum Point. Nets CVPR 2018 49 程博文 3 D Object Detection 2019. 10. 21
Pipeline of Frustum Point. Nets 50 程博文 3 D Object Detection 2019. 10. 21
Coordinate systems for point cloud 51 程博文 3 D Object Detection 2019. 10. 21
Point. RCNN CVPR 2019 52 程博文 3 D Object Detection 2019. 10. 21
Pipeline of Point. RCNN 53 程博文 3 D Object Detection 2019. 10. 21
Pipeline of Point. RCNN Two-stage detection(RPN+RCNN) • Region Proposal Network • Foreground Point Segmentation • Bin-based 3 D Box Generation • Region Refinement • Bin-based 3 D Box Refinement • Confidence Prediction 54 程博文 3 D Object Detection 2019. 10. 21
Foreground Point Segmentation Two-stage detection(RPN+RCNN) • Region Proposal Network • Foreground Point Segmentation • Bin-based 3 D Box Generation • Region Refinement • Bin-based 3 D Box Refinement • Confidence Prediction 55 程博文 3 D Object Detection 2019. 10. 21
Foreground Point Segmentation 56 程博文 3 D Object Detection 2019. 10. 21
Bin-based 3 D Box Generation Two-stage detection(RPN+RCNN) • Region Proposal Network • Foreground Point Segmentation • Bin-based 3 D Box Generation • Region Refinement • Bin-based 3 D Box Refinement • Confidence Prediction 57 程博文 3 D Object Detection 2019. 10. 21
Bin-based 3 D Box Generation 3 D Bounding Box(x, y, z, h, w, l, θ) 58 程博文 3 D Object Detection 2019. 10. 21
Bin-based 3 D Box Generation 3 D Bounding Box(x, y, z, h, w, l, θ) Generate 3 D proposal for each foreground point 59 程博文 3 D Object Detection 2019. 10. 21
Bin-based 3 D Box Generation 3 D Bounding Box(x, y, z, h, w, l, θ) 60 程博文 3 D Object Detection 2019. 10. 21
Bin-based 3 D Box Generation 3 D Bounding Box(x, y, z, h, w, l, θ) 61 程博文 3 D Object Detection 2019. 10. 21
Region Proposal Network Output Two-stage detection(RPN+RCNN) • Region Proposal Network Output • Point cloud coordinates • Semantic features(point-wise) • Foreground mask • 3 D bbox/Region of Interests(Ro. Is) 62 程博文 3 D Object Detection 2019. 10. 21
Bin-based 3 D Box Refinement Two-stage detection(RPN+RCNN) • Region Proposal Network • Foreground Point Segmentation • Bin-based 3 D Box Generation • Region Refinement • Bin-based 3 D Box Refinement • Confidence Prediction 63 程博文 3 D Object Detection 2019. 10. 21
Region Refinement Canonical 3 D bounding box refinement 64 程博文 3 D Object Detection 2019. 10. 21
Region Refinement 65 程博文 3 D Object Detection 2019. 10. 21
Point. RCNN Results 66 程博文 3 D Object Detection 2019. 10. 21
Point. RCNN Results 67 程博文 3 D Object Detection 2019. 10. 21
Point. RCNN Discuss • Insights • Anchor free • Use segmentation mask • Bin-based loss(regression → classification) • Drawbacks • Low efficiency(use Point. Net++) • Bounding box proposal for each foreground point 68 程博文 3 D Object Detection 2019. 10. 21
Vote. Net ICCV 2019 69 程博文 3 D Object Detection 2019. 10. 21
Pipeline of Vote. Net 70 程博文 3 D Object Detection 2019. 10. 21
Vote. Net Two-stage detection • Voting in point clouds • Point cloud feature learning(Point. Net++) • Hough voting with deep networks • Object proposal and classification from votes • Vote clustering through sampling and grouping • Proposal and classification from vote clusters 71 程博文 3 D Object Detection 2019. 10. 21
Vote. Net Two-stage detection • Voting in point clouds • Point cloud feature learning(Point. Net++) • Hough voting with deep networks • Object proposal and classification from votes • Vote clustering through sampling and grouping • Proposal and classification from vote clusters 72 程博文 3 D Object Detection 2019. 10. 21
Hough voting with deep networks Poselets: Body Part Detectors Trained Using 3 D Human Pose Annotations(ICCV 2009) 73 程博文 3 D Object Detection 2019. 10. 21
Hough voting with deep networks 74 程博文 3 D Object Detection 2019. 10. 21
Hough voting with deep networks 75 程博文 3 D Object Detection 2019. 10. 21
Vote clustering through sampling and grouping Two-stage detection • Voting in point clouds • Point cloud feature learning(Point. Net++) • Hough voting with deep networks • Object proposal and classification from votes • Vote clustering through sampling and grouping • Proposal and classification from vote clusters 76 程博文 3 D Object Detection 2019. 10. 21
Proposal and classification from vote clusters Two-stage detection • Voting in point clouds • Point cloud feature learning(Point. Net++) • Hough voting with deep networks • Object proposal and classification from votes • Vote clustering through sampling and grouping • Proposal and classification from vote clusters 77 程博文 3 D Object Detection 2019. 10. 21
Proposal and classification from vote clusters Objectness score Bounding box params classification scores Object or not 78 程博文 3 D Object Detection 2019. 10. 21
Vote. Net Results 79 程博文 3 D Object Detection 2019. 10. 21
Vote. Net Results 80 程博文 3 D Object Detection 2019. 10. 21
Point. RCNN vs. Vote. Net Point. RCNN Vote. Net Two stage Network Use Point. Net++ as backbone Bbox proposal for all Vote for seeds from foreground points set abstraction Feature concatenation Feature offset 81 程博文 3 D Object Detection 2019. 10. 21
THANKS 82 程博文 3 D Object Detection 2019. 10. 21
2 D Object Detection Roadmap 83 程博文 3 D Object Detection 2019. 10. 21
- Slides: 83