Roadmap of Trajectory Modeling 2020 07 21 Contents

研究生学术交流 Roadmap of Trajectory Modeling 2020. 07. 21 刘天禹

Contents 1. Datasets 2. Roadmap & Tasks of Autonomous Driving a) Object Detection from Point Clouds b) Motion Forecasting c) Interaction Modeling 3. Future Works 4. Sp. GNN：Spatially-Aware Graph Neural Networks for Relational Behavior Forecasting from Sensor Data

2. Roadmap 特征 Activity RGB Signal/Sig n Li. DAR FOT Object Locations Depth Feature 地� Motion 其他 模 态 深度 可�光 低级 高级 HD-MAP 麦克� �列 IMU/GP S Ego-car Speed

2. Roadmap 低级 高级 特征 Activity RGB Signal Detection Depth Estimation Li. DAR Depth Feature Voxelization Featurization FOT Map Automation 地� 其他 模 态 深度 可�光 Activity Recognition 麦克� �列 IMU/GP S Signal/Sig n Object Detection Direct Detection Cloud Detection Object Locations Conditional Forecasting Trajectory Modeling Interaction Modeling HD-MAP Activity Prediction Motion Forecasting Ego-car Speed

2 a. Object Detection From Point Cloud 低级 高级 特征 Activity RGB Signal Detection Depth Estimation Li. DAR Depth Feature Voxelization Featurization FOT Map Automation 地� 其他 模 态 深度 可�光 Activity Recognition 麦克� �列 IMU/GP S Signal/Sig n Object Detection Direct Detection Cloud Detection Object Locations Conditional Forecasting Trajectory Modeling Interaction Modeling HD-MAP Activity Prediction Motion Forecasting Ego-car Speed

2 a. Object Detection From Point Cloud 1. Rasterization / Voxelization / Featurization • 原始特征点集不同 [1 ] • 整理为维度不变的特征向量 （Permutation Invariant） [1]. Ngiam, Jiquan, et al. "Starnet: Targeted computation for object detection in point clouds. " ar. Xiv preprint ar. Xiv: 1908. 11069 (2019). [2]. Qi, Charles R. , et al. "Pointnet: Deep learning on point sets for 3 d classification and segmentation. " Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. [3]. Qi, Charles Ruizhongtai, et al. "Pointnet++: Deep hierarchical feature learning on point sets in a metric space. " Advances in neural information processing systems. 2017.

2 a. Object Detection From Point Cloud 1. Rasterization / Voxelization / Featurizer • 原始特征点集不同 • 整理为维度不变的特征向量 （Permutation Invariant） 2. 直接预测[1][2] [1]. Wang, Shenlong, et al. "Deep parametric continuous convolutional neural networks. " Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018. [2]. Zaheer, Manzil, et al. "Deep sets. " Advances in neural information processing systems. 2017. [1 ]

2 a. Object Detection From Point Cloud 1. 3 D-CNN [1 ] [1]. Engelcke, Martin, et al. "Vote 3 deep: Fast object detection in 3 d point clouds using efficient convolutional neural networks. " 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2017. [2]. Li, Bo. "3 d fully convolutional network for vehicle detection in point cloud. " 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017.

2 a. Object Detection From Point Cloud 1. 3 D-CNN [1 ] 2. （2+1）D-CNN [1]. Luo, Wenjie, Bin Yang, and Raquel Urtasun. "Fast and furious: Real time end-to-end 3 d detection, tracking and motion forecasting with a single convolutional net. " Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2018.

2 a. Object Detection From Point Cloud 1. 3 D-CNN [1 ] [3 ] 2. （2+1）D-CNN 3. Front-view Projection [1]. Li, Bo, Tianlei Zhang, and Tian Xia. "Vehicle detection from 3 d lidar using fully convolutional network. " ar. Xiv preprint ar. Xiv: 1608. 07916 (2016). [2]. Chen, Xiaozhi, et al. "Multi-view 3 d object detection network for autonomous driving. " Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. [3]. Meyer, Gregory P. , et al. "Lasernet: An efficient probabilistic 3 d object detector for autonomous driving. " Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.

2 a. Object Detection From Point Cloud 1. 3 D-CNN [1 ] 2. （2+1）D-CNN 3. Front-view Projection 4. 2 D BEV (Bird’s eye view) [1]. Yang, Bin, Wenjie Luo, and Raquel Urtasun. "Pixor: Real-time 3 d object detection from point clouds. " Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2018. [2]. Yang, Bin, Ming Liang, and Raquel Urtasun. "Hdnet: Exploiting hd maps for 3 d object detection. " Conference on Robot Learning. 2018. [3]. Yang, Zetong, et al. "Std: Sparse-to-dense 3 d object detector for point cloud. " Proceedings of the IEEE International Conference on Computer Vision. 2019. [4]. Liang, Ming, et al. "Multi-task multi-sensor fusion for 3 d object detection. " Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.

2 b. Motion Forecasting 低级 高级 特征 Activity RGB Signal Detection Depth Estimation Li. DAR Depth Feature Voxelization Featurization FOT Map Automation 地� 其他 模 态 深度 可�光 Activity Recognition 麦克� �列 IMS/GP S Signal/Sig n Object Detection Direct Detection Cloud Detection Object Locations Conditional Forecasting Trajectory Modeling Interaction Modeling HD-MAP Activity Prediction Motion Forecasting Ego-car Speed

2 b. Motion Forecasting 1. DESIRE[1] [1]. Lee, Namhoon, et al. "Desire: Distant future prediction in dynamic scenes with interacting agents. " Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.

2 b. Motion Forecasting 1. DESIRE[1] [2 ] 2. R 2 P 2 [2] [1]. Lee, Namhoon, et al. "Desire: Distant future prediction in dynamic scenes with interacting agents. " Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. [2]. Rhinehart, Nicholas, Kris M. Kitani, and Paul Vernaza. "R 2 p 2: A reparameterized pushforward policy for diverse, precise generative path forecasting. " Proceedings of the European Conference on Computer Vision (ECCV). 2018.

2 b. Motion Forecasting 1. DESIRE[1] [3 ] 2. R 2 P 2 [2] 3. SIMP [3] [1]. Lee, Namhoon, et al. "Desire: Distant future prediction in dynamic scenes with interacting agents. " Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. [2]. Rhinehart, Nicholas, Kris M. Kitani, and Paul Vernaza. "R 2 p 2: A reparameterized pushforward policy for diverse, precise generative path forecasting. " Proceedings of the European Conference on Computer Vision (ECCV). 2018. [3]. Hu, Yeping, Wei Zhan, and Masayoshi Tomizuka. "Probabilistic prediction of vehicle semantic intention and motion. " 2018 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2018.

2 c. Interaction Modeling 低级 高级 特征 Activity RGB Signal Detection Depth Estimation Li. DAR Depth Feature Voxelization Featurization FOT Map Automation 地� 其他 模 态 深度 可�光 Activity Recognition 麦克� �列 IMS/GP S Signal/Sig n Object Detection Direct Detection Cloud Detection Object Locations Conditional Forecasting Trajectory Modeling Interaction Modeling HD-MAP Activity Prediction Motion Forecasting Ego-car Speed

2 c. Interaction Modeling 1. Game Theory[1] [1]. Ma, Wei-Chiu, et al. "Forecasting interactive dynamics of pedestrians with fictitious play. " Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. [3 ]

2 c. Interaction Modeling 1. Game Theory [2 ] [1 ] 2. LSTMs [1] [2] [3 ] [1]. Alahi, Alexandre, et al. "Social lstm: Human trajectory prediction in crowded spaces. " Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. [2]. Zhao, Tianyang, et al. "Multi-agent tensor fusion for contextual trajectory prediction. " Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019. [3]. Gupta, Agrim, et al. "Social gan: Socially acceptable trajectories with generative adversarial networks. " Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.

2 c. Interaction Modeling 1. DESIRE 2. LSTMs 3. GNN [1]. Casas, Sergio, et al. "Spatially-aware graph neural networks for relational behavior forecasting from sensor data. " ar. Xiv preprint ar. Xiv: 1910. 08233 (2019). [1 ]

2 c. Interaction Modeling 1. DESIRE[1] 2. LSTMs 3. GNN 4. Attention [3] [1]. Sadeghian, Amir, et al. "Car-net: Clairvoyant attentive recurrent network. " Proceedings of the European Conference on Computer Vision (ECCV). 2018. [1 ]

2 c. Interaction Modeling [1 ] And More [3 ] [1]. Santoro, Adam, et al. "A simple neural network module for relational reasoning. " Advances in neural information processing systems. 2017. [2]. Sun, Chen, et al. "Actor-centric relation network. " Proceedings of the European Conference on Computer Vision (ECCV). 2018. [3]. Sun, Chen, et al. "Relational action forecasting. " Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019. [2 ]

3. Possible Future Works 低级 高级 特征 Activity RGB Signal Detection Depth Estimation Li. DAR Depth Feature Voxelization Featurization FOT Map Automation 地� 其他 模 态 深度 可�光 Activity Recognition 麦克� �列 IMS/GP S Signal/Sig n Object Detection Direct Detection Cloud Detection Activity Prediction Object Locations Conditional Forecasting Trajectory Modeling Interaction Modeling HD-MAP Multi-Model Interaction Modeling Ego-car Speed Event-Based Interaction Modeling Motion Forecasting

4. Spatially-Aware Graph Neural Networks [1 ] [1]. Casas, Sergio, et al. "Spatially-aware graph neural networks for relational behavior forecasting from sensor data. " ar. Xiv preprint ar. Xiv: 1910. 08233 (2019).

4. Spatially-Aware Graph Neural Networks Gaussian MRFs & Gaussian Belief Propagation [1 ] [1]. Casas, Sergio, et al. "Spatially-aware graph neural networks for relational behavior forecasting from sensor data. " ar. Xiv preprint ar. Xiv: 1910. 08233 (2019).

4. Spatially-Aware Graph Neural Networks Gaussian MRFs & Gaussian Belief Propagation [1 ] [1]. Casas, Sergio, et al. "Spatially-aware graph neural networks for relational behavior forecasting from sensor data. " ar. Xiv preprint ar. Xiv: 1910. 08233 (2019).

4. Spatially-Aware Graph Neural Networks Gaussian MRFs & Gaussian Belief Propagation [1 ] Belief Propagation Marginal Distribution [1]. Casas, Sergio, et al. "Spatially-aware graph neural networks for relational behavior forecasting from sensor data. " ar. Xiv preprint ar. Xiv: 1910. 08233 (2019).

4. Spatially-Aware Graph Neural Networks [1 ] Graph Neural Networks（State Update） 迭代更新 1. Hidden State (Extracted ROI) 2. Node State (Statistics of Marginal Distribution) 分布 [1]. Casas, Sergio, et al. "Spatially-aware graph neural networks for relational behavior forecasting from sensor data. " ar. Xiv preprint ar. Xiv: 1910. 08233 (2019).

4. Spatially-Aware Graph Neural Networks [1 ] Message Passing MLP*3 State Update GRU MLP*2 [1]. Casas, Sergio, et al. "Spatially-aware graph neural networks for relational behavior forecasting from sensor data. " ar. Xiv preprint ar. Xiv: 1910. 08233 (2019).

4. Spatially-Aware Graph Neural Networks Ga. BP Belief Propogation State Update [1 ] GNN Message Passing State Update [1]. Casas, Sergio, et al. "Spatially-aware graph neural networks for relational behavior forecasting from sensor data. " ar. Xiv preprint ar. Xiv: 1910. 08233 (2019).

4. Spatially-Aware Graph Neural Networks [1 ] Training Results [1]. Casas, Sergio, et al. "Spatially-aware graph neural networks for relational behavior forecasting from sensor data. " ar. Xiv preprint ar. Xiv: 1910. 08233 (2019).

References [1]. Ngiam, Jiquan, et al. "Starnet: Targeted computation for object detection in point clouds. " ar. Xiv preprint ar. Xiv: 1908. 11069 (2019). [2]. Qi, Charles R. , et al. "Pointnet: Deep learning on point sets for 3 d classification and segmentation. " Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. [3]. Qi, Charles Ruizhongtai, et al. "Pointnet++: Deep hierarchical feature learning on point sets in a metric space. " Advances in neural information processing systems. 2017. [4]. Wang, Shenlong, et al. "Deep parametric continuous convolutional neural networks. " Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018. [5]. Zaheer, Manzil, et al. "Deep sets. " Advances in neural information processing systems. 2017. [6]. Engelcke, Martin, et al. "Vote 3 deep: Fast object detection in 3 d point clouds using efficient convolutional neural networks. " 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2017. [7]. Li, Bo. "3 d fully convolutional network for vehicle detection in point cloud. " 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017. [8]. Luo, Wenjie, Bin Yang, and Raquel Urtasun. "Fast and furious: Real time end-to-end 3 d detection, tracking and motion forecasting with a single convolutional net. " Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2018. [9]. Li, Bo, Tianlei Zhang, and Tian Xia. "Vehicle detection from 3 d lidar using fully convolutional network. " ar. Xiv preprint ar. Xiv: 1608. 07916 (2016). [10]. Chen, Xiaozhi, et al. "Multi-view 3 d object detection network for autonomous driving. " Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. [11]. Meyer, Gregory P. , et al. "Lasernet: An efficient probabilistic 3 d object detector for autonomous driving. " Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019. [12]. Yang, Bin, Wenjie Luo, and Raquel Urtasun. "Pixor: Real-time 3 d object detection from point clouds. " Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2018. [13]. Yang, Bin, Ming Liang, and Raquel Urtasun. "Hdnet: Exploiting hd maps for 3 d object detection. " Conference on Robot Learning. 2018. [14]. Yang, Zetong, et al. "Std: Sparse-to-dense 3 d object detector for point cloud. " Proceedings of the IEEE International Conference on Computer Vision. 2019.

References [15]. Liang, Ming, et al. "Multi-task multi-sensor fusion for 3 d object detection. " Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019. [16]. Lee, Namhoon, et al. "Desire: Distant future prediction in dynamic scenes with interacting agents. " Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. [17]. Rhinehart, Nicholas, Kris M. Kitani, and Paul Vernaza. "R 2 p 2: A reparameterized pushforward policy for diverse, precise generative path forecasting. " Proceedings of the European Conference on Computer Vision (ECCV). 2018. [18]. Hu, Yeping, Wei Zhan, and Masayoshi Tomizuka. "Probabilistic prediction of vehicle semantic intention and motion. " 2018 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2018. [19]. Ma, Wei-Chiu, et al. "Forecasting interactive dynamics of pedestrians with fictitious play. " Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017. [20]. Alahi, Alexandre, et al. "Social lstm: Human trajectory prediction in crowded spaces. " Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. [21]. Zhao, Tianyang, et al. "Multi-agent tensor fusion for contextual trajectory prediction. " Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019. [22]. Gupta, Agrim, et al. "Social gan: Socially acceptable trajectories with generative adversarial networks. " Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018. [23]. Casas, Sergio, et al. "Spatially-aware graph neural networks for relational behavior forecasting from sensor data. " ar. Xiv preprint ar. Xiv: 1910. 08233 (2019). [24]. Sadeghian, Amir, et al. "Car-net: Clairvoyant attentive recurrent network. " Proceedings of the European Conference on Computer Vision (ECCV). 2018. [25]. Santoro, Adam, et al. "A simple neural network module for relational reasoning. " Advances in neural information processing systems. 2017. [26]. Sun, Chen, et al. "Actor-centric relation network. " Proceedings of the European Conference on Computer Vision (ECCV). 2018. [27]. Sun, Chen, et al. "Relational action forecasting. " Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.

Q & A 2020. 07. 21 刘天禹