DeepAligned Convolutional Neural Network for Skeletonbased Action Recognition

Deep-Aligned Convolutional Neural Network for Skeleton-based Action Recognition and Segmentation Babak Hosseini 1, 2, Romain Montagne 3, Barbara Hammer 1 1 Bielefeld University, 2 Dortmund University 3 Eurodecision, Twitter: @Babak_hss ICDM 2019, November 10, 2019

Outline: • Introduction • Proposed Architecture • Experiments • Conclusion Slide 2 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Outline: • Introduction • Proposed Architecture • Experiments • Conclusion Slide 3 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Skeleton-based action recognition & segmentation: • Segmentation and recognition walk run lean walk Stand dance Jug wave walk Slide 4 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Skeleton-based action recognition: • Segmentation and recognition walk • Multichannel time-series • Not image ! • Not video ! run lean walk Stand dance Jug wave walk Slide 5 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

State-of-the-art: • Many complex deep architectures • Combination of (CNN, LSTM, RL, GCN, spatial proc. ) [1] [2] [3] [4] • [1] Tang, et. al. , Deep Progressive Reinforcement Learning for Skeleton-based Action Recognition, CVPR 2018 • [2] Kim, et. al. , Interpretable 3 D Human Action Analysis with Temporal Convolutional Networks, CVPR 2017 • [3] Si, et. al. Skeleton-Based Action Recognition with Spatial Reasoning and Temporal Stack Learning, ECCV 2018 • [4] Sun, et. al. Human Action Recognition using Factorized Spatio-Temporal Convolutional Networks Lin, ICCV 2015 Babak Hosseini, Romain Montagne, Barbara Hammer Slide 6 ICDM 2019, November 10, 2019

State-of-the-art: • Difficult to interpretation/understand the model • Hand coded skeleton model • Lego-combinations Not really intuitive Slide 7 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Time-series classification: • 1 D Time-series shapelets [1] • Shapelets as discriminative prototypes • Semantic interpretation [1] Lexiang Ye, Eamonn Keogh, Time Series Shapelets: A New Primitive for Data Mining, KDD 2009 Babak Hosseini, Romain Montagne, Barbara Hammer Slide 8 ICDM 2019, November 10, 2019

i i i+2 time Slide 9 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Our objective: • A convolutional network for action Rec. &Seg. which benefits from: • Time-series shapelets • Time-series alignement kernel i i i+2 time Slide 10 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Outline: • Introduction • Proposed Architecture • Experiments • Conclusion Slide 11 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Network’s structure: Slide 12 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Alignment-filters: • 1 D filters • Convolution Euclidian distance • Each filter applied to all channels • Feature map: Peeks of matching cases input channel feature map scanning the sequence Slide 13 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Slide 14 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Abstract-filters: • Longer filters • Sparser f-map • Abstract information scanning the sequence Slide 15 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Abstract-filters: • Longer filters • Sparser f-map • Abstract information Wanted: • Without adding more weights • Without pre-assumptions for lengths • Finding them during training scanning the sequence Slide 16 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Abstract-filters: • Longer filters • Sparser f-map • Abstract information • Using same weight from Al-filters Wanted: • Without adding more weights • Without pre-assumptions for lengths • Finding them during training Base filter Abs-filter Slide 17 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Abstract-filters: • Using weights of Al-filters • Consecutive activation threshold • Shortest paths between connected nodes • Fusion of Abs-alignment maps • Augmented alignment map: • Sparser patterns Slide 18 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Alignment kernels: Slide 19 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

1 -dimensional CNN architecture: Slide 20 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

1 -dimensional CNN architecture: • Each deep layer: • Two 1 D conv. filters • ELU activation • 1 D max pooling Slide 21 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Slide 22 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

1 -dimensional CNN architecture: • Cross-entropy loss • Vanishing gradient: • Modified alignment function • Conditional gradient function Slide 23 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Fine-prediction: Slide 24 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Fine-prediction: • Abstract alignment maps: • Sparse information • Important patterns • Long segments More consistent segmentation (temporally) Slide 25 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Upsampling path (1 D CNN) Slide 26 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Whole architecture: Slide 27 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Outline: • Introduction • Proposed Architecture • Experiments • Conclusion Slide 28 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Datasets: • Human skeleton-based action benchmarks • CMU Mocap segmentation (multiple actions per seq) • Montalbano V 2 segmentation (multiple actions per seq) • SYSU-3 D dataset Action classification (one action per seq) • NTU dataset Action classification (one action per seq) Slide 29 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Alternative methods: • Skeleton-based action recognition methods • • Two-steam CNN (2015) CNN+LSTM (2017) SR-TSL (2018) ST-GCN (2018) CNN-DPRL (2018) … Action segmentation methods • • • HACA (2013) RNN+CRF (2016) Conv. S 2 S (2017) CNN+LSTM (2018) … Slide 30 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Segmentation result: • CMU mocap DACNN-nf: no fine-prediction Slide 31 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Accuracy result: • Average classification accuracy (%) • SYSU-3 D : 1 st rank • NTU : 2 nd and 3 rd ranks • SR-TSL & CNN+LSTM: rely on the spatial processing of human poses Slide 32 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Interpreting abstract filters: • Abstract filters (long filters) • Associate them with specific classes • Prototypes visualization • NTU dataset • Red joints matched Abs-filters 9 frm 12 frm 15 frm Slide 33 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Conclusion: A deep-aligned convolutional neural network for skeleton-based action recognition: • New concept of temporal alignment filters • An effective choice for temporal data compared to conv. Filters • Finding discriminative temporal patterns in the data • Filters themselves are interpretable regarding their semantic content • Our fully convolutional architecture is effective for temporal segmentation Slide 34 Babak Hosseini, Romain Montagne, Barbara Hammer ICDM 2019, November 10, 2019

Thank you very much! Questions please! Twitter: @Babak_hss 35