Dynamic Image Networks for Action Recognition H Bilen

Dynamic Image Networks for Action Recognition H. Bilen * U. of Oxford A. Vedaldi E. Gavves B. Fernando * Australian N. U. U. of Amsterdam U. of Oxford * Equal Contribution 12/5/2020 CVPR 2016 Las Vegas S. Gould Australian N. U.

12/5/2020 CVPR 2016 Las Vegas 2

12/5/2020 CVPR 2016 Las Vegas 3

Dynamic images: A simple but effective motion representation 1. Compact: summarize a video into a single frame 2. Expressive: long-term dynamics 3. Efficient: a novel rank pooling layer 4. End-to-end learning: backpropagation 12/5/2020 CVPR 2016 Las Vegas 4

Representations for deep action recognition Sequence of still images [Karpathy@CVPR’ 14] Short-term motion 3 D conv [Ji@ICML’ 10, Tran@ICCV’ 15] Short-term motion Two stream network [Simonyan@NIPS’ 14] Short-term motion 12/5/2020 CVPR 2016 Las Vegas 5

From pixels to dynamics via rank pooling • compatible to standard CNN architectures • end-to-end learning • efficient Learning to Rank t [Fernando@CVPR’ 15] 12/5/2020 CVPR 2016 Las Vegas 6

Dynamic Images 12/5/2020 Skiing Cycling Golf. Swing Blow. Dry. Hair CVPR 2016 Las Vegas 7

Learning to rank • 12/5/2020 CVPR 2016 Las Vegas 8

Rank pooling layer • Too slow for CNN training! Non-trivial backpropagate t t 12/5/2020 CVPR 2016 Las Vegas 9

Single / Multi Dynamic Image Networks conv 5 conv 4 conv 3 conv 2 conv 1 fc 8 fc 7 fc 6 max pool conv 5 conv 4 conv 3 conv 2 conv 1 Approximate Rank Pooling conv 5 conv 4 conv 3 conv 2 conv 1 Approximate Rank Pooling t 12/5/2020 CVPR 2016 Las Vegas 10

Dynamic Map Networks Dynamic Map Net • Fast (approximate) rank pooling • Back-propagation is possible fc 8 fc 7 fc 6 conv 5 conv(k+1) Approximate Rank Pooling conv(k) conv 1 t 12/5/2020 CVPR 2016 Las Vegas 11

Experiments • Experiments on 2 standard human action recognition datasets HMDB 51: 6849 clips, 51 categories UCF 101: 13320 clips, 101 categories • Image. Net pre-trained Alex-net 12/5/2020 CVPR 2016 Las Vegas 12

Results – UCF 101 89. 1 90 Accuracy (%) 80 68. 7 70 60 50 70. 9 76. 9 57. 9 52. 6 45. 6 40 30 Max Sum Rank (DI) Multi DI Pooling 12/5/2020 +Fine-tune +RGB +TRJ (Wang 13) Further improvements CVPR 2016 Las Vegas 13

0 Results: Best & Worst Categories • Top 5 classes based on relative performances for Dynamic vs Static networks Soccer. Juggling 38. 5 Cricket. Shot Clean. And. Jerk Pull. Ups 36. 4 32. 1 Drumming Playing. Piano -25. 6 -22 Push. Ups Pizza. Tossing 26. 7 25 Playing. Flute -21. 4 Fencing -18. 2 -47. 9 0 12/5/2020 CVPR 2016 Las Vegas 14

Comparison to state-of-the-art UCF 101 86 88 89. 1 89. 6 89. 1 50 55 60 65 Wang 13 70 75 Simonyan 14 Lan 15 80 85 Zha 15 Ours 90 95 HMDB 51 59. 4 30 35 40 Simonyan 14 12/5/2020 45 Fernando 15 50 55 Wang 13 Lan 15 CVPR 2016 Las Vegas 63. 7 65. 4 66. 8 65. 2 60 Peng 14 65 70 Ours 15

Summary Dynamic Images • • Easy design and fine-tuning (Alexnet, VGGnet, …) Compact Efficient Store Sports 1 M in 20 GB only instead of 10 TB! Code is available (data, models after CVPR) • https: //github. com/hbilen/dynamic-image-nets • Including dynamic image networks pre-trained on Sports-1 M 12/5/2020 CVPR 2016 Las Vegas 16

Thank you! 12/5/2020 CVPR 2016 Las Vegas 17