CVPR 2019 Squeeze NAS Fast neural architecture search

CVPR 2019 Squeeze. NAS: Fast neural architecture search Edit Master text styles Second level for faster semantic segmentation Third level Fourth level Fifth level. Sammy Sidhu, Forrest Iandola Albert Shaw, Daniel Hunter, and the Deep. Scale Team CVPR Workshop on Autonomous Driving Monday, June 17, 2019 — Terrace Theater

CVPR 2019 Overview 1. Motivating the Search for Efficient DNNs Edit Master text styles Second level 2. Related Third Work level on Neural Architecture Search (NAS) Fourth level 3. Applying NAS design DNNs for semantic segmentation Fifthtolevel 4. Squeeze. NAS Results 5. Future Work and Conclusions 6. Questions 2 — Squeeze. NAS

CVPR 2019 Levels of automated driving Edit Master text styles Second level Third level Fourth level. Assistance Advanced. Fifth Driver (e. g. Tesla Autopilot) LEVEL Driver Assistance LEVEL Partial Automation LEVEL Conditional Automation LEVEL High Automation LEVEL Full Automation 1 2 3 Robo-taxis, robo-delivery, … 3 4 5

CVPR 2019 OFFLINE MAPS Edit Master text styles Second level Third level Fourth level Fifth level SENSORS CAMERA RADAR THE FLOW ULTRASONIC LIDAR IMPLEMENTING AUTOMATED DRIVING 4 PATH PLANNING & ACTUATION REAL-TIME PERCEPTION

CVPR 2019 Deep learning is used in the best perception systems for automated driving 180 x higher productivity with deep learning Edit Master text styles Chris Urmson, CEO of Aurora: With deep learning, an engineer can accomplish in one day what would Second level take 6 months of engineering effort with traditional algorithms. [1] Third level Fourth level 100 x fewer errors with deep learning Fifth level Dmitri Dolgov, CTO of Waymo: "Shortly after we started using deep learning, we reduced our error-rate on pedestrian detection by 100 x. "[3] Deep learning has become the go-to approach Andrej Karpathy, Sr Director of AI at Tesla: "A neural network is a better piece of code than anything you or I could create for interpreting images and video. "[2] 5 [1] https: //www. nytimes. com/2018/01/04/technology/self-driving-cars-aurora. html [2] https: //medium. com/@karpathy/software-2 -0 -a 64152 b 37 c 35 [3] https: //medium. com/waymo/google-i-o-recap-turning-self-driving-cars-from-science-fiction-into-reality-with-the-help-of-ai-89 dded 40 c 63

CVPR 2019 We don't just need deep learning… We need efficient deep learning Edit Master text styles Second level Third level Fourth level Fifth level Audi https: //www. slashgear. com/man-vs-machine-my-rematchagainst-audis-new-self-driving-rs-7 -21415540/ 6 BMW + Intel https: //newsroom. intel. com/news-releases/bmwgroup-intel-mobileye-will-autonomous-test-vehiclesroads-second-half-2017/ Waymo

CVPR 2019 We don't just need deep learning… We need efficient deep learning Edit Master text styles Second level Third level Fourth level Fifth level 7 Trunkloads of servers cause problems: • • • Limited trunk space Cost Energy usage Reduced EV battery range Lower reliability Massive heat dissipation

CVPR 2019 From high-end hardware to affordable hardware Edit Master text styles Second level Third level Fourth level Fifth level 8 • 30 to 500 watts • 500 s-5000 s+ of dollars • 10 s-100 s of TOPS/s • 1 to 30 watts (for chip + memory + I/O) • 10 s of dollars • 1 s of TOPS/s

CVPR 2019 Tradeoffs for deployable DNN models for automotive deep learning practitioners Edit Master text styles Second level Third level Fourth level Fifth level Low Under-provisioned less-accurate DNNs Low Compute Resource Usage 9 Development Cost Benchmark-winning off-the-shelf DNNs Low Error Manually design a new DNN from scratch

CVPR 2019 Neural Architecture Search (NAS) to the Rescue NAS can co-optimize resource-efficiency and accuracy Edit Master text styles Second level Third level Fourth level Fifth level Low Under-provisioned less-accurate DNNs Development Cost Benchmark-winning off-the-shelf DNNs Neural Architecture Search (NAS) Low Compute Resource Usage 10 Low Error Manually design a new DNN from scratch

CVPR 2019 Edit Master text styles Second level Third level Fourth level Fifth level 11 Related work on Neural Architecture Search

CVPR 2019 Neural Architecture Search • We want to find networks that are the best tradeoff of compute and accuracy in some search space Edit Master text styles Second level Third level Fourth level Fifth level • Within some search space we can vary the number of layers, channels, kernel size, connections, etc… • It is beyond intractable to train every one • Can we do better? • • 12 Random Search Genetic Search Reinforcement Learning Differential Search (Gradient Based)

CVPR 2019 NAS with Reinforcement Learning Block-level search [1] Edit Master text styles Second level Third level Fourth level Fifth level • Use a Recurrent Neural Network in a RL loop to generate entire child network for the CIFAR dataset updating after each model has trained • Achieved 0. 09% better accuracy at the time and 1. 05 x faster on CIFAR-10 • 800 Nvidia K 40 GPUs for 28 days = 537, 600 GPU Hours • Search performed on small dataset • Better than brute force approach but still too much compute too be practical [1] B. Zoph, Q. Le. Neural Architecture Search with Reinforcement Learning. ICLR, 2018. 13

CVPR 2019 NAS with Reinforcement Learning Cell-level search [2] Edit Master text styles Second level Third level Fourth level Fifth level • Use a Recurrent Neural Network in a RL loop to generate cells using CIFAR 10 as proxy task then adapted to Image. Net • Achieved 1. 20% better accuracy while being 28% faster on Image. Net 1000 • 500 Nvidia P 100 GPUs for 4 days = 48, 000 GPU Hours • cells are all the same (unlike [1]) • More efficient than previous method but still expensive [1] B. Zoph, Q. Le. Neural Architecture Search with Reinforcement Learning. ICLR, 2018. [2] B. Zoph et al. Learning Transferable Architectures for Scalable Image Recognition. CVPR, 2018. 14

CVPR 2019 Differential Search FBNet [3] Edit Master text styles Second level Third level Fourth level Fifth level • Use a gradient based stochastic supernet that is dually optimized on a 10% subset of Image. Net which is then trained on Image. Net 1 k • Uses Gumbel-Softmax to sample from categorical distribution for layer choices weighted by learnable parameters • FBNet-B achieved Mobile. Net. V 2 -1. 3 Accuracy while being 1. 5 x lower latency • 216 P 100 GPU hours Search Cost • Search Space inspired by Mobilenet. V 2 [3] B. Wu, et al. FBNet: Hardware-Aware Efficient Conv. Net Design via Differentiable Neural Architecture Search. CVPR, 2019. 15

CVPR 2019 Other Related Work • Genetic Search • Amoeba. Net[1] Tournament Selection Evolutionary on Cell Space Edit Master • text styles • level 75, 600 K 40 GPU hours Second Third level • Reinforcement Learning Fourth level • Mnas. Net[2] Fifth level • Latency Aware Block level Search on proxy Image. Net • 6912 TPUv 2 hours ≈ 48, 000 P 100 GPU hours • Differential Search • DARTS[3] • Gradient Based Cell Search performed on CIFAR-10 • 96 1080 TI GPU hours 16 [1] E. Real et al. Regularized Evolution for Image Classifier Architecture Search. AAAI, 2019. [2] M. Tan et al. Mnas. Net: Platform-Aware Neural Architecture Search for Mobile. CVPR, 2019. [3] H. Liu et al. DARTS: Differentiable Architecture Search. ICLR, 2019.

CVPR 2019 Edit Master text styles Second level Third level Fourth level Fifth level 17 Applying NAS to design DNNs for semantic segmentation

CVPR 2019 Classification vs Semantic Segmentation Edit Master text styles Second level Third level Fourth level Fifth level Examples of image classification (Image. Net[1]) • • Image level prediction Location Invariant Low Resolution (224 x 224 input) SOTA Networks compute: ~10 GFLOPs • Networks designed for task and are trained from scratch 18 Example of Semantic Segmentation (Cityscapes[2]) • • Pixel level prediction Location Variant High Resolution (1024 x 2048 input) SOTA Networks range: ~1 TFLOPS • SS Networks are adapted from classification networks and then retrained. [1] O. Russakovsky et al. Image. Net Large Scale Visual Recognition Challenge. IJCV, 2015. [2] M. Cordts et al. The Cityscapes Dataset for Semantic Urban Scene Understanding. CVPR, 2016.

CVPR 2019 Classification vs Semantic Segmentation (cont. ) Edit Master text styles Second level Third level Fourth level Fifth level Examples of image classification • • Image level prediction Location Invariant Low Resolution (224 x 224 input) SOTA Networks compute: ~10 GFLOPs • Networks designed for task and are trained from scratch 19 Example of Semantic Segmentation (Deep. Lab. V 3[1]) • • Pixel level prediction Location Variant High Resolution (1024 x 2048 input) SOTA Networks range: ~1 TFLOPS • SS Networks are adapted from classification networks and then retrained. [1] LC. Chen et al. Rethinking Atrous Convolution for Semantic Image Segmentation, 2017.

CVPR 2019 Applying NAS to design DNNs for semantic segmentation • Goal: advance the frontier of accuracy/efficiency on Semantic Segmentation Edit Master text styles Second level • Related Third work level typically starts with Image. Net the DNNFourth for semantic segmentation level Fifth level training, then adapts and fine-tunes • If we do NAS directly on sematic segmentation (rather than starting with Image. Net), will this give us better results? 20 Examples of image classification (Image. Net) Example of Semantic Segmentation (Cityscapes)

CVPR 2019 Squeeze. NAS: An Adaptation of FBNet for Semantic Segmentation Search • Stochastic Super Network • Run allstyles units in parallel Edit Master text • Perform Second level weighted sum of activations where weights are Third levelfrom Gumbel-Softmax sampled Fourth level parameters: Convolution parameters • 2 types of learned Fifth levelparameters and architecture • Resource aware learned architecture parameter • A unit in the meta-network is chosen by its architecture parameter plus a random variable • Optimize parameters of network (convolutions) and parameters for unit choices in an alternating fashion • Proxyless training 21 • We train directly on high-resolution cityscapes • Training until both convolution parameters and unit parameters converge Figure courtesy of Bichen Wu

CVPR 2019 Training scheme FBNet training flow Evaluate candidates on Image. Net-100 Validation set Select best DNNs; train them on Image. Net-1 k (classification) Sample candidate networks from Super. Network Evaluate candidates on Cityscapes fine Validation set Select best DNNs; train them on Image. Net-1 k (classification) Finetune on Cityscapes Fine (segmentation) Finetune on Cityscapes Coarse (segmentation) Finetune on COCO (segmentation) NAS on Sample candidate Edit Master text styles Image. Net-100 networks from Second level (classification) Super. Network Third level Fourth level Fifth level Squeeze. NAS training flow Super. Network Training on Cityscapes Fine (segmentation) 22

CVPR 2019 3 x 3 Expansion 6 Edit Master text styles Second level Third level Fourth level Expansion 3 Fifth level Search Space Expansion 1 (grouped conv) 23 3 x 3 dilated 5 x 5 skip

CVPR 2019 We employ the encoder-decoder depthwise head from Deep. Lab V 3+[1] while allowing the base network to be completely learned Edit Master text styles Second level Third level Fourth level Fifth level Squeeze. NAS Search Space (cont) [1] Chen et al. Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation, ECCV 2018 24

CVPR 2019 Edit Master text styles Second level Third level Fourth level Fifth level 25 Squeeze. NAS Results

CVPR 2019 Squeeze. NAS: City. Scapes Results Name Edit Master text styles Second level Squeeze. NAS-23 Squeeze. NAS-9 Third level Fourth level Mobile. Net. V 2[4] Fifth level Squeeze. NAS-3 EDANet[3] CCC 2[2] Enet[1] CCC DRN A 50[6] MACs (Billions) Class m. IOU on Cityscapes Squeeze. NAS-3 3. 5 67. 5 Squeeze. NAS-9 9. 4 72. 4 Squeeze. NAS-23 23. 2 74. 8 Enet[1] 4. 4 58. 3 CCC 2[2] 6. 3 62. 0 EDANet[3] 9. 0 65. 1 Mobile. Net. V 2 OS=16[4] 21. 3 [5] 70. 7 [5] CCC DRN A 50[6] 68. 7 26 [1] Paszke, Adam et al. ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation, 2016 [2] Park, Hyojin et al. Concentrated-Comprehensive Convolutions for lightweight semantic segmentation, 2018 [3] Lo, Shao-Yuan et al. Efficient Dense Modules of Asymmetric Convolution for Real-Time Semantic Segmentation, 2018 [4] Sandler, Mark et al. Mobile. Net. V 2: Inverted Residuals and Linear Bottlenecks, CVPR 2018. [5] https: //github. com/tensorflow/models/blob/master/research/deeplab/g 3 doc/model_zoo. md [6] Yu, Fisher et al. Dilated Residual Networks, CVPR 2017. 67. 6

CVPR 2019 Legend (Unit Type) 1 x 1 ->unit->1 x 1 3 x 3 5 x 5 Box Width represents channel expansion skip downsample m. IOU % 21. 3 70. 71 Squeeze. NAS-3 3. 5 67. 5 Squeeze. NAS-9 9. 4 72. 4 Squeeze. NAS-23 23. 2 74. 8 Edit Master text styles Second level Third level S q u e e z e NFourth A S level Mobile. Net. V 2 Fifth level. Deep. Lab. V 3 27 3 x 3 dilated 5 x 5 MACs (Giga) Mobile. Net. V 2 Classification Resulting Networks 3 x 3

CVPR 2019 Analysis of Networks • Dilation (Atous) Convolutions use later in the network • All of our Searches led to heavily dilated networks towards the head with no priors • This follows Edit Master text styles human intuition from many SOTA segmentation papers (Deep. Lab V 3) • Second level Downsample Third level Units Fourth level • Our networks also typically picked the most computationally expensive unit when Fifth level performing a Downsample (strided unit) • This is different from typical networks which usually perform a 3 x 3 kernel when performing a strided convolution • Non-Uniform Compute • Our networks varied the amount of compute each unit used unlike typical networks (Res. Net, Mobile. Net, etc) which use the same of amount of compute per unit throughout the network • The low-compute, high-compute alternating pattern could be explained by the network trading off spatial features vs pixel-wise features 28

CVPR 2019 Squeeze. NAS: Search Time Results Name Search Time (GPU Hours) Dataset Searched on Edit Master text styles Second level Third level Squeeze. NAS-3 Fourth level Squeeze. NAS-9 Fifth level Squeeze. NAS-23 gradient 152 City. Scapes gradient 248 City. Scapes gradient 316 City. Scapes Neural Architecture Search with Reinforcement Learning RL 537, 600 CIFAR-10 NASNet RL 48, 000 CIFAR-10 m. Nas. Net RL 48, 000* Proxy Image. Net Amoeba. Net genetic 75, 600 CIFAR-10 FBNet gradient 216 Proxy Image. Net DARTS gradient 96 CIFAR-10 * Approximated from TPUv 2 Hours 29 NAS Method

CVPR 2019 Squeeze. NAS: Implementation details • 13 candidates per unit for 22 layers (1024 possible networks) • Randomly initialize Super. Network and train only convolution weights early on then switch to Edit Master text styles alternating pattern • Second level Thirdin level We train FP 16 (half precision) for network and FP 32 for the architecture parameters. This also Fourth level cut GPU memory use in half and allowed us to leverage tensor cores Fifth level • We train the Super. Network on Cityscapes fine with 768 x 768 patches with flip, crop and scale augmentations • After Super. Network has converged: • Sample candidate networks via the Gumbel-Softmax • Run each sampled candidate network on the validation set of Cityscapes fine. • Put best of these candidates through the full Deep. Labv 3+[1] training Regime* • • • 30 Image. Net 90 epochs at 224 x 224 COCO 30 epochs at 768 x 768 City. Scapes Coarse (40 epochs) and Fine (100 epochs) with the same regime as the Super. Network [1] Chen et al. Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation, ECCV 2018

CVPR 2019 Future Work • Perform latency aware search with publicly available hardware platform • We are currently targeting automotive grade hardware, and we are under a strict NDA Edit Master textthe styles with hardware manufacturer. We're working on a solution for this. • Second level Third level Fourth level Since the time of this work, Mobile. Net. V 3[1] has released results on low Fifth level compute segmentation • Search alternative spaces for better results at lower compute • Release publicly available paper soon! 31 [1] A. Howard et al. Searching for Mobile. Net. V 3. ar. Xiv Preprint. May 2019

CVPR 2019 Conclusions • New compute vs accuracy curve created for a low compute Semantic Segmentation • NAS helps with this! Edit Master text styles Secondarchitecture level • Some patterns follow human intuition and some don’t Third level • We can learn new design paradigms from NAS Fourth level Fifth level • Smaller and well designed search spaces lead to shorter search time • Researchers can now design Neural Architecture search spaces instead of individual networks 32

CVPR 2019 Legend (Unit Type) 1 x 1 ->unit->1 x 1 3 x 3 5 x 5 Box Width represents channel expansion skip downsample m. IOU % 21. 3 70. 71 Squeeze. NAS-3 3. 5 67. 5 Squeeze. NAS-9 9. 4 72. 4 Squeeze. NAS-23 23. 2 74. 8 Edit Master text styles Second level Third level T h a n k y o. Fourth u ! level Mobile. Net. V 2 Fifth level Segmentation 33 3 x 3 dilated 5 x 5 MACs (Giga) Mobile. Net. V 2 Classification Questions? 3 x 3

CVPR 2019 Edit Master text styles Second level Third level Fourth level Fifth level 34 Extras