Scalpel Customizing DNN Pruning to the Underlying Hardware

Scalpel: Customizing DNN Pruning to the Underlying Hardware Parallelism Jiecao Yu 1, Andrew Lukefahr 1, David Palframan 2, Ganesh Dasika 2, Reetuparna Das 1, Scott Mahlke 1 1 University of Michigan – Ann Arbor ISCA, June 28, 2017 2 ARM Research

Deep Neural Networks • High storage and computation cost − Alex. Net: 240 MB, 1. 5 GFLOP − VGG: 550 MB, 30. 9 GFLOP • DNN compression is necessary − Removing unimportant parameters 2

DNN Architecture 0 0 1 0 0 9 6 0 2 4 0 1 4 3 0 0 0 5 8 0 0 0 1 7 0 X 0 2 1 = Output Vector 0 2 0 8 0 5 Input Vector Weight Matrix • High redundancy 3

Weight Pruning * Weight Matrix 0 2 0 8 0 5 0 0 1 0 0 9 6 0 2 4 0 1 4 3 0 0 0 5 8 0 0 0 1 7 0 0 2 1 |Weights| > Threshold Computation and storage * Han, Song, et al. "Learning both weights and connections for efficient neural network. " NIPS. 2015. 4

Drawbacks • Sparse format needs extra storage 0 1 2 3 4 5 8 5 9 6 4 3 5 7 8 4 3 CSR Format Weights A = 6 4 3 5 7 8 8 4 3 5 9 Column indexes JA = 2 3 3 4 5 4 0 2 3 0 1 IA = 0 2 5 6 9 9 11 • One column index for each weight 5

Drawbacks • Execution time increase − Computation reduction not fully utilized − Extra computation for decoding sparse format • Alex. Net * 4 Relative Model Size, Computation and Exec. Time 334% 3 2 1 125% 22% 42% Size Computation Unpruned Baseline 0 Time-CPU Time-GPU * Alex. Net not tested on microcontroller 6

Scalpel Trained DNN Low • Low parallelism - Micro. High Hardware Parallelism Moderate FC Layer Type SIMD-Aware Weight Pruning CONV Node Pruning − − No cache Low storage (~100 KB) • High parallelism - GPU − − TLP High bandwidth / long latency memory • Moderate parallelism CPU − ILP / MLP Pruned DNN 7

Matrix-Vector Multiplication with SIMD 0 0 0 6 0 0 4 0 1 0 0 0 3 0 2 5 0 4 0 0 0 7 0 0 0 1 0 x 1 + 0 x 2+ 0 = 0 6 x 1 + 0 x 2+ 0 = 6 5 x 1 + 7 x 1+ 6 = 18 2 X 1 2 1 1 • Assume SIMD width = 2 − Two loads / multiply-accumulate (MAC) in one instruction • SIMD benefits dense matrix-vector multiplication 8

Sparse Matrix-Vector Multiplication 0 0 0 0 1 0 0 0 2 6 0 0 4 0 1 3 0 0 0 3 0 2 4 5 0 4 0 0 0 5 7 0 0 0 Weights 1 X 6 5 7 4 4 3 1 2 2 Indexes 1 2 4 5 4 2 3 2 1 x + 6 x 1 + 0 = 1 x + 5 x 1 + 6 = x + 7 x 1 + 11 = 1 • SIMD not fully utilized • Extra storage for column indexes 9

Weights in Groups Weights 0 1 2 3 4 5 6 0 5 7 4 0 4 3 1 2 X 1 6 0 5 7 4 0 4 3 1 2 2 Indexes 1 2 4 4 2 2 2 1 6 x 1 + 0 x 2+ 0 = 6 1 5 x 1 + 7 x 1+ 6 = 18 • SIMD units fully utilized • Fewer column indexes 10

SIMD-Aware Weight Pruning • Start with original weight matrix 0 0 6 4 0 1 • Weights grouping 2 0 • Calculate importance of groups 8 − Root mean square 0 5 − Group size = SIMD width 0 1 0 0 9 0 2 4 0 1 3 0 0 5 8 0 0 0 7 0 0 2 1 • Remove redundant groups − Importance < Threshold Num of weights Model size Execution time 11

Performance Benefit – Low Parallelism • ARM Cortex-M 4 Microcontroller 48% Relative Exec. Time 2. 5 68% 2 1. 5 Dense Baseline 1 0. 5 0 0. 3 0. 4 0. 5 0. 6 0. 7 Pruning Rate Traditional Weight Pruning 0. 8 0. 9 1 SIMD-Aware Weight Pruning 12

Performance – High Parallelism • NVIDIA GTX Titan X − Traditional weight pruning 96. 3% Relative Exec. Time 2. 5 2 1. 5 Dense Baseline 1 0. 5 0 0. 92 0. 94 0. 96 0. 98 1 Pruning Rate • Sparsity hurts performance Pruning without sparsity 13

Node Pruning • Keep regular DNN structure: no sparsity • Remove redundant nodes − Nodes: neurons (FC) / feature maps (CONV) − Mask layers finds unimportant nodes • Add mask layers Mask Layer A • Train mask layers • Remove redundant nodes and mask layers 14

Combined Pruning – Moderate Parallelism Relative Exec. Time • Intel Core i 7 -6700 CPU • Impact of sparsity on computation performance FC CONV 2. 5 1. 2 84% 3% layers Dense 1 0. 8 0. 6 0. 4 0. 2 2 1. 5 Dense 1 0. 5 0 0. 5 Pruning Rate SIMD-Aware Weight Pruning SIMD-aware weight pruning 1 0. 7 0. 8 0. 9 Pruning Rate 1 SIMD-Aware Weight Pruning Node pruning 15

Evaluation Methodology • Networks − MNIST: Le. Net-300 -100, Le. Net-5 − CIFAR-10: Conv. Net, Network-in-Network − Image. Net: Alex. Net • Hardware − Low-parallelism: ARM Cortex-M 4 microcontroller − Moderate-parallelism: Intel Core i 7 -6700 CPU − High-parallelism: NVIDIA GTX Titan X • Baseline: Traditional weight pruning * * Han, Song, et al. "Learning both weights and connections for efficient neural network. " NIPS. 2015. 16

Results: ARM Cortex-M 4 microcontroller Relative Exec. Time Relative Model Size 1. 2 1. 0 0. 5 1 1. 0 0. 4 0. 8 0. 3 0. 6 0. 2 0. 4 0. 1 0. 2 • 28% execution time ea n t ge om Ne ex Al N NI Co nv Ne t-5 Le 00 t-3 om ge Scalpel -1 0 0 Traditional Le Ne ea n t Ne ex Al N NI t Co nv Ne Ne t-5 Le 00 -1 00 t-3 Le Ne Original Ne t 0 0 Original Traditional Scalpel • 12% model size 17

Results: Intel Core i 7 -6700 CPU Relative Exec. Time Relative Model Size 2. 7 1. 4 1. 2 1 0. 8 0. 6 0. 4 0. 2 0 1. 2 1 0. 8 0. 6 0. 4 ea n t ge om Ne ex Al N NI Co nv Ne Ne t-5 Le Ne Le t-3 00 -1 0 om ge t 0 0 ea n t Ne ex Al N NI t Co nv Ne Ne t-5 Le Le Ne t-3 00 -1 0 0 0. 2 Original Traditional Scalpel • 38% execution time Original Traditional Scalpel • 18% model size 18

Results: NVIDIA GTX Titan X Relative Exec. Time Relative Model Size 4 3. 5 3 2. 5 2 1. 5 1 0. 5 0 1. 4 1. 2 1 0. 8 0. 6 0. 4 0. 2 ea n t ge om Ne ex Al N NI t Co nv Ne Ne t-5 Le -1 0 00 Le Ne t-3 om ge 0 ea n t Ne ex Al N NI t Co nv Ne Ne t-5 Le Le Ne t-3 00 -1 00 0 Original Traditional Scalpel • 80% execution time Original Traditional Scalpel • 47% model size 19

Conclusions • Traditional weight pruning has drawbacks − Sparse format needs extra storage − Execution time increase • Scalpel customizes pruning for hardware with different parallelism − SIMD-aware weight pruning: utilizing SIMD units − Node pruning: avoiding sparsity • Microcontroller / CPU / GPU: − Traditional: 53%, 94%, and 242% execution time − Scalpel: 28%, 38%, and 80% execution time 20

Q & A

Thank you!

Networks Models Num of Layers FC CONV Le. Net-300 -100 0 3 Le. Net-5 2 2 Conv. Net 3 1 Network-In-Network 9 0 Alex. Net 5 3 Dataset MNIST CIFAR-10 Image. Net Error Rate 1. 50% 0. 68% 18. 14% 10. 43% 19. 73% (top-5) 23

Group Importance Measurement Relative Accuracy (%) • Maximum absolute value (MAX) • Mean absolute value (MEAN) • Root-mean-square (RMS) 98% 0. 10 0. 00 -0. 10 -0. 20 0. 88 0. 92 0. 94 0. 96 0. 98 1 Pruning Rate of fc 1 in Le. Net-300 -100 RMS MEAN MAX 24

Nodes Removed in Network-In-Network 200 150 100 50 0 conv 1 cccp 2 conv 2 cccp 3 Remaining cccp 4 conv 3 cccp 5 cccp 6 Removed 25

Results: High Parallelism Relative Exec. Time Relative Model Size 4 3. 5 3 2. 5 2 1. 5 1 0. 5 0 1. 4 1. 2 1 0. 8 0. 6 0. 4 0. 2 ea n t ge om Ne Al ex N NI t Ne nv Co Ne t-5 Le -1 0 00 Le Ne t-3 om ge 0 ea n t Ne Al ex N NI t Ne nv Co Ne t-5 Le Le Ne t-3 00 -1 0 0 0 Original Traditional Optimized • 80% execution time Scalpel Original Traditional Optimized Scalpel • 47% model size NVIDIA GTX Titan X 26

Combined Pruning – Moderate Parallelism • Intel Core i 7 -6700 CPU • Impact of sparsity on computation performance Relative Exec. Time 1. 2 1 CONV layers 84% 2. 5 Dense Relative Exec. Time FC 3% layers 0. 8 0. 6 0. 4 0. 2 2 1. 5 Dense 1 0. 5 0 0. 5 Pruning Rate SIMD-Aware Weight Pruning • SIMD-aware weight pruning 1 0. 7 0. 8 0. 9 Pruning Rate 1 SIMD-Aware Weight Pruning • Node pruning 27