Multiobjective Precision Optimization of DNNs for Edge Devices
Multi-objective Precision Optimization of DNNs for Edge Devices Nhut-Minh Ho, Ramesh Vaddi and Weng-Fai Wong
Introduction • Deep NN accelerator’s boom in recent years • Various approximation techniques applied • Edge devices : • Floating point => fixed point inference engine • => reduced fixed point precision • Tolerate classification error, how many bits to represent each layer? https: //medium. com/@Raghav. Prabhu/understanding-of-convolutional-neural-network-cnn-deep-learning-99760835 f 148 30 September 2020 Multi-objective Precision Optimization of DNNs for Edge Devices 2
Motivation • Prior work: • Coarse Granularity : whole network 1 bitwidth • Long analysis time (searching), some not published their search time • This work: • Layer-wise granularity • Fast analysis time • Complexity ~ O(L) times running the DNN, L: number of layers • One bitwidth fits all hardware designs? • Parameterize hardware constraints to precision tuning. 30 September 2020 Multi-objective Precision Optimization of DNNs for Edge Devices 3
Outline r re aye l r e nt I ip sh n o i t la rro E d r a Forw ion t a g pa r pro Classification Accuracy Output error Input error Fixed point rounding ard w k c Ba ysi l a n a s Multiple solutions 30 September 2020 Multi-objective Precision Optimization of DNNs for Edge Devices 4
Fixed-point rounding (of a value) Fixed point number I. F: I: Integer bitwidth F: Fractional bitwidth Float 30 September 2020 Fixed point 3 fraction bits The error (the noise) Multi-objective Precision Optimization of DNNs for Edge Devices 5
Fixed-point rounding (of arrays) Data in fixedpoint (I_bits. F_bits) e. g. 5. 1 bits Array of data (N elements) log 2 30 September 2020 = e. g. 80, 000 elements Multi-objective Precision Optimization of DNNs for Edge Devices 6
Fixed-point rounding (of feature maps) N images N samples ≈ 30 September 2020 Multi-objective Precision Optimization of DNNs for Edge Devices 7
Propagation of rounding error (cont) • Dot products Property 1 • Dot product with W (W : constants w/o rouding error) • Y = dot(X, W) 30 September 2020 Cascading dot products + popular RELUs + poolings, output error still looks like normal distribution (excluding zeroes) Multi-objective Precision Optimization of DNNs for Edge Devices 8
Propagation of rounding error (cont) • Full DNN https: //medium. com/@Raghav. Prabhu/understanding-of-convolutional-neural-network-cnn-deep-learning-99760835 f 148 30 September 2020 Multi-objective Precision Optimization of DNNs for Edge Devices 9
Near-linear relationship Property 2 RELU + {max, average} pooling + batch norm do not affect the near-linear relationship VGG 30 September 2020 Multi-objective Precision Optimization of DNNs for Edge Devices 10
Propagation of rounding error (cont) Measurable constants • Full DNN https: //medium. com/@Raghav. Prabhu/understanding-of-convolutional-neural-network-cnn-deep-learning-99760835 f 148 30 September 2020 Multi-objective Precision Optimization of DNNs for Edge Devices 11
Measurement Forward pass Repeat 20 times for linear regression https: //medium. com/@Raghav. Prabhu/understanding-of-convolutional-neural-network-cnn-deep-learning-99760835 f 148 30 September 2020 Multi-objective Precision Optimization of DNNs for Edge Devices 12
Applications • Combining the effect of rounding on different layers Property 3 Forward pass Additive effect of multiple independent noise sources. https: //medium. com/@Raghav. Prabhu/understanding-of-convolutional-neural-network-cnn-deep-learning-99760835 f 148 30 September 2020 Multi-objective Precision Optimization of DNNs for Edge Devices 13
Parameterize the objective 30 September 2020 Multi-objective Precision Optimization of DNNs for Edge Devices 14
Workflow Classification Accuracy Trained Weights Get fixedpoint representation of layer K 30 September 2020 Multi-objective Precision Optimization of DNNs for Edge Devices 15
First objective (MAC energy) 10 120 100 80 60 40 20 0 Bitwidth 8 6 4 2 0 Layer 1 2 3 4 5 Total Baseline Energy Optimized Energy Baseline Bitwidth Optimized Bitwidth Normalized MAC energy Alex. Net's MAC energy Judd, Patrick, et al. "Stripes: Bit-serial deep neural network computing. " Microarchitecture (MICRO), 2016 49 th Annual IEEE/ACM International Symposium on. IEEE, 2016 30 September 2020 Multi-objective Precision Optimization of DNNs for Edge Devices 16
Alexnet's DRAM bandwidth 10 150 Bitwidth 8 100 6 4 50 2 0 Layer 30 September 2020 0 1 2 3 4 5 Total Baseline bandwidth Optimized bandwidth Baseline Bitwidth Optimized Bitwidth Multi-objective Precision Optimization of DNNs for Edge Devices Normalized DRAM Bandwidth Second objective (Input Read bandwidth) 17
• 30 September 2020 Multi-objective Precision Optimization of DNNs for Edge Devices 18
Minimizing input read bandwidth Judd, Patrick, et al. "Stripes: Bit-serial deep neural network computing. " Microarchitecture (MICRO), 2016 49 th Annual IEEE/ACM International Symposium on. IEEE, 2016 and uniform bitwidth Bandwidth saving compared to the baseline 29 1% accuracy drop Improvement (%) 24 19 14 9 4 -1 30 September 2020 Ni. N Google. Net Vgg 19 Res. Net-50 Res. Net-152 Squeeze. Net Mobile. Net Multi-objective Precision Optimization of DNNs for Edge Devices Alex. Net 19
Minimizing Energy of MAC operations Energy saving on MAC operations (simulated) Improvement (%) 50 1% accuracy drop 40 30 20 10 30 September 2020 t Ne ex le ob i M Al Ne t et e. N ez Sq ue 52 t-1 Re s. N et -5 0 9 Vg g 1 t Ne le Go og Ni N 0 Multi-objective Precision Optimization of DNNs for Edge Devices 20
Limitation & Future work • Can reduce bitwidth further with retraining & finetuning • Weight’s bitwidth is also important. Currently uniform weight bitwidth across DNN. • Need better model of energy from all components other than MAC alone. 30 September 2020 Multi-objective Precision Optimization of DNNs for Edge Devices 21
Conclusion • Fast analysis time • Can embed hardware design constraints to the solver • Open ended optimization • Implemented in Caffe, available at: • https: //www. github. com/minhhn 2910/mupod 30 September 2020 Multi-objective Precision Optimization of DNNs for Edge Devices 22
Thank you Q & A 30 September 2020 Multi-objective Precision Optimization of DNNs for Edge Devices 23
Appendix Output error Derive bitwidth Measured constants
- Slides: 24