Quantization Content Motivation from tensorflow doc Overview of

Quantization

Content • Motivation from tensorflow doc • Overview of quantization methods from survey paper • Two methods in more depth – One based on two tools – The other based on a research paper

From https: //www. tensorflow. org/performance/quantization

From https: //www. tensorflow. org/performance/quantization storage latency energy

https: //petewarden. com/2015/05/23/why-are-eight-bits-enough-for-deep-neural-networks/ accuracy

accuracy From https: //www. tensorflow. org/performance/quantization

How to declare fixed point variables in code? How to do fixed point math in code? https: //github. com/eteran/cpp-utilities Wrapper class with integer arithmetic.

Quantization results comparison Network: Alexnet Dataset: Imagenet Accuracy measured: Top-5 error From V. Sze, T. -J. Yang, Y. -H. Chen, J. Emer, "Efficient Processing of Deep Neural Networks: A Tutorial and Survey, " ar. Xiv, 2017.

Quantization results: only weights Network: Alexnet Dataset: Imagenet Accuracy measured: Top-5 error From V. Sze, T. -J. Yang, Y. -H. Chen, J. Emer, "Efficient Processing of Deep Neural Networks: A Tutorial and Survey, " ar. Xiv, 2017.

Quantization results: both weights and activations Network: Alexnet Dataset: Imagenet Accuracy measured: Top-5 error From V. Sze, T. -J. Yang, Y. -H. Chen, J. Emer, "Efficient Processing of Deep Neural Networks: A Tutorial and Survey, " ar. Xiv, 2017.

Quantization results: binary values https: //github. com/Matthieu. Courbariaux/Binary. Connect https: //github. com/allenai/XNOR-Net Network: Alexnet Dataset: Imagenet Accuracy measured: Top-5 error From V. Sze, T. -J. Yang, Y. -H. Chen, J. Emer, "Efficient Processing of Deep Neural Networks: A Tutorial and Survey, " ar. Xiv, 2017.

Quantization results: more details Network: Alexnet Dataset: Imagenet Accuracy measured: Top-5 error From V. Sze, T. -J. Yang, Y. -H. Chen, J. Emer, "Efficient Processing of Deep Neural Networks: A Tutorial and Survey, " ar. Xiv, 2017.

Dynamic fixed point

Ristretto quantization methods: fixed point, dynamic fixed point, minifloat Deterministic rounding Stochastic rounding Dynamic fixed point

Dynamic Fixed Point, where to put the point? Range vs. precision IEEE 754 standard floating point Dynamic Fixed point

Tensor. RT: where to saturate? Symmetric linear quantization

Empirical observations: Dynamic range of parameters and activations in LENET

Empirical observations: Dynamic range of weights and parameters in Caffe. Net

Tensor. RT: empirical observations

Empirical Observations • Different networks have different range of weights and activations • Weights and activations within same network have different numerical ranges

Ristretto: a quantization tool built on Caffe

For Tensor. RT

For Tensor. RT

Different networks need different bitwidths and formats, depends on quantization method

Different networks need different bit-widths (depends on quantization method)

Tensor. RT: network specific thresholds

Dynamic fixed point better than static fixed point -> exploits different layers within the network having different numerical ranges

Dynamic fixed point performs best

Tensor. RT accuracy

Tensor. RT performance

Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding Song Han, Huizi Mao, William J. Dally ICLR 2016

K-means clustering based quantization • Bit-width representation of each weight doesn’t change, so each weight is still the same size • Each weight represented by the mean value of the cluster it belongs to • Number of unique weights change, storage requirements reduce due to this weight sharing Minimizes within cluster sum of squares

Cluster initialization

Linear Initialization gives best results Larger weights play a more important role than smaller weights (Han et al. , 2015), but there are fewer of these large weights. Thus for both random initialization and density-based initialization, very few centroids have large absolute value which results in poor representation of these few large weights. Linear initialization does not suffer from this.

Quantization after pruning Unsupervised K-Means clustering of weights benefits from pruning. Pruning retains more important weights, that can be grouped into more meaningful clusters.

Benefits of pruning

Huffman encoding Opportunity for Huffman Encoding, as seen from the empirical values of weights

Overall flow

FC layer performance improvements

Accuracy and data storage -> network weights and activations/ quantization bit widths. Latency/energy -> hardware platform (CPU with SIMD vs. GPU vs. FPGA? has floating point co-processor unit? 8 -bit vs. 16 bit vs. 32 bit addressing ……)

Floating vs. fixed point performance, hardware dependency (1) http: //www. ti. com/lit/wp/spry 061. pdf (2) https: //www. eetimes. com/document. asp? doc_id=1275364 (3) https: //stackoverflow. com/questions/25351114/is-fixed-pointmath-faster-than-floating-point-on-armv 7 -a (4) https: //community. arm. com/processors/f/discussions/1466/neonfixed-point-coding-and-fixed-vs-floating-point-operations-performance -comparison (5) challenges of fixed point: http: //www. artistembedded. org/docs/Events/2009/Embedded. Control/SLIDES/Fix. Point. pdf slide 20 (6) https: //blogs. mentor. com/colinwalls/blog/2012/09/10/the-floating -point-argument/ (7) https: //dsp. stackexchange. com/questions/4835/relative-merits-offixed-point-vs-floating-point-computation

Next …. • Riju to-do: update page with slides/ other content • Fri one-to-one class Pranshu, Daniel • Tue: finish factorization with CP/Tucker decomposition for tensors, sum up decomposition, pruning, quantization i. e. all post training network optimization methods