Machine Learning Revisited Eugene Wong Academia Sinica and

Machine Learning Revisited Eugene Wong Academia Sinica and UC Berkeley

Introduction • Machine Learning is a rage • Real, not hype • But consumes enormous computing power and time • My two fold purpose • Deeper understanding • Reducing computing time • Work in progress • More questions than answers • More ideas than results

Neural Network x Single Layer y z f Multiple Layer Network Input Layer Output Layer Hidden Layer f

Deep Learning What’s given? A set of sample input vectors: S = {x} A desired output vector for each x: t(x) Training Cycle: use each x in S once in each cycle do computation after each x adjust weights at the end of cycle Machine Learning in a Nut Shell: what to compute and how to adjust the weights?

Measuring how well the neural net is doing during training How to adjust weights? At the end of each learning cycle, compute Adjust each weight by Then the cost will change by for each i, j, and m α > 0

How to compute ? Answer: by back propagation An old idea in multi-stage optimum control for space guidance

Back Propagation Explained the mth layer f Layer m-1 Layer m+1 error signal × error signal

Special Cases for the Feedback Loop Input Layer: m=1 Output Layer: m=N x

Note: each row in the weight matrix changes by an amount proportional to the input vector for every layer. Idea goes back to perceptron algorithm. Consequence of bi-linearity

Consider an example: 3 -layer NN for image classification Input: 1000 x 1000= 1, 000 pixels dim(x) = 1, 000 Final output: 10 categories dim(t(x)) = 10 2 nd layer: dim(input)= 1000, dim(output)=100 Training set: 1, 000 samples Number of weights: 1 st layer: 1 billion 2 nd: 100, 000 Number of ops per training cycle: > 2 trillion+ Might take a month to reach convergence! 3 rd: 1, 000 0. 5 hour

How good is the design? Are the dimensions right? 1 billion+ weights for only 1 billion data points (overfitting) Need more samples or should reduce weights in 1 st layer What do the first two layers do? 1 st layer: nonlinear transformation of the input space 2 nd layer: feature extraction 1 st layer should be 1, 000 by 1, 000 (1 trillion weights)!

Big Data or Statistics • Big Data philosophy • Use all the data • No sampling, no separate modeling, limited pre-analysis • Applied to ML means • • Large training set Over fitting Long training Matching to samples, not learning • Alternative • Use statistical modeling to reduce weights and learning time • But we have been here before!

Comparison of Complexity Worst Compute for every sample Compute every training cycle Need to compute only once every P cycles (say P>=10) Best Need to compute only once for the entire training process

Reducing Computation • Pre-compute • Reduce number of weights in 1 st layer • Reduce training set • ML does not eliminate the need for pre-analysis • Linearize feedback • No need for per-sample matrix multiplication in back propagation • Linearize forward loop • Isolate layer during training cycle • Speed up convergence • Reduce quantizing error propagation • Eliminate layer (left to right) • Shorten training cycle

Feedback Linearization the mth layer error signal × error signal

Number of weights used per sample Pre-computed Diagonal Feedback linearization Number of adjustable Weights

One Layer at a time Linearize layer m+1 onward, then layer m can adjust weights without error signal feedback. Learning can take place one layer at a time, left to right. Possible advantages: faster convergence limit error propagation due to quantum step change alternative weight change procedure

Observations • ML incomparable as means to build decision maker using samples • Large training set definitely required • Overfitting is a potential problem • Scalability? Yes for each cycle, but questionable for convergence • Reformulation as an optimization problem with many variables • Questionable, need algorithm that works for a trillion variables? • Combining analysis and optimization is a virtue, but some pre-analysis may be useful • Gradient descent may need re-visiting