Neural Network Architectures CMPT 726 Machine Learning Oliver

Neural Network Architectures CMPT 726 Machine Learning Oliver Schulte

Feed-Forward Networks

Network View of Basis Functions Y w 0 w 2 w 1 x 1 x 2 x 1 Output layer: Regression model Y = w 0 + w 1 (x 1 x 2) + w 2 (x 2 x 3) x 2 x 3 x 2 x 3 Hidden layer: 2 basis functions representing one quadratic interaction each Input layer: observed inputs e. g. age, weight, games played. . . 3/13

Trainable Basis Functions Y w 0 w 2 w 1 ϕ(x 1, x 2; w) x 1 Output layer: Regression model Y = w 0 + w 1 (x 1 x 2) + w 2 (x 2 x 3) x 2 ϕ(x 2, x 3; w ) x 3 Hidden layer: 2 basis functions with adjustable parameters Input layer: observed inputs e. g. age, weight, games played. . . 4/13

Allow All Interactions Y w 0 w 2 w 1 ϕ(x; w) x 1 Output layer: Regression model Y = w 0 + w 1 (x 1 x 2) + w 2 (x 2 x 3) ϕ(x; w) x 2 x 3 Fully connected Hidden layer: 2 basis functions with adjustable parameters Input layer: observed inputs e. g. age, weight, games played. . . 5/13

What kind of basis function? Y w 0 w 2 w 1 ϕ(x; w) w 11 x 1 ϕ(x; w) w 12 w 13 x 2 similarly we have w 21, w 22, w 23 x 3 • Any function that maps n numbers to 1 could be a basis function. • e. g. Gaussian basis function (RBF network). • How about linear functions? • We now how to do linear regression! 6/13

Deep Neural Networks

Definition �A deep neural network is a neural network with at least 2 hidden layers. �That’s it! output layer fully connected linear/logistic regression hidden layer rectified linear unit Y w 2 w 1 ϕ(x; w) x 1 w 0 ϕ(x; w) x 2 ϕ(x; w) x 3 8/13

Rectified Linear Unit �f(x) = max(0, x) CC 0, https: //en. wikipedia. org/w/index. php? curid=48817276 9/13

Advantages of RLUs �Gradients linear in input Ø avoids vanishing gradients § many units have 0 activation Ø avoids co-adaption 10/13

The Vanishing Gradient Problem • The sigmoid function becomes flat quickly away from 0 • Recall that the gradient is multiplied by the derivative of the sigmoid function • In multiple layers, each layer adds a gradient multiplication small gradient 11/13