From linear classifiers to neural network Skeleton Recap

From linear classifiers to neural network

Skeleton • Recap – linear classifier • Nonlinear discriminant function • Nonlinear cost function • Example Linear Classifiers • Introduction to neural network • Representation power of sigmoidal neural network

Linear Classifier: Recap & Notation •

Discriminant function • 1 -1

Loss function • -1 1

Structure of linear classifier 1 1

Alternatively •

Structure of linear classifier 1 1

Ideal case - Problem? • Nonlinear function 1 -1 • Loss function • Not differentiable. Cannot train using gradient methods • We need proxy for both.

Skeleton • Recap – linear classifier • Nonlinear discriminant function • Nonlinear cost function • Example Linear Classifiers • Introduction to neural network • Representation power of sigmoidal neural network

Nonlinear Discriminant function • Our Goal: Find a function that is • Differentiable • Approximates the step function • Solution: Sigmoid function • Definition: • A bounded, differentiable and monotonically increasing function.

Sigmoid Function - Examples •

Sigmoid Function - Examples •

Skeleton • Recap – linear classifier • Nonlinear discriminant function • Nonlinear cost function • Example Linear Classifiers • Introduction to neural network • Representation power of sigmoidal neural network

Nonlinear Loss function • Our Goal: Find a function that is • Differentiable • Is an UPPER BOUND of the step function • Why? • For training: min loss => error not large • For test: generalized error < generalized loss < upper bounds

Loss function example •

Loss function example •

Skeleton • Recap – linear classifier • Nonlinear discriminant function • Nonlinear cost function • Example Linear Classifiers • Introduction to neural network • Representation power of sigmoidal neural network

Linear classifiers example Nonlinear Discriminant Function Loss Function Linear Square Sigmoid Hinge • Linear + Square: MSE classifier • Sigmoid + Squared: Nonlinear MSE classifier • Linear + Hinge + Regularization : SVM

MSE Classifier •

Nonlinear MSE Classifier •

Training a Nonlinear MSE Classifier •

Skeleton • Recap – linear classifier • Nonlinear discriminant function • Nonlinear cost function • Example Linear Classifiers • Introduction to neural network • Representation power of sigmoidal neural network

Introduction of neural network •

Introduction of neural network

Introduction of neural network

Introduction of neural network

Notation

Notation

Question •

Skeleton • Recap – linear classifier • Nonlinear discriminant function • Nonlinear cost function • Example Linear Classifiers • Introduction to neural network • Representation power of sigmoidal neural network

Proof: representation power •

Proof: representation power •