Logistic Regression ECE5424 G CS5824 JiaBin Huang Virginia

Logistic Regression ECE-5424 G / CS-5824 Jia-Bin Huang Virginia Tech Spring 2019

Administrative • Please start HW 1 early! • Questions are welcome!

Two principles for estimating parameters • Slide credit: Tom Mitchell

Naïve Bayes classifier •

Naïve Bayes classifier • Slide credit: Tom Mitchell

Example • Bayes rule Conditional indep.

• Slide credit: Tom Mitchell

• F = 1 iff you live in Fox Ridge • S = 1 iff you watched the superbowl last night • D = 1 iff you drive to VT • G = 1 iff you went to gym in the last month

Naïve Bayes: Subtlety #1 • Slide credit: Tom Mitchell

Naïve Bayes: Subtlety #2 • Slide credit: Tom Mitchell

• Slide credit: Tom Mitchell

Things to remember •

Logistic Regression • Hypothesis representation • Cost function • Logistic regression with gradient descent • Regularization • Multi-classification

1 (Yes) • Malignant? 0 (No) Tumor Size Slide credit: Andrew Ng

• Slide credit: Andrew Ng

Hypothesis representation • Slide credit: Andrew Ng

Interpretation of hypothesis output • Slide credit: Andrew Ng

Logistic regression • Slide credit: Andrew Ng

Decision boundary • Age Tumor Size Slide credit: Andrew Ng

• Slide credit: Andrew Ng

Where does the form come from? • Slide credit: Tom Mitchell

• Applying Bayes rule Slide credit: Tom Mitchell

Logistic Regression • Hypothesis representation • Cost function • Logistic regression with gradient descent • Regularization • Multi-classification

• Slide credit: Andrew Ng

Cost function for Linear Regression • Slide credit: Andrew Ng

Cost function for Logistic Regression • Slide credit: Andrew Ng

Logistic regression cost function • Slide credit: Andrew Ng

Logistic regression • Slide credit: Andrew Ng

Where does the cost come from? • Slide credit: Tom Mitchell

• Slide credit: Tom Mitchell

Expressing conditional log-likelihood •

Logistic Regression • Hypothesis representation • Cost function • Logistic regression with gradient descent • Regularization • Multi-classification

Gradient descent • Good news: Convex function! Bad news: No analytical solution Slide credit: Andrew Ng

Gradient descent • Slide credit: Andrew Ng

• Slide credit: Andrew Ng

Logistic Regression • Hypothesis representation • Cost function • Logistic regression with gradient descent • Regularization • Multi-classification

How about MAP? • Maximum conditional likelihood estimate (MCLE) • Maximum conditional a posterior estimate (MCAP)

• Slide credit: Tom Mitchell

MLE vs. MAP •

Logistic Regression • Hypothesis representation • Cost function • Logistic regression with gradient descent • Regularization • Multi-classification

Multi-classification • Email foldering/taggning: Work, Friends, Family, Hobby • Medical diagrams: Not ill, Cold, Flu • Weather: Sunny, Cloudy, Rain, Snow Slide credit: Andrew Ng

Binary classification Multiclassification

One-vs-all (one-vs-rest) Class 1: Class 2: Class 3: Slide credit: Andrew Ng

One-vs-all • Slide credit: Andrew Ng

Further readings • Tom M. Mitchell Generative and discriminative classifiers: Naïve Bayes and Logistic Regression http: //www. cs. cmu. edu/~tom/mlbook/NBayes. Log. Reg. pdf • Andrew Ng, Michael Jordan On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes http: //papers. nips. cc/paper/2020 -on-discriminative-vs-generativeclassifiers-a-comparison-of-logistic-regression-and-naive-bayes. pdf

Things to remember • Hypothesis representation • Cost function • Logistic regression with gradient descent • Regularization • Multi-classification

Coming up… • Regularization • Support Vector Machine