Classification Probabilistic Generative Model Disclaimer This PPT is

Classification: Probabilistic Generative Model Disclaimer: This PPT is modified based on Dr. Hung-yi Lee http: //speech. ee. ntu. edu. tw/~tlkagk/courses_ML 17. html

Classification Function Class n • Credit Scoring • Input: income, savings, profession, age, past financial history …… • Output: accept or refuse • Medical Diagnosis • Input: current symptoms, age, gender, past medical history …… • Output: which kind of diseases Input: output: • Handwritten digit recognition 0 • Face recognition • Input: image of a face, output: person

Example Application

Example Application

How to do Classification • Training data for Classification as Regression? Binary classification as example Training: Class 1 means the target is 1; Class 2 means the target is -1

b + w 1 x 1 + w 2 x 2 = 0 to decrease error Class 2 -1 -1 1 1 x 2 outliers >>1 Class 1 error y = b + w 1 x 1 + w 2 x 2 x 1 Penalize to the examples that are “too correct” … (Bishop, P 186) • Multiple class: Class 1 means the target is 1; Class 2 means the target is 2; Class 3 means the target is 3 …… problematic

Ideal Alternatives • Function (Model): Output = class 1 Output = class 2 • Loss function (indicator function): The number of times f get incorrect results on training data. • Find the best function: • Example: Perceptron, SVM Not Today

Two Boxes Box 1 Box 2 Prior P(B 1) = 2/3 Prior P(B 2) = 1/3 P(Blue|B 1) = 4/5 P(Green|B 1) = 1/5 from one of the boxes Where does it come from? (Bayes’ Rule) P(B 1 | Blue) P(Blue|B 1) = 2/5 P(Green|B 1) = 3/5

Two Boxes Estimating the Probabilities from training data Class 1 Class 2 P(C 1) P(C 2) P(x|C 1) Given an x, which class does it belong to Generative Model P(x|C 2)

Prior Probability Class 1 Class 2 P(C 1) P(C 2) Water Normal Water and Normal type with ID < 400 for training, rest for testing Training: 79 Water, 61 Normal P(C 1) = 79 / (79 + 61) =0. 56 P(C 2) = 61 / (79 + 61) =0. 44

Probability from Class P(x|C 1) = ? P( |Water) = ? Each Pokémon is represented as a vector by its attribute. Training data Water Type Feature vectors 79 in total

P(x|C 1) = ? Probability from Class - Feature • Considering Defense and SP Defense P(x|Water)=? 0 Assume the points are Water sampled from a Type Gaussian distribution. ?

Review: Gaussian (Normal) Distribution Input: vector x, output: probability of sampling x http: //www. k-wave. org/documentation/get. Win. php https: //www. google. com/search? q=2 d+gaussian+mo del&rlz=1 C 1 GCEU_en. US 821&source=lnms&tb m=isch&sa=X&ved=0 ah. UKEwj 0 iv. Pqgsjg. Ah. UQp. Fk. KHf. J p. AD 0 Q_AUIDig. B&biw=1024&bih=546

https: //blog. slinuxer. com/tag/pca Gaussian Distribution Input: vector x, output: probability of sampling x

https: //blog. slinuxer. com/tag/pca Gaussian Distribution Input: vector x, output: probability of sampling x

Probability from Class Assume the points are sampled from a Gaussian distribution Find the Gaussian distribution behind them Probability for new points New x How to find them? Water Type

Maximum Likelihood Different Likelihood

We have the “Water” type Pokémons: sample mean sample covariance matrix

Maximum Likelihood to find mean �� and covariance matrix �� Class 1: Water Class 2: Normal

Maximum Likelihood end… now we can find P(x|C 1) = ? P(x|C 2) = ?

Now we can do classification P(C 1) = 79 / (79 + 61) =0. 56 P(C 2) = 61 / (79 + 61) =0. 44 x belongs to class 1 (Water)

Blue points: C 1 (Water), Red points: C 2 (Normal) How’s the results? Testing data: 47% accuracy All: total, hp, att, sp att, de, speed (6 features) 64% accuracy …

Modifying Model Class 1: Water Class 2: Normal Less parameters

Modifying Model Ref: Bishop, chapter 4. 2. 2 • Maximum likelihood “Water” type Pokémons: “Normal” type Pokémons:

Modifying Model The boundary is linear The same covariance matrix (linear boundary) All: total, hp, att, sp att, de, speed 54% accuracy 73% accuracy

Three Steps • Otherwise, output: class 2

Naïve Bayes Classifier -- Probability distribution • You can always choose the distribution you like …… …… 1 -D Gaussian (diagonal covariance matrix) For binary features, you may assume they are from Bernoulli distributions. If you assume all the dimensions are independent, then you are using Naive Bayes Classifier. If this assumption is not true generative model

Posterior Probability Sigmoid function

Warning of Math

Posterior Probability sigmoid

End of Warning

(2) How about directly find w and b? Then we find w and b

Reference • Bishop: Chapter 4. 1 – 4. 2 • Data: https: //www. kaggle. com/abcsds/pokemon • Useful posts: • https: //www. kaggle. com/nishantbhadauria/d/abcsds/p okemon/pokemon-speed-attack-hp-defense-analysis-by -type • https: //www. kaggle. com/nikos 90/d/abcsds/pokemon/ mastering-pokebars/discussion • https: //www. kaggle. com/ndrewgele/d/abcsds/pokemo n/visualizing-pok-mon-stats-with-seaborn/discussion

Review: STT 315: 2. 10 The Law of total prob and bayes’ rules p Def: For some positive integer k, let B 1, B 2, …Bk be such that Then the collection of sets {B 1, B 2, …Bk }is said to be a partition of S. Example: Based on the example, we have: 38

Review: STT 315: 2. 10 The Law of total prob and bayes’ rules p Thm 2. 8 If the events B 1, B 2, …, Bk constitute a partition of the sample space S such that P(Bi)≠ 0 for i=1, 2, …, k, then for any event A of S, p Thm 2. 9, Bayes’ Rule: If the events B 1, B 2, …, Bk constitute a partition of the sample space S such that P(Bi)≠ 0 for i=1, 2, …, k, then for any event A in S such that P(A)≠ 0, 39

Review: STT 315: 2. 10 The Law of total prob and bayes’ rules p When r=2, the special case is : p Ex 2. 124: 40% rep and 60% dem. 30% of rep and 70% of the dem favor an election issue. If a random picked voter is in favor of the issue, find the prob that this person is a dem. 40