BAYESIAN RULE GAUSSIAN MIXTURE MODELS Jianping Fan Dept

BAYESIAN RULE & GAUSSIAN MIXTURE MODELS Jianping Fan Dept of Computer Science UNC-Charlotte

BASIC CLASSIFICATION New Mail Not-spam Spam

What we have learned from K-Means clustering? y Test Sample

Difference Between Classification and Clustering Classification labels vs. Clustering non-labels

What are the critical issues for classification? New Mail Not-spam Spam

BASIC CLASSIFICATION Input Spam filtering Output Spam vs. Not-Spam Binary !!!!$$$!!!! Spam email prior; Not-spam email prior Spam email distribution/density Not-spam email distribution/density &

BASIC CLASSIFICATION Input Spam filtering Spam vs. Not-Spam !!!!$$$!!!! Output Binary

BASIC CLASSIFICATION Input Character recognition Output Multi-Class C C vs. other 25 characters

STRUCTURED CLASSIFICATION Input Handwriting recognition 3 D object recognition Output Structured output Graph Model brace building tree

OVERVIEW OF BAYESIAN DECISION Bayesian classification: E. g. one example How to decide if a patient is sick or healthy, based on A probabilistic model of the observed data (data distributions) Prior knowledge (ratio or importance) Majority has bigger voice

OVERVIEW OF BAYESIAN DECISION Test patient (a) Group assignment for test patient; (b) Prior knowledge about the assigned group (c) Properties of the assigned group (sick or healthy)

OVERVIEW OF BAYESIAN DECISION Test patient Observations: Bayesian decision process is a data modeling process, e. g. , estimate the data distribution K-means clustering: any relationship & difference?

BAYES’ RULE: Who is who in Bayes’ rule normalization

CLASSIFICATION PROBLEM Training data: examples of the form (d, h(d)) where d are the data objects to classify (inputs) and h(d) are the correct class info for d, h(d) {1, …K} Goal: given dnew, provide h(dnew)

WHY BAYESIAN? Provides practical learning algorithms E. g. Naïve Bayes Prior knowledge and observed data can be combined It is a generative (model based) approach, which offers a useful conceptual framework E. g. sequences could also be classified, based on a probabilistic model specification Any kind of objects can be classified, based on a probabilistic model specification

BAYES’ RULE: P(d|h): data distribution of group h P(h): importance of group h data distribution of whole data set

BAYES’ RULE: Works to support Bayesian decision: Estimating the data distribution for whole data set Estimating the data distribution for specific group Prior knowledge about specific group

Gaussian Mixture Model (GMM) How to estimate the distribution for a given dataset?

Gaussian Mixture Model (GMM)

Gaussian Mixture Model (GMM) What does GMM mean? 100 = 5*10 + 2*20 + 2*5 100 = 100*1 100 = 10*10 ………………. GMM may prefer larger K with ``smaller” Gaussians

Gaussian Mixture Model (GMM)

Gaussian Mixture Model (GMM) Real Data Distribution Any complex function (distribution) can be approximated by using a limited number of other functions (distributions) such as Gaussian functions.

Gaussian Mixture Model (GMM) Real Data Distribution Approximated Gaussian functions

GAUSSIAN FUNCTION Why we select Gaussian function not others?

When one Gaussian Function is used to approximate data distribution a likelihood function (often simply the likelihood) is a function of the parameters of a statistical model Given a dataset, how to use likelihood function to determine Gaussian parameters?

GAUSSIAN FUNCTION AND SAMPLES Sampling

MAXIMUM LIKELIHOOD Sampling Given x, it is a function of and 2 We want to maximize it.

LOG-LIKELIHOOD FUNCTION Maximize this instead By setting and

MAX. THE LOG-LIKELIHOOD FUNCTION

When multiple Gaussian Functions are used to approximate data distribution a likelihood function (often simply the likelihood) is a function of the parameters of a statistical model Given a dataset, how to use likelihood function to determine parameters for multiple Gaussian functions?

Gaussian Mixture Model (GMM) Parameters to be estimated: Training Set: We assume K is available (or pre-defined)! Algorithms may always prefer larger K!

GAUSSIAN MIXTURE MODELS Rather than identifying clusters by “nearest” centroids Fit a Set of k Gaussians to the data Maximum Likelihood over a mixture model

GMM EXAMPLE

MIXTURE MODELS Formally a Mixture Model is the weighted sum of a number of pdfs where the weights are determined by a distribution,

GAUSSIAN MIXTURE MODELS GMM: the weighted sum of a number of Gaussians where the weights are determined by a distribution,

MAXIMUM LIKELIHOOD OVER A GMM As usual: Identify a likelihood function Log-likelihood

MAXIMUM LIKELIHOOD OF A GMM Optimization of means. k=1, …. . K

MAXIMUM LIKELIHOOD OF A GMM Optimization of covariance k=1, …. . K

MAXIMUM LIKELIHOOD OF A GMM Optimization of mixing term

EM FOR GMMS Initialize the parameters Evaluate the log likelihood Expectation-step: Evaluate the responsibilities Maximization-step: Re-estimate Parameters Evaluate the log likelihood Check for convergence

EM FOR GMMS E-step: Evaluate the Responsibilities

EM FOR GMMS M-Step: Re-estimate Parameters

EM FOR GMMS Evaluate the log likelihood and check for convergence of either the parameters or the log likelihood. If the convergence criterion is not satisfied, return to E-Step.

VISUAL EXAMPLE OF EM

INCORRECT NUMBER OF GAUSSIANS

TOO MANY PARAMETERS For n-dimensional data, we need to estimate at least 3 n parameters

RELATIONSHIP TO K-MEANS K-means makes hard decisions. Each data point gets assigned to a single cluster. GMM/EM makes soft decisions. Each data point can yield a posterior p(z|x) Soft K-means is a special case of EM.

SOFT MEANS AS GMM/EM Assume equal covariance matrices for every mixture component: Likelihood: Responsibilities: As epsilon approaches zero, the responsibility approaches unity.

SOFT K-MEANS AS GMM/EM Overall Log likelihood as epsilon approaches zero: The expectation of soft k-means is the inter-cluster variability Note: only the means are re-estimated in Soft K-means. The covariance matrices are all tied.

GENERAL FORM OF EM Given a joint distribution over observed and latent variables: Want to maximize: 1. Initialize parameters 2. E Step: Evaluate: 3. M-Step: Re-estimate parameters (based on expectation of complete-data log likelihood) 4. Check for convergence of parameters or likelihood

GAUSSIAN MIXTURE MODEL

The conditional probability p(zk = 1|x) denoted by γ(zk ) is obtained by Bayes' theorem, We view πk as the prior probability of zk = 1, and γ(zk ) as the posterior probability. γ(zk ) is the responsibility that k-component takes for explaining the observation x.

MAXIMUM LIKELIHOOD ESTIMATION Given a data set of X = {x 1, …, x. N}, the log of the likelihood function is s. t.

Setting the derivatives of ln p(X|π, μ, Σ) with respect to μk to zero, we obtain

Finally, we maximize ln p(X|π, μ, Σ) with respect to the mixing coefficients πk. We use a Largrange multiplier objective ftn. f(x) max f(x) s. t. g(x)=0 constraints g(x)

which gives we find λ = -N and

1. Initialise the means μk , covariances Σk and mixing coefficients πk. 2. Ε step: Evaluate the responsibilities using the current parameter values 3. M step: RE-estimate the parameters using the current responsibilities 4. Evaluate the log likelihood and check for convergence of either the parameters or the log likelihood. If the convergence criterion is not satisfied, return to step 2.

SOURCE CODE FOR GMM 1. EPFL http: //lasa. epfl. ch/sourcecode/ 2. Google Source Code on GMM http: //code. google. com/p/gmmreg/ 3. GMM & EM http: //crsouza. blogspot. com/2010/10/gaussia n-mixture-models-and-expectation. html

PROBABILITIES – AUXILIARY SLIDE FOR MEMORY REFRESHING Have two dice h 1 and h 2 The probability of rolling an i given die h 1 is denoted P(i|h 1). This is a conditional probability Pick a die at random with probability P(hj), j=1 or 2. The probability for picking die hj and rolling an i with it is called joint probability and is P(i, hj)=P(hj)P(i| hj). For any events X and Y, P(X, Y)=P(X|Y)P(Y) If we know P(X, Y), then the so-called marginal probability P(X) can be computed as

DOES PATIENT HAVE CANCER OR NOT? A patient takes a lab test and the result comes back positive. It is known that the test returns a correct positive result in only 98% of the cases and a correct negative result in only 97% of the cases. Furthermore, only 0. 008 of the entire population has this disease. 1. What is the probability that this patient has cancer? 2. What is the probability that he does not have cancer? 3. What is the diagnosis?

CHOOSING HYPOTHESES Maximum Likelihood hypothesis: Generally we want the most probable hypothesis given training data. This is the maximum a posteriori hypothesis: Useful observation: it does not depend on the denominator P(d)

NOW WE COMPUTE THE DIAGNOSIS To find the Maximum Likelihood hypothesis, we evaluate P(d|h) for the data d, which is the positive lab test and chose the hypothesis (diagnosis) that maximises it: To find the Maximum A Posteriori hypothesis, we evaluate P(d|h)P(h) for the data d, which is the positive lab test and chose the hypothesis (diagnosis) that maximises it. This is the same as choosing the hypotheses gives the higher posterior probability.

NAÏVE BAYES CLASSIFIER What can we do if our data d has several attributes? Naïve Bayes assumption: Attributes that describe data instances are conditionally independent given the classification hypothesis it is a simplifying assumption, obviously it may be violated in reality in spite of that, it works well in practice The Bayesian classifier that uses the Naïve Bayes assumption and computes the MAP hypothesis is called Naïve Bayes classifier One of the most practical learning methods Successful applications: Medical Diagnosis Text classification

EXAMPLE. ‘PLAY TENNIS’ DATA

NAÏVE BAYES SOLUTION Classify any new datum instance x=(a 1, …a. T) as: To do this based on training examples, we need to estimate the parameters from the training examples: For each target value (hypothesis) h For each attribute value at of each datum instance

Based on the examples in the table, classify the following datum x: x=(Outl=Sunny, Temp=Cool, Hum=High, Wind=strong) That means: Play tennis or not? Working:

LEARNING TO CLASSIFY TEXT Learn from examples which articles are of interest The attributes are the words Observe the Naïve Bayes assumption just means that we have a random sequence model within each class! NB classifiers are one of the most effective for this task Resources for those interested: Tom Mitchell: Machine Learning (book) Chapter 6.

RESULTS ON A BENCHMARK TEXT CORPUS

REMEMBER Bayes’ rule can be turned into a classifier Maximum A Posteriori (MAP) hypothesis estimation incorporates prior knowledge; Max Likelihood doesn’t Naive Bayes Classifier is a simple but effective Bayesian classifier for vector data (i. e. data with several attributes) that assumes that attributes are independent given the class. Bayesian classification is a generative approach to classification

RESOURCES Textbook reading (contains details about using Naïve Bayes for text classification): Tom Mitchell, Machine Learning (book), Chapter 6. Software: NB for classifying text: http: //www-2. cs. cmu. edu/afs/cs/project/theo-11/www/naivebayes. html Useful reading for those interested to learn more about NB classification, beyond the scope of this module: http: //www-2. cs. cmu. edu/~tom/New. Chapters. html

GMM & BAYESIAN DECISION Open Issues Number of Gaussians K Initializations of parameters Pay more attentions on majority

GMM & BAYESIAN RULE Advantages Intuitive modeling Probabilistic Multi-class solution Disadvantages Number of Gaussians K Initializations of Parameters Not-discriminative Feature dimensions Majority vs. minority