Machine Learning Probability and Bayesian Networks Doug Downey

Machine Learning Probability and Bayesian Networks Doug Downey (adapted from Bryan Pardo, Northwestern University)

An Introduction • Bayesian Decision Theory came long before Version Spaces, Decision Tree Learning and Neural Networks. It was studied in the field of Statistical Theory and more specifically, in the field of Pattern Recognition. Doug Downey (adapted from Bryan Pardo, Northwestern University)

An Introduction • Bayesian Decision Theory is at the basis of important learning schemes such as… – Naïve Bayes Classifier – Bayesian Belief Networks – EM Algorithm • Bayesian Decision Theory is also useful as it provides a framework within which many non. Bayesian classifiers can be studied – See [Mitchell, Sections 6. 3, 4, 5, 6]. Doug Downey (adapted from Bryan Pardo, Northwestern University)

Discrete Random Variables • A is a Boolean random variable if it denotes an event where there is uncertainty about whether it occurs • Examples – The next US president will be Barack Obama – You will get an A in the course • P(A) = probability of A = the fraction of all possible worlds where A is true Doug Downey (adapted from Bryan Pardo, Northwestern University)

Vizualizing P(A) All Possible Worlds where A is True Doug Downey (adapted from Bryan Pardo, Northwestern University)

Axioms of Probability • Let there be a space S composed of a countable number of events • The probability of each event is between 0 and 1 • The probability of the whole sample space is 1 • When two events are mutually exclusive, their probabilities are additive Doug Downey (adapted from Bryan Pardo, Northwestern University)

Vizualizing Two Boolean RVs A B Doug Downey (adapted from Bryan Pardo, Northwestern University)

Conditional Probability A B The conditional probability of A given B is represented by the following formula NOT Independent Can we do the following? Only if A and B are independent Doug Downey (adapted from Bryan Pardo, Northwestern University)

Independence • variables A and B are said to be independent if knowing the value of A gives you no knowledge about the likelihood of B…and vice-versa P(A|B) = P(A) and P(B|A) = P(B) Doug Downey (adapted from Bryan Pardo, Northwestern University)

An Example: Cards • Take a standard deck of 52 cards. • On the first draw I pull the Ace of Spades. • I don’t replace the card. – What is the probability I’ll pull the Ace of Spades on the second draw? • Now, I replace the Ace after the 1 st draw, shuffle, and draw again. – What is the chance I’ll draw the Ace of Spades on the 2 nd draw? Doug Downey (adapted from Bryan Pardo, Northwestern University)

Discrete Random Variables • A is a discrete random variable if it takes a countable number of distinct values • Examples – Your grade G in the course – The number of heads k in n coin flips • P(A=k) = the fraction of all possible worlds where A equals k • Notation: PD(A = k) prob. relative to a distribution D – Pfair grading(G = “A”), Pcheating(G = “A”) Doug Downey (adapted from Bryan Pardo, Northwestern University)

Bayes Theorem • Definition of Conditional Probability • Corollary: The Chain Rule • Bayes Rule (Thomas Bayes, 1763) Doug Downey (adapted from Bryan Pardo, Northwestern University)

ML in a Bayesian Framework • Any ML technique can be expressed as reasoning about probabilities • Goal: Find hypothesis h that is most probable given training data D • Provides a more explicit way of describing & encoding our assumptions

Some Definitions • Prior probability of h, P(h): – The background knowledge we have about the chance that h is a correct hypothesis (before having observed the data). • Prior probability of D, P(D): – the probability that training data D will be observed given no knowledge about which hypothesis h holds. • Conditional Probability of D, P(D|h): – the probability of observing data D given that hypothesis h holds. • Posterior probability of h, P(h|D): – the probability that h is true, given the observed training data D. – the quantity that Machine Learning researchers are interested in. Doug Downey (adapted from Bryan Pardo, Northwestern University)

Maximum A Posteriori (MAP) • Goal: To find the most probable hypothesis h from a set of candidate hypotheses H given the observed data D. • MAP Hypothesis, h. MAP Doug Downey (adapted from Bryan Pardo, Northwestern University)

Maximum Likelihood (ML) • ML hypothesis is a special case of the MAP hypothesis where all hypotheses are equally likely to begin with Doug Downey (adapted from Bryan Pardo, Northwestern University)

Example: Brute Force MAP Learning • Assumptions – The training data D is noise-free – The target concept c is in the hypothesis set H – All hypotheses are equally likely • Choice: Probability of D given h Doug Downey (adapted from Bryan Pardo, Northwestern University)

Brute Force MAP (continued) Bayes Theorem Given our assumptions VSH, D is the version space Doug Downey (adapted from Bryan Pardo, Northwestern University)

Find-S as MAP Learning • We can characterize the FIND-S learner (chapter 2) in Bayesian terms – Again P(D | h) is 1 if h is consistent on D, and 0 otherwise – P(h) increases with… • specificity of h – Then: MAP hypothesis = output of Find-S Doug Downey (adapted from Bryan Pardo, Northwestern University)

Neural Nets in a Bayesian Framework • Under certain assumptions regarding noise in the data, minimizing the mean squared error (what multilayer perceptrons do) corresponds to computing the maximum likelihood hypothesis. Doug Downey (adapted from Bryan Pardo, Northwestern University)

Least Squared Error = ML Assume e is drawn from a normal distribution f e Doug Downey (adapted from Bryan Pardo, Northwestern University) h. ML

Least Squared Error = ML Doug Downey (adapted from Bryan Pardo, Northwestern University)

Decision Trees in Bayes Framework • Decent choice for P(h): simpler hypotheses have higher probability – Occam’s razor • This can be encoded in terms of finding the “Minimum Description Length” encoding – Provides a way to “trade off” hypothesis size for training error – Potentially prevents overfitting Doug Downey (adapted from Bryan Pardo, Northwestern University)

Most Compact Coding • Lets minimize the bits used to encode a message • Idea: – Assign shorter codes to more probable messages • According to Shannon & Weaver – An optimal code assigns –log 2 P(i) bits to encode item i • thus… Doug Downey (adapted from Bryan Pardo, Northwestern University)

Minimum Description Length (MDL) Doug Downey (adapted from Bryan Pardo, Northwestern University)

What does all that mean? • The “optimal” hypothesis is the one that is the smallest when we count… – How long the hypothesis description must be – How long the data description must be, given the hypothesis • Key idea: since we’re given h, we need only encode h’s mistakes Doug Downey (adapted from Bryan Pardo, Northwestern University)

What does all that mean? • If the hypothesis is perfect, we don’t need to encode any data. • For each misclassification, we must – say which item is misclassified • Takes log 2 m bits, where m = size of the dataset – Say what the right classification is • Takes log 2 k bits, where k = number of classes Doug Downey (adapted from Bryan Pardo, Northwestern University)

The best MDL hypothesis • The best hypothesis is the best tradeoff between – Complexity of the hypothesis description – Number of times we have to tell people where it screwed up. Doug Downey (adapted from Bryan Pardo, Northwestern University)

Is MDL always MAP? • Only given significant assumptions: – If we know a representation scheme such that size of h in H is -log 2 P(h) – Likewise, the size of the exception representation must be –log 2 P(D|h) – THEN • MDL = MAP Doug Downey (adapted from Bryan Pardo, Northwestern University)

Making Predictions • The reason we learned h to begin with • Does it make sense to choose just one h? h 1 : Looks matter h 2 : Money matters We want a prediction: yes or no? h 3 : Ideas matter Obama Elected President Doug Downey (adapted from Bryan Pardo, Northwestern University)

Maximum A Posteriori (MAP) • Find most probable hypothesis • Use the predictions of that hypothesis h 1 : Looks matter h 2 : Money matters h 3 : Ideas matter …. do we really want to ignore the other hypotheses? Imagine 8 hypotheses. Seven of them say “yes” and have a probability of 0. 1 each. One says “no” and has a probability of 0. 3. Who do you believe? Doug Downey (adapted from Bryan Pardo, Northwestern University)

Bayes Optimal Classifier • Bayes Optimal Classification: The most probable classification of a new instance is obtained by combining the predictions of all hypotheses, weighted by their posterior probabilities: …where V is the set of all the values a classification can take and v is one possible such classification. No other method using the same H and prior knowledge is better (on average). Doug Downey (adapted from Bryan Pardo, Northwestern University)

Naïve Bayes Classifier • Unfortunately, Bayes Optimal Classifier is usually too costly to apply! ==> Naïve Bayes Classifier We’ll be seeing more of these… Doug Downey (adapted from Bryan Pardo, Northwestern University)

The Joint Distribution • Make a truth table listing all combinations of variable values • Assign a probability to each row • Make sure the probabilities sum to 1 A 0 0 1 1 B 0 0 1 1 C 0 1 0 1 Doug Downey (adapted from Bryan Pardo, Northwestern University) Prob 0. 1 0. 2 0. 1 0. 05 0. 25 0. 05

Using The Joint Distribution • Find P(A) • Sum the probabilities of all rows where A=1 P(A=1) = 0. 05+ 0. 25+ 0. 05 = 0. 55 A • P(A) = 0 1 P(A) 0. 45 0. 55 A 0 0 1 1 B 0 0 1 1 C 0 1 0 1 Doug Downey (adapted from Bryan Pardo, Northwestern University) Prob 0. 1 0. 2 0. 1 0. 05 0. 25 0. 05

Using The Joint Distribution • Find P(A|B) • P(A=1 | B=1) =P(A=1, B=1)/P(B=1) =(0. 25+0. 05)/ (0. 25+0. 05+0. 1+0. 05) A B P(A|B) 1 1 0. 67 0 1 0. 33 1 0 0. 45 0 0 0. 55 A 0 0 1 1 B 0 0 1 1 C 0 1 0 1 Doug Downey (adapted from Bryan Pardo, Northwestern University) Prob 0. 1 0. 2 0. 1 0. 05 0. 25 0. 05

Using The Joint Distribution • Are A and B Independent? NO. They are NOT independent A 0 0 1 1 B 0 0 1 1 C 0 1 0 1 Doug Downey (adapted from Bryan Pardo, Northwestern University) Prob 0. 1 0. 2 0. 1 0. 05 0. 25 0. 05

Why not use the Joint Distribution? • Given m boolean variables, we need to estimate 2 m-1 values. • 20 yes-no questions = a million values • How do we get around this combinatorial explosion? – Assume independence of variables!! Doug Downey (adapted from Bryan Pardo, Northwestern University)

…back to Independence • The probability I have an apple in my lunch bag is independent of the probability of a blizzard in Japan. • This is DOMAIN Knowledge, typically supplied by the problem designer Doug Downey (adapted from Bryan Pardo, Northwestern University)

Naïve Bayes Classifier • Cases described by a conjunction of attribute values – These attributes are our “independent” hypotheses • The target function has a finite set of values, V • Could be solved using the joint distribution table • What if we have 50, 000 attributes? – Attribute j is a Boolean signaling presence or absence of the jth word from the dictionary in my latest email. Doug Downey (adapted from Bryan Pardo, Northwestern University)

Naïve Bayes Classifier Doug Downey (adapted from Bryan Pardo, Northwestern University)

Naïve Bayes Continued Conditional independence step Instead of one table of size 250000 we have 50, 000 tables of size 2 Doug Downey (adapted from Bryan Pardo, Northwestern University)

Bayesian Belief Networks • Bayes Optimal Classifier – Often too costly to apply (uses full joint probability) • Naïve Bayes Classifier – Assumes conditional independence to lower costs – This assumption often overly restrictive • Bayesian belief networks – provide an intermediate approach – allows conditional independence assumptions that apply to subsets of the variable. Doug Downey (adapted from Bryan Pardo, Northwestern University)

Example • I'm at work, neighbor John calls to say my alarm is ringing, but neighbor Mary doesn't call. Sometimes it's set off by minor earthquakes. Is there a burglar? • Variables: Burglary, Earthquake, Alarm, John. Calls, Mary. Calls • Network topology reflects "causal" knowledge: – – A burglar can set the alarm off An earthquake can set the alarm off The alarm can cause Mary to call The alarm can cause John to call Doug Downey (adapted from Bryan Pardo, Northwestern University)

Example contd. Doug Downey (adapted from Bryan Pardo, Northwestern University)

Bayesian Networks Parents Pa of Alarm Qualitative part: Directed acyclic graph (DAG) • Nodes - random vars. • Edges - direct influence Earthquake Burglary Alarm Mary. Calls Together: Define a unique distribution in a factored form [Pearl 91] E B P(A | B, E) e b 0. 95 0. 05 e b 0. 94 0. 06 e b 0. 29 0. 01 e b 0. 001 0. 999 John. Calls Quantitative part: Set of conditional probability distributions Doug Downey (adapted from Bryan Pardo, Northwestern University) Traditional Approaches

Compactness • A CPT for Boolean Xi with k Boolean parents has 2 k rows for the combinations of parent values • Each row requires one number p for Xi = true (the number for Xi = false is just 1 -p) • If each variable has no more than k parents, the complete network requires O(n · 2 k) numbers • I. e. , grows linearly with n, vs. O(2 n) for the full joint distribution • For burglary net, 1 + 4+ 2 = 10 numbers (vs. 25 -1 = 31) Doug Downey (adapted from Bryan Pardo, Northwestern University)

Semantics The full joint distribution is defined as the product of the local conditional distributions: P (X 1, … , Xn) = πin= 1 P (Xi | Parents(Xi)) Example: P(j m a b e) = P (j | a) P (m | a) P (a | b, e) P ( b) P ( e) Doug Downey (adapted from Bryan Pardo, Northwestern University)

Learning BB Networks: 3 cases 1. The network structure is given in advance and all the variables are fully observable in the training examples. Trivial Case: just estimate the conditional probabilities. 2. The network structure is given in advance but only some of the variables are observable in the training data. Similar to learning the weights for the hidden units of a Neural Net: Gradient Ascent Procedure 3. The network structure is not known in advance. Use a heuristic search or constraint-based technique to search through potential structures. Doug Downey (adapted from Bryan Pardo, Northwestern University)

Constructing Bayesian networks • 1. Choose an ordering of variables X 1, … , Xn • 2. For i = 1 to n – add Xi to the network – select parents from X 1, … , Xi-1 such that P (Xi | Parents(Xi)) = P (Xi | X 1, . . . Xi-1) This choice of parents guarantees: P (X 1, … , Xn) = πi =1 P (Xi | X 1, … , Xi-1) (chain rule) = πni =1 P (Xi | Parents(Xi)) (by construction) n Doug Downey (adapted from Bryan Pardo, Northwestern University)

Example • Suppose we choose the ordering M, J, A, B, E P(J | M) = P(J)? Doug Downey (adapted from Bryan Pardo, Northwestern University)

Example • Suppose we choose the ordering M, J, A, B, E P(J | M) = P(J)? No P(A | J, M) = P(A | J)? Doug Downey (adapted from Bryan Pardo, Northwestern University)

Example contd. • Deciding conditional independence is hard in noncausal directions – Causal models and conditional independence seem hardwired for humans! • Network is less compact Doug Downey (adapted from Bryan Pardo, Northwestern University)

Inference in BB Networks • A Bayesian Network can be used to compute the probability distribution for any subset of network variables given the values or distributions for any subset of the remaining variables. • Unfortunately, exact inference of probabilities in general for an arbitrary Bayesian Network is known to be NP-hard (#P-complete) • In theory, approximate techniques (such as Monte Carlo Methods) can also be NP-hard, though in practice, many such methods are shown to be useful. Doug Downey (adapted from Bryan Pardo, Northwestern University)

Expectation Maximization Algorithm • Learning unobservable relevant variables • Example: Assume that data points have been uniformly generated from k distinct Gaussian with the same known variance. The problem is to output a hypothesis h=< 1, 2 , . . , k> that describes the means of each of the k distributions. In particular, we are looking for a maximum likelihood hypothesis for these means. • We extend the problem description as follows: for each point xi, there are k hidden variables zi 1, . . , zik such that zil=1 if xi was generated by normal distribution l and ziq= 0 for all q l. Doug Downey (adapted from Bryan Pardo, Northwestern University)

The EM Algorithm (Cont’d) • An arbitrary initial hypothesis h=< 1, 2 , . . , k> is chosen. • The EM Algorithm iterates over two steps: – Step 1 (Estimation, E): Calculate the expected value E[zij] of each hidden variable zij, assuming that the current hypothesis h=< 1, 2 , . . , k> holds. – Step 2 (Maximization, M): Calculate a new maximum likelihood hypothesis h’=< 1’, 2’ , . . , k’>, assuming the value taken on by each hidden variable zij is its expected value E[zij] calculated in step 1. Then replace the hypothesis h=< 1, 2 , . . , k> by the new hypothesis h’=< 1’, 2’ , . . , k’> and iterate. The EM Algorithm can be applied to more general problems Doug Downey (adapted from Bryan Pardo, Northwestern University)

Gibbs Classifier • Bayes optimal classification can be too hard to compute • Instead, randomly pick a single hypothesis (according to the probability distribution of the hypotheses) • use this hypothesis to classify new cases h 2 h 1 h 3 Doug Downey (adapted from Bryan Pardo, Northwestern University)