Theory of Information Lecture 12 More Probability Section

Theory of Information Lecture 12 More Probability (Section 0. 2) 1

Independent Events DEFINITION Two events E and F from the sample space are independent if P(E F)=P(E) P(F). Intuition: If E and F are independent, then knowing that E happened does not change our expectation regarding whether F also happened. Example: Flipping fair coin twice: ({HH, HT, TH, TT}, ¼, ¼) Are the following events independent? E={HH, HT} and F={HT, TT} E={HH, TT} and F={HT, TH} E={HH} and F={HH, HT} 2

Multiple Independent Events DEFINITION Three events E, F and G from the sample space are independent if P(E F G)=P(E) P(F) P(G), and also E and F, E and G, F and G are (pairwise) independent. DEFINITION n events E 1, …, En from the sample space are independent if the probability of the intersection of every subcollection of these events is equal to the product of the probabilities of these events. Intuition: If events are independent, then knowing that any subcollection of these events happened does not change our expectation regarding whether any other remaining events also happened. Example: {HHH, HHT, HTH, HTT, THH, THT, TTH, TTT} with uniform probability. Are the following 3 events independent? {HHH, HHT, HTH, HTT}, {HHH, HHT, THH, THT}, {HHH, HTH, THH, TTH} 3

Example 0. 2. 8(1) Suppose that binary strings of length 5 are sent over a noisy communication line, where the probability of receiving a bit correctly is 0. 75. Assume that the event that one bit is received correctly is independent of the event that another bit is received correctly. 1. What is the probability that a string will be received correctly? Let Ei be the event that the ith bit is received correctly. P(string received correctly) = 4

Example 0. 2. 8(2) Suppose that binary strings of length 5 are sent over a noisy communication line, where the probability of receiving a bit correctly is 0. 75. Assume that the event that one bit is received correctly is independent of the event that another bit is received correctly. 2. What is the probability that exactly 3 bits will be received correctly? Let Ei be the event that the ith bit is received correctly. P(the first 3 bits received correctly) = How many choices are for 3 correct bits? P(exactly 3 bits received correctly) = 5

Example 0. 2. 9(1 -2) Suppose that binary strings of length n are sent over a noisy communication line, where the probability of receiving a bit correctly is p. Assume that the event that one bit is received correctly is independent of the event that another bit is received correctly. Let Ei be the event that the ith bit is received correctly. 1. What is the probability that an entire string will be received correctly? P(string received correctly) = 2. What is the probability that a specified set of k bits (such as the first k bits) will be received correctly while the remaining bits received incorrectly? 6

Example 0. 2. 9(3 -4) Suppose that binary strings of length n are sent over a noisy communication line, where the probability of receiving a bit correctly is p. Assume that the event that one bit is received correctly is independent of the event that another bit is received correctly. 3. What is the probability that exactly k bits (any k bits) will be received correctly? 4. What is the probability that at least k bits will be received correctly? 7

Conditional Probability DEFINITION Assume P(F) 0. The conditional probability of E, given that F has occurred, is P(E F) P(E|F)= -----P(F) Note: P(E F) = P(E|F) P(F) 8

Example 0. 2. 10 A certain surgery results in complete recovery 60% of the time, partial recovery 30% of the time, and death 10% of the time. What is the probability of complete recovery, given that a patient survives the surgery? E --- complete recovery, F --- surviving the surgery. P(F) = What event is E F? E F = P(E|F)= 9

Partitions Let S be a sample space. The events E 1, …, En form a partition of S if the following three conditions are satisfied: 1. 2. 3. The probability of each event Ei is >0; The events are mutually exclusive; The union of those events is S. S S E 1 35% E 2 40% S E 1 35% E 3 25% E 2 40% 35% E 3 15% E 2 45% E 3 25% 10

Theorem of Total Probabilities THEOREM 0. 2. 3 Let S be a sample space, A an event in S, and let E 1, …, En form k=1 a partition of S. Then P(A) = P(A | Ek)P(Ek) n S A 10% E 2 30% 5% 5% E 3 E 1 50% 20% P(A|E 1)= P(A|E 2)= P(A|E 1)= P(A)= 11

Example on Total Probabilities Suppose that bits are sent over a noisy communication line, where 0 is sent (event E 1) 60% of the time, and 1 is sent (event E 2) 40% of the time. Assume that the probability that 0 is received correctly is 80%, and the probability that 1 is received correctly is 90%. Let A be the event that an (arbitrary) bit is received correctly. What is P(A)? S= E 1 = E 2 = A= P(A|E 1)= P(A|E 2)= P(A)= 12

Homework Consider the process of flipping a fair coin three times. Let E be the event that all three flips result in the same outcome, and F the event that the first two flips result in the same outcome. a) b) c) d) e) f) g) h) What is the sample space S and its probability distribution? List all elements of E, and all elements of F. What is the probability of E? What is the probability of F? Are E and F independent? Why yes or no? What is the probability of E, given that F occurred, i. e. P(E|F)=? P(F|E)=? Give two independent events from your sample space (if you answered “yes” to question (e), then you should provide events different from E and F). g) Let G 1 be the event that the first flip results in heads, G 2 the event that the second flip results in heads, and G 3 the event that the third flip results in heads. Do G 1, G 2 and G 3 form a partition of your sample space? Why yes or not? 13