Theory of Information Lecture 2 Basic Concepts on

Theory of Information Lecture 2 Basic Concepts on Probability (Section 0. 2) 1

Sample Spaces and Events DEFINITION The set of all possible outcomes of a process is called the sample space of that process. Any subset of the sample space, i. e. any set of outcomes, is called an event. Example 1: Process = flipping a coin; sample space = {H, T}. events: {H, T}, {H}, {T}, {} Example 2: Process = rolling a pair of dice; sample space = {(1, 1), (1, 2), …, (2, 1), …, (6, 6)} events: {(1, 1), (2, 2), (3, 3), (4, 4), (5, 5), (6, 6)}, {(1, 3), (3, 1), (2, 2)}, etc. 2

Probability Distribution/Law DEFINITION Let S={s 1, …, sn} be an (ordered) sample space. A probability law for S is an assignment P that sends each si to a real number pi (0 pi 1) such that p 1+…+pn=1. The sequence p 1, …, pn is said to be a probability distribution for S. Example: For the process of flipping a fair coin, an adequate probablity law is P(H)=1/2, P(T)=1/2. If the coin is not fair and H occurs twice as often as T, then we would have 3

Probability of an Event DEFINITION Let E be an event in a sample space S, and P a probability law for P. Then the probability of E, denoted by P(E), is the sum of the probabilities of each outcome in the event. (The empty sum is =0). Example: For fair coin flipping, we have the following probabilities for the following events: P({H})= P({T})= P({H, T})= P({})= 4

Probability of an Event Example: Consider the process of rolling a fair pair of dice. What is P(i, j) for each pair i, j? What is the event of the second die being twice as big as the first one? What is the probability of the above event? What is the probability of the event that both dice are the same? What is the probability of the event that the two dice are different? 5

Uniform Probability Distribution DEFINITION A uniform probability distribution for a sample space of size n is 1/n, …, 1/n. THEOREM 0. 2. 1 Let S be a sample space with uniform probability law. If E is an event in E, |E| P(E) = ----|S| 6

The Basic Properties of Probability THEOREM 0. 2. 2 Let S be a sample space. Then 1. P({}) = 2. P(S) = 3. P(E) 4. P(Ec) = , for all events E S 5. If events E and F are disjoint (mutually exclusive), then P(E F) = More generally, if E 1, …, En are mutually disjoint, then P(E 1 … En) = 7

Example 0. 2. 3 EXAMPLE 0. 2. 3 A black box emits 4 bits, with each bit with equal probability. What is the probability of emitting an even number of 1 s? What is the size of the sample space? How many 4 -bit strings are there with k (0 k 4) 1 s? How many 4 -bit strings are there with an even number of 1 s? So, P(even number of 1 s) = 8

Example 0. 2. 4 EXAMPLE 0. 2. 4 Five cards are drawn, each with equal probability, from a deck of 52 cards. What is the probability of getting exactly 3 aces (event E)? What is the size of the sample space S? What is the size of the event E? P(E) = 9

Example 0. 2. 5 EXAMPLE 0. 2. 4 Five decimal digits a 1, …, a 5 are chosen at random. What is the probability that a 5 is the same as one of the previous 4 digits (event E)? What is the size of the sample space S? What is the size of Ec? P(Ec) = P(E) = 10

Homework 2. 1. A fair coin is flipped 3 times. a) What is the sample space here? (list all of its elements). b) List all of the elements of the event that all flips result in the same outcome (all heads, or all tails). c) What is the probability of the above event? 2. 2. What is the probability that the last card of a deck of 52 cards is an ace? In your answer, indicate the size of the sample space and the size of the event. 2. 3. Four cards are drawn, each with equal probability, from a deck of 52 cards. What is the probability of getting exactly 2 aces? 2. 4. Five (binary) bits b 1, …, b 5 are chosen at random. What is the probability that b 5 is the same as one of the previous 4 bits? 11