Probability Intro Professor Jim Ritcey EE 416 Revised

Probability Intro Professor Jim Ritcey EE 416 Revised Oct 6 2009 Please elaborate with your own sketches

Disclaimer • These notes are not complete, but they should help in organizing the class flow. • Please augment these notes with your own sketches and math. You need to actively participate. • It is virtually impossible to learn this from a verbal description or these ppt bullet points. You must create your own illustrations and actively solve problems.

Random Experiments • An experiment is prescribed with N outcomes • finite {1 2 3} countable {1 2 3 …} uncountable {z>0} • A chance mechanism selects outcome – uncertainty • We have some knowledge of the likelihood of occurrence – often measured empirically through relative frequency in repeated independent trials • Kolmogorov developed consistent axiomatic framework for probability

Relationship to Set Theory • Set Theory Probability • Universe • Elements • Algebra of subsets Sample space Outcomes Events • To each event theory allows/assigns a number • Probability (event) indicates its likelihood of occurrence 0<= p <= 1

Set Operations – PPT Notation • Complement A^c = {c: c in S but c not in A} • Union (cup) A+B = { c: c in A or c in B } • Intersection (cap) AB = {c: c in A and c in B } • Often we use cap/cup/overbar in traditional math notation • MATLAB provides set operations as functions very useful when working with large sets

Algebra 0 f Events • Each experiment has outcomes in sample space S • Subsets of outcomes are called events. • The set of events must be an algebra –closure • • For events A, B, C, … then All complements are events A^c, B^c, … All finite unions & intersections are events All countable unions & intersections are events • For finite sample spaces the events consist of the set of all subsets. But too large for infinite S

Probability Triple • (S, E, P) (Sample Space, Events, Probability) • Sample space – set of all possible outcomes • Events – family/collection of all subsets to which we assign probabilities. Subject to closure under set operations • Probability – maps events to unit interval [0, 1] subject to 3 Axioms. For events A, B, C, … • (1) P(A) >= 0 non-negative probability • (2) P(S) = 1 something happens w probability 1 • (3) A, B disjoint P(A+B) = P(A) + P(B) • (3’) Can extend (3) to countable unions

Examples • S = {T, H} = {0, 1} 1 coin toss • E = { S^c = {}, {0}, {1}, S = {0, 1} } • Given P(1) can compute probability of any e in E • S = {TT, HT, TH, HH} = {00, 10, 01, 11} 2 coin toss • E = { S^c = {}, {0}, {1}, S = {0, 1} } • Given P(1) can compute probability of any e in E • Each event can be given an intuitive name

S = {00, 10, 01, 11} {H=1, T=0} e 0 1 2 3 outcomes 1 0 0 Empty set = nothing happens 2 0 0 0 1 Exactly {11} “ 2 heads” 3 0 0 1 0 Exactly {01} “tail then head” 4 0 0 1 1 {01} or {11} “head on second toss} 5 0 1 0 0 Exactly {10} 1 0 0 1 Exactly “ 2 of the same” 1 1 1 0 “Anything but 2 heads” complement of #2 6 7 10 14 15 16 S = something happens

Probability of an Event • Event is a subset of outcomes • An event is realized (occurs) if any of its members occurs • The outcomes of an experiment are by definition disjoint • P (event) is the sum of the probabilities of outcomes that define it • P(event) = P(outcome_1) + … • But events can contain many outcomes

How to we know P • P( any event ) is known in theory. But how? • Relative frequency in independent repeated trials • Uniformity considerations – insufficient reason to suspect otherwise “at random” • Subjective – expert opinion • What is the probability that the sun will rise tomorrow p=1? p<1? • What is the probability that you will obtain above 3. 4 in EE 416? • Engineering usage focuses on relative frequency • Based on past ( but relevant) history

Theorems are derived from Axioms • P(A^c) = 1 - P(A) • P(S) = 1 P(empty set) = 0 • P(A+B) = P(A) + P(B) - P(AB) • P(A+B) < = P(A) + P(B) • A subset of B then P(A) <= P(B) • Proofs are instructive usually they relate to disjoint events

Partitions • • If A us an event, and A in B in S (sample space) Let B_1, …B_N partition B Then P(A) = P(AB_1) + … +P(AB_N) • Simplifies more complicated events

General Problem of Computing Prob • Given a combination of events, with knowing prob of each separately, determine prob of combination • Typical combinations “at least one” “k or more of” • Always can be written as set operations • Always can be enumerated, but can be a long list

1 Coin Toss • • Experiment – toss a coin S –{T, H} |S| =2 Here let the events be the power set Pwr(S) Set of all subsets with 2^2 -4 elements S^c = Empty set, H, T, {H or T} =S closed family Assign P(H) = p, p = 1/2 only for a fair coin Consistent with axioms if p in [0, 1] & P(T) = 1 -p Often insufficient reason to take p neq 1/2 Often p should be measured empirically, the fraction of heads in many trials (how many? ) • The outcomes have prob {p. T= 1 -p, p. H = p}

Geometric Probability • In many board games spinners are used. You cut a circular region into n congruent pizza slices, and flip a spinner. It is obvious that the probability of any outcome is p = 1/n. • This can be argued to be the ratio of the arc length of the wedge to arc length of the circle • Geometric probabilities reduce to calculation of relative length/area/volume and are based on an “at random” assumption. This implies uniformity.

2 Coin Toss • S = {TT, TH, HT, HH} ordered (T 1, T 2) • Pwr(S) has 2^4 = 16 possible events • It is not possible to combine outcomes in any other way. Next assign a consistent probability to • (p 1, p 2, p 3, p 4) p_i in [0, 1] sum p_i = 1 • Insufficient reason takes p_i = ¼ • But it depends on the random experiment (toss) • Allowable is any consistent set of probabilities. • Then eg, P(T 1=Tor. T 2=T) = P(HT + TH + TT) = =P(HT) + P(TH) + P(TT)

Independence • • Two events A, B are independent when P(AB) = P(A)P(B) multiply probabilities This is a property of the probability assignment Examine the 2 coin toss in more detail • Dependent events – statistical relationship/linking • This allows for prediction of one given the other • Not a one-way causal relationship

2 Coin Toss • • • Label the outcomes (00, 01, 10, 11) where T=0, H=1 Suppose we measured P(00) = 0. 10 P(01) = 0. 30 P(10) = 0. 22 P(11) = 0. 38 Are the tosses independent? P(t 1=0)=P(00)+P(01) = 0. 1+0. 3 =0. 40 P(t 1=1)= 1 -. 4 P(t 2=0)=P(00)+P(10) = 0. 1+0. 22=0. 32 P(t 2=1 = 1 -. 32 Independence says that P(t 1=0)P(t 2=0) = P(00) but this does not hold! Under this P, Events are not independent!

Are disjoint events independent? NO • Let A, B mutually exclusive AB = empty set • P(AB) = P(empty set) = 1 -P(S) = 0 • They are not independent • Disjoint events are highly dependent – if one event occurs, the other cannot have occurred • This must be distinguished from independence • P(AB)=P(A)P(B) which might be zero if P(A)=0 • A property of the Prob Assnment, not the events

Communications Networks • • A network is made up of nodes and links (edges) The link either work 1 or fail 0 with p=P(1) q=P(0) Links can be connected in various ways Find Probability P_N that a networks • Series Net: all links work P_N = p^N • Parallel Net: at least 1 link works P_N = 1 - q^N • Need to work through the derivation • Networks can combine series and parallel

Target Detection • A radar takes N looks at a target, target is detected (when present) with p=0. 9 • How does detection prob improve as we increase N • P_N : = 1 –(1 -p)^N but we need to explore numerically • For small N, • P = [ p, 2 p-p^2, 3 p -3 p^2 +p^3, … ] • Is this a series or parallel problem?

Birthday Problem • • • Birthdays occur at random on [1: 365] no leap yr Sample r people’s birthdays, n=365 find P(hit) =P( 2 or more birthdays in sample) P(no hit) = # ways to select wo rep/# ways w rep 1 -P(hit)= (n)_r / n^r where (n)_r = n!/(n-r)! • Numerical implications are interesting

Birthday Problem Numerics • Place r balls at random in n bins. What is the probability that 2 or more land in any bin • N=365, r=23 for a 50: 50 chance of hits • Randomness causes clumping! • Table shows P(hit) vs r, number of balls placed • Balls can be calls, bins can be channels/frequencies 20 0. 4 23 0. 507 30 0. 706 35 0. 814 40 0. 89

Birthday Problem Analysis • • Give a simple expression for P(r, n) = 1 –(n)_r/n^r, when n is large Write Q = 1 -P as a product Q=1 -P(r, n) (n)_r/n^r = Prod_1^(r-1) [1 -j/n] Go to a log form exp( sum of logs) Approximate the sum & simplify This is computational probability • Or just compute use Gamma functions

Birthday Prob (n=365)

Matlab Code • • • function birthdayprob( N, K); %function birthdaypob( N, K); %compute all birthday probabilities. kall =[0: K-1]; %Pnohit =(N)_k/N^k; • • • Pnohit = exp( cumsum( log(1 -kall/N) ) ); Pnoapp = exp( -(kall+1). *(kall)/(2*N) ); % a very good approximation! X = [1: K]'*[ 1 1]; P 1 = [ Pnohit; 1 -Pnohit]'; P 2 = [ Pnoapp; 1 -Pnoapp]'; • • • % note that we are plotting both tails figure(1); semilogy( X, P 1, '-*'); hold on; semilogy( X, P 2, '-o'); hold off title('Exact vs Approx hit Probabilities'); xlabel('Number of Balls Thrown'); grid;

Ranking Problems • You meet Alice, she has two brothers Bob and Chris. You don’t know anything more. • What is the probability that Alice is the oldest? • Use Insufficient Reason to determine • P(A oldest) = 1/3 • Now she tells you that Alice is older than Bob. • Determine P(A is oldest). • Hint: Consider the sample space uses outcomes are all possible rankings of age. (A>B>C, etc)