Bayesian inference of normal distribution Problem statement Objective

Bayesian inference of normal distribution • Problem statement – Objective is to estimate unknown two parameters q={m, s 2} of normal distribution based on observations y = {y 1, y 2, …}. • Prior – Joint pdf of non-informative prior • Joint posterior distribution • This is function of two variables, which can be plotted as a surface or contour. -1 -

Joint posterior distribution • Distribution – There is no inherent pdf function by matlab in this case. • This is function of two variables, which can be plotted as a surface or contour. – Let’s consider a case with n=20; y =2. 9; s=0. 2; • Remark – Analysis of posterior pdf: mean, median & confidence bounds. • Marginal distribution • Once we have the marginal pdf, we can evaluate its mean and confidence bounds. – Posterior prediction: predictive distribution of new y based on observed y. This is likelihood of new y This is posterior pdf of m & s 2 -2 -

Marginal distributions • Analytical solutions – Marginal mean – Marginal variance • How to evaluate marginal distributions – Before doing this, note that in normal dist. , variable y can be transformed to standard normal variable z. – Marginal mean & variance can be evaluated in the same manner. – One can evaluate various characteristics using the matlab functions. -3 -

Marginal distributions • Supplementary information for normal pdf – The original pdf f. Y(y) and normalized pdf f. Z(z) have following relation. – Therefore, – If we want use f(z) instead of f(y), use this relation. • Supplementary information for chi 2 pdf – The original pdf f. Y(y) and normalized pdf f. Z(z) have following relation. – Therefore, -4 -

Posterior prediction • Predictive distribution of new y based on observed y posterior pdf of m & s 2 Likelihood of new y • Analytical solution – One can evaluate characteristics of this using the matlab function. – Compare with the marginal mean. • Marginal mean: t-distribution with location y and variance s^2/n with n-1 dof. • Predictive distribution: t-distribution with location y and variance s^2*(1+1/n) with n-1 dof. -5 -

Simulation of joint posterior distribution • Why ? – Even when analytic solutions available, some are not easy to evaluate. – Once exercised, may find it more convenient and more general. – Practice simulation & validate with analytic solution. • Factorization approach – Review: joint probability of A & B – Likewise, joint posterior pdf of m & s 2 • The two at the right are marginal pdfs. • Simulation by drawing random samples – We can evaluate characteristics of joint posterior pdf using simulation techniques. -6 -

Simulation of joint posterior distribution • Approach 1: use marginal variance. Conditional pdf of m on s 2 Marginal pdf of s 2 – Conditional pdf of m on s 2 is already derived, which is the pdf of mean with known variance. (page 10 of Lec. #4) – Marginal pdf of s 2 is given in this lecture. – In order to sample the posterior pdf of p(m, s 2|y) 1. Draw s 2 from the marginal pdf 2. Draw m from the conditional pdf -7 -

Simulation of joint posterior distribution • Approach 1: use marginal variance. – Once you have obtained samples of joint pdf, compare (validate) the results with the analytic solution. 1. Compare the samples of (M, S 2) with the analytic joint pdf. • • In terms of scattered pot & contour. In terms of 3 -D histogram & surface (mesh). 2. Compare the samples of M with the marginal pdf, which is t distribution. 3. Compare the samples of S 2 with the marginal pdf which is inv-chi 2 distribution. 4. Extract features of the samples M and compare with analytic solution. 5. Extract features of the samples S 2 and compare with analytic solution. -8 -

Simulation of joint posterior distribution • Approach 2: use marginal mean. Conditional pdf of s 2 on m Marginal pdf of m – Conditional pdf of s 2 on m is already derived, which is the pdf of variance with known mean. (page 13 of Lec. #4) – Marginal pdf of m is given in this lecture. – In order to sample the posterior pdf of p(m, s 2|y) 1. Draw m from the marginal pdf 2. Draw s 2 from the conditional pdf – Compare (validate) the resulting samples with the analytic solution. -9 -

Simulation of joint posterior distribution • Approach 2: use marginal mean. – Once you have obtained samples of joint pdf, compare (validate) the results with the analytic solution. – Details are omitted for limited time. - 10 -

Posterior prediction by simulation • Analytic approach – Result of analytic solution by double integral over infinity to get • Simulation – In practice, posterior predictive distribution is obtained by random draws. – Once we have posterior distribution for m & s 2 in the form of samples, the predictive new y are easily obtained by drawing each one from conditional on each individual m & s 2. – Mean & conf. intervals of posterior prediction can be obtained easily. - 11 -

Homework - 12 -

Homework - 13 -

Homework • Problems 1 1) Use the approach 1 to obtain samples of joint pdf (M, S 2). 2) Compare scattered dots with contour of analytic solution. Compare 3 -D histogram with mesh shape of analytic solution. 3) Compare the samples of M with the marginal pdf, which is t distribution. 4) Compare the samples of S 2 with the marginal pdf which is inv-chi 2 dist. 5) Obtain mean, 95% conf. interval of M, & compare with analytic solution. 6) Obtain mean, 95% conf. interval of S 2, & compare with analytic solution. 7) Obtain samples of posterior prediction, and obtain mean, 95% conf. interval of ynew & compare with analytic solution. - 14 -