SOME GENERAL PROBLEMS 1 Problem A certain lion
SOME GENERAL PROBLEMS 1
Problem • A certain lion has three possible states of activity each night; they are ‘very active’ (denoted by θ 1), ‘moderately active’ (denoted by θ 2), and ‘lethargic (lacking energy)’ (denoted by θ 3). Also, each night this lion eats people; it eats i people with probability p(i|θ), θ ϵ Θ={θ 1, θ 2, θ 3}. Of course, the probability distribution of the number of people eaten depends on the lion’s activity state θ ϵ Θ. The numeric values are given in the following table. 2
Problem i p(i|θ 1) p(i|θ 2) p(i|θ 3) 0 0 0. 05 0. 9 1 0. 05 0. 08 2 0. 05 0. 8 0. 02 3 0. 8 0. 1 0 4 0. 1 0 0 If we are told X=x 0 people were eaten last night, how should we estimate the lion’s activity state (θ 1, θ 2 or θ 3)? 3
Solution • One reasonable method is to estimate θ as that in Θ for which p(x 0|θ) is largest. In other words, the θ ϵ Θ that provides the largest probability of observing what we did observe. : the MLE of θ based on X (Taken from “Dudewicz and Mishra, 1988, Modern Mathematical Statistics, Wiley”) 4
Problem • Consider the Laplace distribution centered at the origin and with the shape parameter β, which for all x has the p. d. f. Find MME and MLE of β. 5
Problem • Let X 1, …, Xn be independent r. v. s each with lognormal distribution, ln N( , 2). Find the MMEs of , 2 6
STATISTICAL INFERENCE PART III BETTER OR BEST ESTIMATORS, FISHER INFORMATION, CRAMERRAO LOWER BOUND (CRLB) 7
RECALL: EXPONENTIAL CLASS OF PDFS • If the pdf can be written in the following form then, the pdf is a member of exponential class of pdfs. (Here, k is the number of parameters) 8
EXPONENTIAL CLASS and CSS • Random Sample from Regular Exponential Class is a css for . 9
RAO-BLACKWELL THEOREM Let X 1, X 2, …, Xn have joint pdf or pmf f(x 1, x 2, …, xn; ) and let S=(S 1, S 2, …, Sk) be a vector of jss for . If T is an UE of ( ) and (S)=E(T S), then i) (S) is an UE of ( ). ii) (S) is a fn of S, so it is free of . iii) Var( (S) ) Var(T) for all . • (S) is a better unbiased estimator of ( ). • 10
RAO-BLACKWELL THEOREM • Notes: • (S)=E(T S) is at least as good as T. • For finding the best UE, it is enough to consider UEs that are functions of a ss, because all such estimators are at least as good as the rest of the UEs. 11
Example • • • Hogg & Craig, Exercise 10. 10 X 1, X 2~Exp(θ) Find joint p. d. f. of ss Y 1=X 1+X 2 for θ and Y 2=X 2. Show that Y 2 is UE of θ with variance θ². Find φ(y 1)=E(Y 2|Y 1) and variance of φ(Y 1). 12
THE MINIMUM VARIANCE UNBIASED ESTIMATOR • Rao-Blackwell Theorem: If T is an unbiased estimator of , and S is a ss for , then (S)=E(T S) is – an UE of , i. e. , E[ (S)]=E[E(T S)]= and – with a smaller variance than Var(T). 13
LEHMANN-SCHEFFE THEOREM • Let Y be a css for . If there is a function Y which is an UE of , then the function is the unique Minimum Variance Unbiased Estimator (UMVUE) of . • Y css for . • T(y)=fn(y) and E[T(Y)]=. ®T(Y) is the UMVUE of . ®So, it is the best unbiased estimator of . 14
THE MINIMUM VARIANCE UNBIASED ESTIMATOR • Let Y be a css for . Since Y is complete, there could be only a unique function of Y which is an UE of . • Let U 1(Y) and U 2(Y) be two function of Y. Since they are UE’s, E(U 1(Y) U 2(Y))=0 imply W(Y)=U 1(Y) U 2(Y)=0 for all possible values of Y. Therefore, U 1(Y)=U 2(Y) for all Y. 15
Example • Let X 1, X 2, …, Xn ~Poi(μ). Find UMVUE of μ. • Solution steps: – Show that is css for μ. – Find a statistics (such as S*) that is UE of μ and a function of S. – Then, S* is UMVUE of μ by Lehmann-Scheffe Thm. 16
Note • The estimator found by Rao-Blackwell Thm may not be unique. But, the estimator found by Lehmann-Scheffe Thm is unique. 17
RECALL: EXPONENTIAL CLASS OF PDFS • If the pdf can be written in the following form then, the pdf is a member of exponential class of pdfs. (Here, k is the number of parameters) 18
EXPONENTIAL CLASS and CSS • Random Sample from Regular Exponential Class is a css for . If Y is an UE of , Y is the UMVUE of . 19
EXAMPLES Let X 1, X 2, …~Bin(1, p), i. e. , Ber(p). This family is a member of exponential family of distributions. is a CSS for p. is UE of p and a function of CSS. is UMVUE of p. 20
EXAMPLES X~N( , 2) where both and 2 is unknown. Find a css for and 2. 21
FISHER INFORMATION AND INFORMATION CRITERIA • X, f(x; ), , x A (not depend on ). Definitions and notations: 22
FISHER INFORMATION AND INFORMATION CRITERIA The Fisher Information in a random variable X: The Fisher Information in the random sample: Let’s prove the equalities above. 23
FISHER INFORMATION AND INFORMATION CRITERIA 24
FISHER INFORMATION AND INFORMATION CRITERIA 25
FISHER INFORMATION AND INFORMATION CRITERIA The Fisher Information in a random variable X: The Fisher Information in the random sample: Proof of the last equality is available on Casella & Berger (1990), pg. 310 -311. 26
CRAMER-RAO LOWER BOUND (CRLB) • • Let X 1, X 2, …, Xn be sample random variables. Range of X does not depend on . Y=U(X 1, X 2, …, Xn): a statistic; does’nt contain . Let E(Y)=m( ). • Let prove this! 27
CRAMER-RAO LOWER BOUND (CRLB) • -1 Corr(Y, Z) 1 • 0 Corr(Y, Z)2 1 • Take Z= ′(x 1, x 2, …, xn; ) • Then, E(Z)=0 and V(Z)=In( ) (from previous slides). 28
CRAMER-RAO LOWER BOUND (CRLB) • Cov(Y, Z)=E(YZ)-E(Y)E(Z)=E(YZ) 29
CRAMER-RAO LOWER BOUND (CRLB) • E(Y. Z)=mʹ( ), Cov(Y, Z)=mʹ( ), V(Z)=In( ) The Cramer-Rao Inequality (Information Inequality) 30
CRAMER-RAO LOWER BOUND (CRLB) • CRLB is the lower bound for the variance of an unbiased estimator of m( ). • When V(Y)=CRLB, Y is the MVUE of m( ). • For a r. s. , remember that In( )=n I( ), so, 31
ASYMPTOTIC DISTRIBUTION OF MLEs • : MLE of • X 1, X 2, …, Xn is a random sample. 32
EFFICIENT ESTIMATOR • T is an efficient estimator (EE) of if – T is UE of , and, – Var(T)=CRLB • T is an efficient estimator (EE) of its expectation, m( ), if its variance reaches the CRLB. • An EE of m( ) may not exist. • The EE of m( ), if exists, is unique. • The EE of m( ) is the unique MVUE of m( ). 33
ASYMPTOTIC EFFICIENT ESTIMATOR • Y is an asymptotic EE of m( ) if 34
EXAMPLES A r. s. of size n from X~Poi(θ). a) Find CRLB for any UE of θ. b) Find UMVUE of θ. c) Find an EE for θ. d) Find CRLB for any UE of exp{-2θ}. Assume n=1, and show that is UMVUE of exp{2θ}. Is this a reasonable estimator? 35
EXAMPLE A r. s. of size n from X~Exp( ). Find UMVUE of , if exists. 36
Summary • We covered 3 methods for finding good estimators (possibly UMVUE): – Rao-Blackwell Theorem (Use a ss T, an UE U, and create a new statistic by E(U|T)) – Lehmann-Scheffe Theorem (Use a css T which is also UE) – Cramer-Rao Lower Bound (Find an UE with variance=CRLB) 37
Problems • Let be a random sample from gamma distribution, Xi~Gamma(2, θ). The p. d. f. of X 1 is given by: a) Find a complete and sufficient statistic for θ. b) Find a minimal sufficient statistic for θ. c) Find CRLB for the variance of an unbiased estimator of θ. d) Find a UMVUE of θ. 38
Problems • Suppose X 1, …, Xn are independent with density for θ>0 a) Find a complete sufficient statistic. b) Find the CRLB for the variance of unbiased estimators of 1/θ. c) Find the UMVUE of 1/θ if there is one. 39
- Slides: 39