Statistical Model A statistical model for some data

Statistical Model • A statistical model for some data is a set of distributions, one of which corresponds to the true unknown distribution that produced the data. • The statistical model corresponds to the information a statistician brings to the application about what the true distribution is or at least what he or she is willing to assume about it. • The variable θ is called the parameter of the model, and the set Ω is called the parameter space. • From the definition of a statistical model, we see that there is a unique value , such that fθ is the true distribution that generated the data. We refer to this value as the true parameter value. STA 248 week 3 1

Examples • Suppose there are two manufacturing plants for machines. It is known that the life lengths of machines built by the first plant have an Exponential(1) distribution, while machines manufactured by the second plant have life lengths distributed Exponential(1. 5). You have purchased five of these machines and you know that all five came from the same plant but do not know which plant. Further, you observe the life lengths of these machines, obtaining a sample (x 1, …, x 5) and want to make inference about the true distribution of the life lengths of these machines. • Suppose we have observations of heights in cm of individuals in a population and we feel that it is reasonable to assume that the distribution of height is the population is normal with some unknown mean and variance. The statistical model in this case is where Ω = R×R+, where R+ = (0, ∞). STA 248 week 3 2

Goals of Statistics • Estimate unknown parameters of underlying probability distribution. • Measure errors of these estimates. • Test whether data gives evidence that parameters are (or are not) equal to a certain value or the probability distribution have a particular form. STA 248 week 3 3

Point Estimation • Most statistical procedures involve estimation of the unknown value of the parameter of the statistical model. • A point estimator of the parameter θ is a function of the underlying random variables and so it is a random variable with a distribution function. • A point estimate of the parameter θ is a function on the data; it is a statistic. For a given sample an estimate is a number. • Notation… STA 248 week 3 4

What Makes a Good Estimator? • Unbiased • Consistent • Minimum variance • With know probability distribution STA 248 week 3 5

Properties of Point Estimators - Unbiased • Let be a point estimator for a parameter θ. Then estimator if is an unbiased • There may not always exist an unbiased estimator for θ. • unbiased for θ, does not mean is unbiased for g(θ). STA 248 week 3 6

Example - Common Point Estimators • A natural estimate for the population mean μ is the sample mean (in any distribution). The sample mean is an unbiased estimator of the population mean. • There are two common estimator for the population variance … STA 248 week 3 7

Claim • Let X 1, X 2, …, Xn be random sample of size n from a population with mean μ and variance σ2. The sample variance s 2 is an unbiased estimator of the population variance σ2. • Proof… STA 248 week 3 8

Example • Suppose X 1, X 2, …, Xn is a random sample from U(0, θ) distribution. Let. Find the density of and its mean. Is unbiased? STA 248 week 3 9

Asymptotically Unbiased Estimators • An estimator is asymptotically unbiased if • Example: STA 248 week 3 10

Consistency • An estimator is a consistent estimator of θ, if converge in probability to θ. STA 248 week 3 , i. e. , if 11

Minimum Variance • An estimator for θ is a function of underlying random variables and so it is a random variable and has its own probability distribution function. • This probability distribution is called sampling distribution. • We can use the sampling distribution to get variance of an estimator. • A better estimate has smaller variance; it is more likely to produce estimated close to the true value of the parameter if it is unbiased. • The standard deviation of the sampling distribution of an estimator is usually called the standard error of the estimator. STA 248 week 3 12

Examples STA 248 week 3 13

How to find estimators? • There are two main methods for finding estimators: 1) Method of moments. 2) The method of Maximum likelihood. • Sometimes the two methods will give the same estimator. STA 248 week 3 14

Method of Moments • The method of moments is a very simple procedure for finding an estimator for one or more parameters of a statistical model. • It is one of the oldest methods for deriving point estimators. • Recall: the k moment of a random variable is These will very often be functions of the unknown parameters. • The corresponding k sample moment is the average. • The estimator based on the method of moments will be the solutions to the equation μk = mk. STA 248 week 3 15

Examples STA 248 week 3 16

The Likelihood Function • Let x 1, …, xn be sample observations taken on corresponding random variables X 1, …, Xn whose distribution depends on a parameter θ. The likelihood function defined on the parameter space Ω is given by • Note that for the likelihood function we are fixing the data, x 1, …, xn, and varying the value of the parameter. • The value L(θ | x 1, …, xn) is called the likelihood of θ. It is the probability of observing the data values we observed given that θ is the true value of the parameter. It is not the probability of θ given that we observed x 1, …, xn. STA 248 week 3 17

Maximum Likelihood Estimators • In the likelihood function, different values of θ will attach different probabilities to a particular observed sample. • The likelihood function, L(θ | x 1, …, xn), can be maximized over θ, to give the parameter value that attaches the highest possible probability to a particular observed sample. • We can maximize the likelihood function to find an estimator of θ. • This estimator is a statistics – it is a function of the sample data. It is denoted by STA 248 week 3 18

The log likelihood function • l(θ) = ln(L(θ)) is the log likelihood function. • Both the likelihood function and the log likelihood function have their maximums at the same value of • It is often easier to maximize l(θ). STA 248 week 3 19

Examples STA 248 week 3 20

Properties of MLE • Maximum likelihood estimators (MLEs) are consistent. • The MLE of any parameter is asymptotically unbiased. • MLE has variance that is nearly as small as can be achieved by any estimator (asymptotically). • Distribution of MLSs is approximately Normal (asymptotically). STA 248 week 3 21