Part 2 b Parameter Estimation CSE 717 FALL

Part 2 b Parameter Estimation CSE 717, FALL 2008 CUBS, Univ at Buffalo

Parametric Distribution n 2 -Class Problem Given class labels {c 1, c 2} and random variable X, the posterior probability of class C is n Parametric Representation of Distribution n The conditional p. d. f is usually represented by a math expression of x with various parameters θ 1, θ 2, …

Parametric Representation of Distributions n The conditional p. d. f is usually represented by a math expression of x with various parameters θ 1, θ 2, … n Parameter Estimation Estimate parameters from samples of X n Example: Normal Distribution n p. d. f n Parameters: ,

Maximum Likelihood Estimation Given X, p. d. f. p. X(x; θ), n values x 1, …, xn obtained by independent samplings X 1, …, Xn: the Maximum Likelihood Estimation of θ is given by

Maximum Likelihood Estimation (Cont. ) By independence assumption

Normal Distribution with Unknown and Let

Normal Distribution with Unknown and (Cont. ) Let and

Bias of Estimator n An estimator of is unbiased if biased otherwise. n is an unbiased estimator of Proof ; is

Bias of Estimator (Cont. ) is a biased estimator of n Proof

Bias of Estimator (Cont. ) is an unbiased n estimator of Proof

Bias of Estimator (Cont. ) n is an asymptotically unbiased estimator of n Proof if

Variance of Estimator n The variance of n of

Mean Square Error n Mean Square Error of Estimator n Relation between MSE, Bias and Variance

Bias/Variance Dilemma n Bias: the quality of the estimator n Variance: the consistency of the estimator at different groups of selected samples n MSE: overall quality of the estimator n Low bias sometimes leads to high variance and high MSE n Overfitting/overtraining problem

Biased vs. Unbiased Estimators § n n : biased; : unbiased; n n Unbiased estimators are NOT always desirable