Shrinkage Greg Francis PSY 626 Bayesian Statistics for

Shrinkage Greg Francis PSY 626: Bayesian Statistics for Psychological Science Fall 2020 Purdue University

Estimation l Suppose you want to estimate a population mean from a normal distribution l The standard approach is to use the sample mean l Indeed, the sample mean is an unbiased estimate of the population mean l It is the value that minimizes squared error, relative to the sample: l l It is also the Maximum Likelihood Estimate of the population mean It is hard to do better than this! X 1 X 2 X 3 X 4

Estimation l Suppose you want to estimate the population mean from a normal distribution with mean μ and standard deviation σ l You want to minimize squared error: l You cannot do better (on average) than the sample mean l For sample size n, the expected squared error will be

Estimation l l Suppose you want to estimate p=5 population means by minimizing mean squared error You might try using the sample means Suppose the populations all have the same standard deviation but different means μi This would give an MSE of

Sample means l Let’s try some simulations w n=20 for each sample w σ=1 (assumed to be known to us) w μi selected from a normal distribution N(0, 1) l l Average MSE=0. 2456 You might think this is as good as you can do w But you can do rather better

Shrinkage l l Consider a different (James-Stein) estimate of the population means [Fix throughout should be sigma ^2 in numerator] This estimate produces a population mean estimate closer to 0 than the sample mean

Shrinkage l l l Consider a different (James-Stein) estimate of the population means This estimate produces a population mean estimate closer to 0 than the sample mean (Sometimes just a bit)

Shrinkage l l Consider a different (James-Stein) estimate of the population means This estimate produces a population mean estimate closer to 0 than the sample mean

Shrinkage l Shrinking each sample mean toward 0 decreases (on average) the MSE l Average MSE(Sample means)=0. 2456 l Average MSE (James-Stein)=0. 2389 l l In fact, the James-Stein estimator has a smaller MSE than the sample means for 62% of the simulated cases This is true despite the fact that the sample mean is a better estimate for each individual mean!

Shrinkage l You might think the James-Stein estimators work only by shrinking every estimate toward the mean of the population means w [μi selected from a normal distribution N(0, 1)] l l There is an advantage when you have such information, but shrinkage can help (on average) by shrinking toward any value When v is far from the sample means, the denominator tends to be large, so the adjustment is close to 1

Shrinkage l l Set v=3 Not much change, but still a small drop in MSE

Shrinkage l l l Set v=3 Not much change, but here a small decrease in MSE Average MSE for v=3 is 0. 2451 (Compared to 0. 2456 for sample means) These estimates beat the sample means in 54% of cases

Shrinkage l l l Set v=30 Not much change, but still a small drop in MSE Average MSE for v=30 is 0. 2456 (Compared to 0. 2456 for sample means) These estimates beat the sample means in 50. 2% of cases

Shrinkage l l l There is a “best” value of v that will minimize MSE, but you do not know it Even a bad choice tends to help (on average), albeit not as much as is possible, and it does not hurt (on average) There is a trade off w You have smaller MSE on average w You increase MSE for some cases and decrease it for other cases w You never know which case you are in w These are biased estimates of the population means

Shrinkage l l These effects have odd philosophical implications If you want to estimate a single mean (e. g. , the IQ of psychology majors, mathematics majors, English majors, physics majors, or economics majors) w You are best to use the sample mean l If you want to jointly estimate the mean IQs of these majors w You are best to use the James-Stein estimator l Of course, you get different answers for the different questions! w Which one is right? There is no definitive answer! w What is truth?

Shrinkage l It is actually even weirder! l If you want to jointly estimate w Family income w Mean number of windows in apartments in West Lafayette w Mean height of chairs in a classroom w Mean number of words in economics textbook sentences w The speed of light l You are better off using a James-Stein estimator than the sample means w Even though the measures are unrelated to each other! w (You may not get much benefit, because the scales are so very different. ) w You get different values for different units

Using wisely l You get more benefit with more means being estimated together w The more complicated your measurement situation, the better you do! Compare to hypothesis testing, where controlling Type I error becomes harder with more comparisons!

Data driven l l l There is a “best” value of v that will minimize MSE, but you do not know it Oftentimes, a good guess is the average of the sample means Called the Morris-Efron estimator w Sometimes use (p-3) instead of (p-2)

Morris-Efron estimators l l Not quite as good as James-Stein when population means come from N(0, 1) l n=20, p=5 l MSE(samples means) = 0. 2456 l MSE(James-Stein)= 0. 2389, beats sample means 62% l MSE(Morris-Efron) = 0. 2407, beats sample means 59% Better if population means come from N(7, 1) l n=20, p=5 l MSE(samples means) = 0. 2456 l MSE(James-Stein)= 0. 2454, beats sample means 51% l MSE(Morris-Efron) = 0. 2407, beats sample means 59%

Why does shrinkage work? l MSE can be split up into bias and variance l Consider a single mean, i, l Expand the square

Bias and variance l l l Apply the expectation to the terms The last term is unaffected by the expectation (it is a constant) In the middle term, the second difference is a constant and can be pulled out, same for the number 2

Bias and variance l l Apply the expectation to the middle terms The double expectation is just the expectation, so the middle term equals 0 l The term on the left is variance of the estimator l The term on the right is the square of bias

MSE and shrinkage l Sample means are unbiased estimators of population means, so the second term equals 0 w MSE is all about variance l James-Stein and Morris-Efron estimators are biased estimators, but they have a smaller variance than sample means (because they are shrunk toward a fixed value) w The trade-off works only for p>2

Prediction l Shrinkage works for prediction too! l Suppose you have 5 population means drawn from N(0, 1) l You take samples of n=20 for each population l You want to predict sample means for replication samples, Y l Your criterion for the quality of prediction is MSE

Prediction l l Averaged across 1000 simulations l n=20, p=5 l MSErep(samples means) = 0. 5028 l MSErep (James-Stein)= 0. 4968, beats sample means 57% l MSErep (Morris-Efron) = 0. 4988, beats sample means 54% Larger MSE values than previous simulations because there is variability in the initial sample and in the replication sample

Conclusions l l l Shrinkage demonstrates that there is no universally appropriate estimator or predictor How to estimate/predict depends on what you want to estimate/predict Shrinkage can be used in frequentist approaches to statistics w Lasso, regularization, ridge regression, hierarchical models, (random effects) meta-analysis l Shrinkage falls out naturally from Bayesian approaches to statistics w Hierarchical models w Priors do not matter very much!