Synthesis and Review 32612 Multiple Comparisons Review of

Synthesis and Review 3/26/12 • Multiple Comparisons • Review of Concepts • Review of Methods - Prezi Essential Synthesis 3 Professor Kari Lock Morgan Duke University

To Do • Study and prepare for Exam 2 (Wednesday and Thursday)

Exam Policies • An exam absence is only excused if a short term illness form is submitted before the exam • In this case, your final exam grade will be substituted • Keep in mind that you will be responsible for a LOT more material on the final exam, and it is already worth 25% of your grade • You can ONLY take the lab exam during your designated section. Set two alarms if needed.

Exam Policies • Any cheating (either on the in-class exam or the lab exam) will result in an automatic 0, and will be treated as a serious case of academic misconduct • This includes, but is not limited to, • Looking at someone else’s exam or computer screen • For the in-class exam, using pages of notes prepared by someone else • Communicating (in any form) with anyone besides myself or your TAs during the exam • Communicating (in any way) with any classmates about the lab exam, or sharing any code or materials related to the lab exam, before 4 pm on Thursday, 3/29

Talk Analytic Approaches to Basketball Mike Zarren (Boston Celtics) Tuesday, 3/27, 5 pm in 2231 French Family Science Michael Zarren is the Boston Celtics’ Assistant General Manager and Associate Team Counsel. Mike is widely recognized as one of the leaders in the field of advanced statistical analysis of basketball players and teams, and is an important part of the team’s strategic planning and player personnel evaluation processes. Mike is also the team’s salary cap expert and lead inhouse counsel, and is responsible for the development of new technologies for team use, including the team’s statistical database and video archive/delivery system. Read more here: http: //goo. gl/l 4 P 3 I.

Office Hours before Exam You have LOTS of opportunities for help! • • Monday, 3 – 4 pm (Prof Morgan) Monday, 4 – 6 pm (Christine) Tuesday, 3 – 6 pm (Prof Morgan) Tuesday, 6 – 8 pm (Yue)

Importing from a Google Doc • RStudio no longer supports importing data from a google doc

Extrasensory Perception • Is there such a thing as ESP? • Let’s find out by conducting our own experiments!

Extrasensory Perception • Get into pairs. • “Randomly” choose A, B, C, or D, and write it down • Try to transmit this information to your partner, without communicating the letter in any way that can be perceived by any of the five senses! • Partner: guess the letter. • Repeat this 10 times each, and keep track of the number of correct guesses. • Once you have n = 20, come to the board and plot your sample proportion • Test whether your experiment provides evidence of ESP

Extrasensory Perception Did your experiment provide evidence of extrasensory perception, using = 0. 05? (a) Yes (b) No

Test for a Proportion Which of the following ways are appropriate to test whether your sample proportion is significantly different from p = ¼? a) b) c) d) e) Randomization Test (only) Normal distribution (only) t-distribution (only) Either (a) or (b) Either (a), (b), or (c)

Randomization Distribution IF there is no such thing as ESP, then you all just created a randomization distribution.

Extrasensory Perception If there is no such thing as ESP, what percentage of experiments on ESP will get results that are significant, using = 0. 05? (a) None (b) All of them (c) 95% (d) 5%

www. causeweb. org Author: JB Landers

www. causeweb. org Author: JB Landers

www. causeweb. org Author: JB Landers

Multiple Comparisons • Consider a topic that is being investigated by research teams all over the world 5% of teams are going to find something significant, even if the null is true

Multiple Comparisons • Consider a research team/company doing many hypothesis tests 5% of tests are going to be significant, even if the nulls are all true

Multiple Comparisons • Consider an experiment that randomizes units to treatment groups, and then looks at many response variables 5% of variables are going to be significantly different between the groups, just by random chance

Pairwise Comparisons • Consider a study with many different treatment groups, and so many possible pairwise comparisons 5% of comparisons are going to be significantly different, even if no differences actually exist (This is the main reason for only testing pairwise comparisons if the overall ANOVA is found to be significant)

Publication Bias • publication bias: usually, only the significant results get published • The one study that turns out significant gets published, and no one knows about all the insignificant results

Jelly Beans Cause Acne! http: //xkcd. com/882/

http: //xkcd. com/882/

http: //xkcd. com/882/

http: //xkcd. com/882/

Multiple Comparisons • This is a serious problem • The most important thing is to simply be aware of this issue, and not to trust claims that are obviously one of many tests (unless they specifically mention an adjustment for multiple testing)

REVIEW

Data Collection Was the sample randomly selected? Yes No Possible to generalize to the population Should not generalize to the population Was the explanatory variable randomly assigned? Yes Possible to make conclusions about causality No Can not make conclusions about causality

Confidence Interval • A confidence interval for a parameter is an interval computed from sample data by a method that will capture the parameter for a specified proportion of all samples • A 95% confidence interval will contain the true parameter for 95% of all samples

Hypothesis Testing • How unusual would it be to get results as extreme (or more extreme) than those observed, if the null hypothesis is true? • If it would be very unusual, then the null hypothesis is probably not true! • If it would not be very unusual, then there is not evidence against the null hypothesis

p-value • The p-value is the probability of getting a statistic as extreme (or more extreme) as that observed, just by random chance, if the null hypothesis is true • The p-value measures evidence against the null hypothesis

Hypothesis Testing 1. State Hypotheses 2. Calculate a test statistic, based on your sample data 3. Create a distribution of this test statistic, as it would be observed if the null hypothesis were true 4. Use this distribution to measure how extreme your test statistic is

Distribution of the Sample Statistic 1. Sampling distribution: distribution of the statistic based on many samples from the population 2. Bootstrap Distribution: distribution of the statistic based on many samples with replacement from the original sample 3. Randomization Distribution: distribution of the statistic assuming the null hypothesis is true 4. Normal, t, 2, F: Theoretical distributions used to approximate the distribution of the statistic

Sample Size Conditions • For large sample sizes, either simulation methods or theoretical methods work • If sample sizes are too small, only simulation methods can be used

Using Distributions • For confidence intervals, you find the desired percentage in the middle of the distribution, then find the corresponding value on the x-axis • For p-values, you find the value of the observed statistic on the x-axis, then find the area in the tail(s) of the distribution

Confidence Intervals

Confidence Intervals Return to original scale with

Hypothesis Testing

General Formulas • When performing inference for a single parameter (or difference in two parameters), the following formulas are used:

Standard Error • The standard error is the standard deviation of the sample statistic • The formula for the standard error depends on the type of statistic (which depends on the type of variable(s) being analyzed)

Multiple Categories • These formulas do not work for categorical variables with more than two categories, because there are multiple parameters • For one or two categorical variables with multiple categories, use 2 tests • For testing for a difference in means across multiple groups, use ANOVA

Inference Methods http: //prezi. com/c 1 xz 1 on-p 4 eb/stat-101/