STAT 250 Dr Kari Lock Morgan Hypothesis Testing

STAT 250 Dr. Kari Lock Morgan Hypothesis Testing: p -value SECTION 4. 2 • p-value Statistics: Unlocking the Power of Data Lock 5

Two Plausible Explanations �If the sample data support the alternative, there are two plausible explanations: 1. The alternative hypothesis (Ha) is true 2. The null hypothesis (H 0) is true, and the sample results were just due to random chance �Do the data provide enough evidence to rule out #2? �We need a way to quantify evidence against the null… Statistics: Unlocking the Power of Data Lock 5

Question #1 of the Day Is Desipramine or Lithium better at treating cocaine addiction? Is each better than a placebo? Statistics: Unlocking the Power of Data Lock 5

Cocaine Addiction • In a randomized experiment on treating cocaine addiction, 48 people were randomly assigned to take either Desipramine (a new drug), or Lithium (an existing drug), and then followed to see who relapsed • Question of interest: Is Desipramine better than Lithium at treating cocaine addiction? Statistics: Unlocking the Power of Data Lock 5

Cocaine Addiction Is Desipramine better than Lithium at treating cocaine addiction (preventing relapses)? What parameter(s) are we interested in? a) Proportion b) Mean c) Difference in proportions d) Difference in means e) Correlation Statistics: Unlocking the Power of Data Lock 5

Cocaine Addiction Is Desipramine better than Lithium at treating cocaine addiction (preventing relapses)? p. D: proportion of cocaine addicts who relapse after Desipramine p. L: proportion of cocaine addicts who relapse after Lithium What are the relevant hypotheses? a) H 0: p. D = p. L, Ha: p. D ≠ p. L H 0: p. D = p. L, Ha: p. D < p. L c) H 0: p. D = p. L, Ha: p. D > p. L d) H 0: p. D < p. L, Ha: p. D = p. L e) H 0: p. D > p. L, Ha: p. D = p. L b) Statistics: Unlocking the Power of Data Lock 5

R R R R R R R R R R R R 1. Randomly assign units to treatment groups Desipramine R R Lithium R R R R R R R R R R Statistics: Unlocking the Power of Data Lock 5

2. Conduct experiment 3. Observe relapse counts in each group R = Relapse N = No Relapse 1. Randomly assign units to treatment groups Desipramine Lithium R R R R N R R R R N R N N R R R N N N 10 relapse, 14 no relapse Statistics: Unlocking the Power of Data 18 relapse, 6 no relapse Lock 5

Measuring Evidence against H 0 To see if a statistic provides evidence against H 0, we need to see what kind of sample statistics we would observe, just by random chance, if H 0 were true Statistics: Unlocking the Power of Data Lock 5

R R R R N N R R R N N N N 10 relapse, 14 no relapse Statistics: Unlocking the Power of Data 18 relapse, 6 no relapse Lock 5

R R R R N N R R R N N N N Simulate another randomization Desipramine Lithium R N N N N R R R R N N N R R R N N N R R R R 16 relapse, 8 no relapse Statistics: Unlocking the Power of Data 12 relapse, 12 no relapse Lock 5

Simulate another randomization Desipramine Lithium R R R R N R R N N R R R R N R N R R N N N 17 relapse, 7 no relapse Statistics: Unlocking the Power of Data 11 relapse, 13 no relapse Lock 5

Distribution of Statistic under H 0 How extreme is our observed statistic of 0. 333? Statistics: Unlocking the Power of Data Lock 5

p-value The p-value is the chance of obtaining a sample statistic as extreme (or more extreme) than the observed sample statistic, if the null hypothesis is true Statistics: Unlocking the Power of Data Lock 5

Calculating a p-value 1. What kinds of statistics would we get, just by random chance, if the null hypothesis were true? (randomization distribution) 2. What proportion of these statistics are as extreme as our original sample statistic? (p-value) Statistics: Unlocking the Power of Data Lock 5

Cocaine Addiction Distribution of statistic if H 0 true p-value Proportion as extreme as observed statistic If the two drugs are equal regarding cocaine relapse rates, we have a 2% chance of seeing a difference in proportions as extreme as we observed. Statistics: Unlocking the Power of Data Lock 5

p-value: The chance of obtaining a statistic as extreme as that observed, just by random chance, if the null hypothesis is true Statistics: Unlocking the Power of Data Lock 5

Cocaine Addiction �In the cocaine addiction experiment, people were actually randomized to one of three groups: Desipramine, Lithium, or Placebo �Does Desipramine do better than just a placebo at preventing relapses? �Does Lithium do better than just a placebo at preventing relapses? Statistics: Unlocking the Power of Data Lock 5

Desipramine vs Placebo Distribution of statistic if H 0 true p-value observed statistic If there were no difference between Desipramine and placebo regarding cocaine relapses, we would only see a difference as extreme as that observed 1 out of 1000 times. Statistics: Unlocking the Power of Data Lock 5

Plausible Null? Based on the Despiramine vs placebo results, do you think the null of no difference is plausible? a) Yes b) No Statistics: Unlocking the Power of Data Lock 5

Lithium vs Placebo Distribution of statistic if H 0 true p-value observed statistic If there were no difference between Lithium and placebo regarding cocaine relapses, we would see a difference as extreme as the one observed about 34% of the time. Statistics: Unlocking the Power of Data Lock 5

Plausible Null? Based on the Lithium vs placebo results, do you think the null of no difference is plausible? a) Yes b) No Statistics: Unlocking the Power of Data Lock 5

Question #2 of the Day Is sleep or caffeine better for memory? Statistics: Unlocking the Power of Data Lock 5

Sleep versus Caffeine • Students were given words to memorize, then randomly assigned to take either a 90 min nap, or a caffeine pill. 2 ½ hours later, they were tested on their recall ability. • Explanatory variable: sleep or caffeine • Response variable: number of words recalled Mednick, Cai, Kanady, and Drummond (2008). “Comparing the benefits of caffeine, naps and placebo on verbal, motor and perceptual memory, ” Behavioral Brain Research, 193, 79 -86. Statistics: Unlocking the Power of Data Lock 5

Sleep versus Caffeine What is the parameter of interest in the sleep versus caffeine experiment? a) Proportion b) Difference in proportions c) Mean d) Difference in means e) Correlation Statistics: Unlocking the Power of Data Lock 5

Sleep versus Caffeine • Let s and c be the true mean number of words recalled after sleeping and after caffeine. • Is sleep better than caffeine for memory? • What are the null and alternative hypotheses? a) H 0: s ≠ c, Ha: s = c b) H 0: s = c, Ha: s ≠ c H 0: s ≠ c, Ha: s > c d) H 0: s = c, Ha: s > c e) H 0: s = c, Ha: s < c c) Statistics: Unlocking the Power of Data Lock 5

Sleep versus Caffeine Statistics: Unlocking the Power of Data Lock 5

Sleep or Caffeine for Memory? Words 9 11 13 14 14 15 16 17 17 18 18 21 Group sleep sleep sleep Words 6 7 10 10 12 12 13 14 14 15 16 18 Group caffeine caffeine caffeine Is sleep actually better than caffeine for memory, or did the people with better memory just happen to get randomly assigned to sleep? sleep mean caffeine mean = 15. 25 = 12. 25 sleep mean – caffeine mean = 3 Statistics: Unlocking the Power of Data Lock 5

Sleep or Caffeine for Memory? Words 9 11 13 14 14 15 16 17 17 18 18 21 Group sleep sleep sleep Words 6 7 10 10 12 12 13 14 14 15 16 18 Group caffeine caffeine caffeine What kinds of results would you see, just by random chance, if there were no difference between sleep and caffeine? mean = 15. 25 mean = 12. 25 Statistics: Unlocking the Power of Data Lock 5

Sleep or Caffeine for Memory? Words 9 11 13 14 14 15 16 17 17 18 18 21 Group sleep caffeine sleep caffeine sleep Words 6 7 10 10 12 12 13 14 14 15 16 18 sleep mean = 14. 25 caffeine mean = 13. 25 Group caffeine sleep caffeine caffeine sleep caffeine What kinds of results would you see, just by random chance, if there were no difference between sleep and caffeine? sleep mean – caffeine mean = 1 Statistics: Unlocking the Power of Data Lock 5

Sleep versus Caffeine Distribution of statistic if H 0 true p-value observed statistic If there were no difference between sleep and caffeine regarding memory, we would see a difference as extreme as the one observed about 2% of the time. Statistics: Unlocking the Power of Data Lock 5

Sleep versus Caffeine • Let s and c be the true mean number of words recalled after sleeping and after caffeine. • Now: Is there a difference in average word recall between sleep and caffeine? • What are the null and alternative hypotheses? a) H 0: s ≠ c, Ha: s = c b) H 0: s = c, Ha: s ≠ c H 0: s ≠ c, Ha: s > c d) H 0: s = c, Ha: s > c e) H 0: s = c, Ha: s < c c) Statistics: Unlocking the Power of Data Lock 5

Alternative Hypothesis • A one-sided alternative contains either > or < • A two-sided alternative contains ≠ • The p-value is the proportion in the tail in the direction specified by Ha • For a two-sided alternative, the p-value is twice the proportion in the smallest tail Statistics: Unlocking the Power of Data Lock 5

p-value and Ha Upper-tail (Right Tail) Lower-tail (Left Tail) Two-tailed Statistics: Unlocking the Power of Data Lock 5

Sleep or Caffeine for Memory? www. lock 5 stat. com/statkey Statistics: Unlocking the Power of Data p-value = 2 × 0. 022 = 0. 044 Lock 5

p-value and H 0 �The p-value measures how extreme the observed result would be, if the null hypothesis were true �If the p-value is small, then a statistic as extreme as that observed would be unlikely if the null hypothesis were true, providing evidence against H 0 �The smaller the p-value, the stronger the evidence against the null hypothesis (and in favor of the alternative) Statistics: Unlocking the Power of Data Lock 5

p-value and H 0 Which of the following p-values gives the strongest evidence against H 0? a) 0. 005 b) 0. 1 c) 0. 32 d) 0. 56 e) 0. 94 Statistics: Unlocking the Power of Data Lock 5

p-value and H 0 Which of the following p-values gives the strongest evidence against H 0? a) 0. 22 b) 0. 45 c) 0. 03 d) 0. 8 e) 0. 71 Statistics: Unlocking the Power of Data Lock 5

p-value and H 0 Two different studies obtain two different pvalues. Study A obtained a p-value of 0. 002 and Study B obtained a p-value of 0. 2. Which study obtained stronger evidence against the null hypothesis? a) Study A b) Study B Statistics: Unlocking the Power of Data Lock 5

p-value and H 0 The smaller thep-value, The smaller the p-value, the stronger the evidence against H o. the stronger the evidence the stronger the against H. o evidence against H o. Statistics: Unlocking the Power of Data Lock 5

Formal Decisions A formal hypothesis test has only two possible conclusions: 1. The p-value is small: reject the null hypothesis in favor of the alternative 2. The p-value is not small: do not reject the null hypothesis How small? Statistics: Unlocking the Power of Data Lock 5

To Do �Read Section 4. 2 �HW 4. 2 due Friday 10/23 Statistics: Unlocking the Power of Data Lock 5