Power The Four Components to a Statistical Conclusion

Power

The Four Components to a Statistical Conclusion Sample size The number of units (e. g. , people) accessible to study Effect size The salience of the program relative to the noise Alpha level The odds the observed result is due to chance Power The odds you’ll observe a treatment effect when it occurs

The Four Components to a Statistical Conclusion Sample size Amount of information Effect size Salience of program Alpha level Willingness to risk Power Ability to see effect that’s there

The Effect Size Is a ratio of. . .

The Effect Size Is a ratio of. . . Signal Noise

The Effect Size Is a ratio of. . . Signal Noise Difference between groups Standard error of the difference

Given Values for Any Three, You Can Compute the Fourth. l l n = f(effect size, a, power) effect size = f(n, a, power) a = f(n, effect size, power) power = f(n, effect size, a)

The Decision Matrix In reality What we conclude

The Decision Matrix In reality What we conclude Null true Alternative false In reality. . . • • • There is no real program effect There is no difference, gain Our theory is wrong

The Decision Matrix In reality What we conclude Accept null Reject alternative We say. . . • • • There is no real program effect There is no difference, gain Our theory is wrong Null true Alternative false In reality. . . • • • There is no real program effect There is no difference, gain Our theory is wrong

The Decision Matrix In reality What we conclude Accept null Reject alternative We say. . . • • • There is no real program effect There is no difference, gain Our theory is wrong Null true Alternative false In reality. . . • • • There is no real program effect There is no difference, gain Our theory is wrong 1 - THE CONFIDENCE LEVEL The odds of saying there is no effect or gain when in fact there is none # of times out of 100 when there is no effect, we’ll say there is none

The Decision Matrix In reality What we conclude Reject null Accept alternative We say. . . • • • There is a real program effect There is a difference, gain Our theory is correct Null true Alternative false In reality. . . • • • There is no real program effect There is no difference, gain Our theory is wrong

The Decision Matrix In reality What we conclude Reject null Accept alternative We say. . . • • • There is a real program effect There is a difference, gain Our theory is correct Null true Alternative false In reality. . . • • • There is no real program effect There is no difference, gain Our theory is wrong TYPE I ERROR The odds of saying there is an effect or gain when in fact there is none # of times out of 100 when there is no effect, we’ll say there is one

The Decision Matrix In reality What we conclude Null false Alternative true In reality. . . • • • There is a real program effect There is a difference, gain Our theory is correct

The Decision Matrix In reality What we conclude Accept null Reject alternative We say. . . • • • There is no real program effect There is no difference, gain Our theory is wrong Null false Alternative true • • • In reality. . . There is a real program effect There is a difference, gain Our theory is correct

The Decision Matrix In reality What we conclude Accept null Reject alternative We say. . . • • • There is no real program effect There is no difference, gain Our theory is wrong Null false Alternative true • • • In reality. . . There is a real program effect There is a difference, gain Our theory is correct TYPE II ERROR The odds of saying there is no effect or gain when in fact there is one # of times out of 100 when there is an effect, we’ll say there is none

The Decision Matrix In reality What we conclude Reject null Accept alternative We say. . . • • • There is a real program effect There is a difference, gain Our theory is correct Null false Alternative true • • • In reality. . . There is a real program effect There is a difference, gain Our theory is correct

The Decision Matrix In reality What we conclude Reject null Accept alternative We say. . . • • • There is a real program effect There is a difference, gain Our theory is correct Null false Alternative true • • • In reality. . . There is a real program effect There is a difference, gain Our theory is correct 1 - POWER The odds of saying there is an effect or gain when in fact there is one # of times out of 100 when there is an effect, we’ll say there is one

The Decision Matrix In reality What we conclude Accept null Reject alternative We say. . . • • • There is no real program effect There is no difference, gain Our theory is wrong Reject null Accept alternative We say. . . • • • There is a real program effect There is a difference, gain Our theory is correct Null false Alternative true Null true Alternative false In reality. . . • • • There is no real program effect There is no difference, gain Our theory is wrong 1 - THE CONFIDENCE LEVEL • • • In reality. . . There is a real program effect There is a difference, gain Our theory is correct TYPE II ERROR The odds of saying there is no effect or gain when in fact there is none The odds of saying there is no effect or gain when in fact there is one # of times out of 100 when there is no effect, we’ll say there is none # of times out of 100 when there is an effect, we’ll say there is none 1 - TYPE I ERROR The odds of saying there is an effect or gain when in fact there is none POWER The odds of saying there is an effect or gain when in fact there is one # of times out of 100 when there is no effect, we’ll say there is one # of times out of 100 when there is an effect, we’ll say there is one

The Decision Matrix In reality What we conclude Accept null Reject alternative We say. . . • • • Accept alternative We say. . . • • • There is no real program effect There is no difference, gain Our theory is wrong • • • In reality. . . There is a real program effect There is a difference, gain Our theory is correct 1 - THE CONFIDENCE LEVEL TYPE II ERROR There is no real program effect There is no difference, gain Our theory is wrong Reject null • Null false Alternative true Null true Alternative false In reality. . . There is a real program effect There is a difference, gain Our theory is correct 1 - TYPE I ERROR POWER

The Decision Matrix In reality What we conclude Accept null Reject alternative We say. . . Null false Alternative true Null true Alternative false In reality. . . • • • There is no real program effect There is no difference, gain Our theory is wrong • • • In reality. . . There is a real program effect There is a difference, gain Our theory is correct 1 - THE CONFIDENCE LEVEL TYPE II ERROR CORRECT • • • There is no real program effect There is no difference, gain Our theory is wrong Reject null Accept alternative We say. . . • • • There is a real program effect There is a difference, gain Our theory is correct 1 - TYPE I ERROR POWER CORRECT

The Decision Matrix In reality What we conclude Accept null Reject alternative We say. . . • • • Accept alternative We say. . . • • • There is no real program effect There is no difference, gain Our theory is wrong 1 - THE CONFIDENCE LEVEL • • • In reality. . . There is a real program effect There is a difference, gain Our theory is correct TYPE II ERROR The odds of saying there is no effect or gain when in fact there is none The odds of saying there is no effect or gain when in fact there is one 1 - TYPE I ERROR The odds of saying there is an effect or gain when in fact there is none POWER The odds of saying there is an effect or gain when in fact there is one If you try to increase power, you increase the chance of winding up in the bottom # of times out of 100 when is noof effect, we’ll say. I error. there is an effect, we’ll say rowthere and Type there is none There is no real program effect There is no difference, gain Our theory is wrong Reject null • Null false Alternative true Null true Alternative false In reality. . . There is a real program effect There is a difference, gain Our theory is correct # of times out of 100 when there is no effect, we’ll say there is one # of times out of 100 when there is an effect, we’ll say there is one

The Decision Matrix In reality If you try to • There is no real program effect What is no difference, gain decrease Type I • • There Our theory is wrong we conclude errors, you Accept null 1 - increase the Reject alternative THE CONFIDENCE LEVEL We sayof. . . winding chance The odds of saying there is no effect or gain when in up • in. There the top is no real row fact there is none program effect and ofis Type II • There no difference, # of times out of 100 when gain there is no effect, we’ll say • Ourerror. theory is wrong there is none Reject null Accept alternative We say. . . • • • There is a real program effect There is a difference, gain Our theory is correct Null false Alternative true Null true Alternative false In reality. . . • • • In reality. . . There is a real program effect There is a difference, gain Our theory is correct TYPE II ERROR The odds of saying there is no effect or gain when in fact there is one # of times out of 100 when there is an effect, we’ll say there is none 1 - TYPE I ERROR The odds of saying there is an effect or gain when in fact there is none POWER The odds of saying there is an effect or gain when in fact there is one # of times out of 100 when there is no effect, we’ll say there is one # of times out of 100 when there is an effect, we’ll say there is one