NULL HYPOTHESIS TESTING 5 CONTENTS Logic of Hypothesis

NULL HYPOTHESIS TESTING 5

CONTENTS Logic of Hypothesis Testing Details of NHST

Contents Uncertainty Reckoning with Uncertainty Hypothesis Testing LOGIC OF HYPOTHESIS TESTING Null Hypothesis

UNCERTAINTY Sampl e Population Sampl e The view from the Population: one fixed population will give rise to many different samples

UNCERTAINTY Population Sample Population The view from the Sample: one fixed sample could have come from many different populations

QUANTIFY UNCERTAINTY • The view from the population ‣ sampling distribution: • spread of possible samples ‣ standard error: • sd of sampling distribution • The view from the sample ‣ likelihood function: • spread of possible populations ‣ standard error: • also sd of likelihood function

RECKONING WITH UNCERTAINTY • There are 2 basic ways to reckon with the uncertainty • Estimate the range of likely populations ‣ confidence interval • Estimate the probability this sample came from a specific population ‣ hypothesis testing

HYPOTHESIS TESTING • Given: ‣ a hypothesis – a predicted effect size ‣ a sample – a sample effect size • How do we compare these two

HYPOTHESIS TESTING • How do we compare hypothesis and sample? • Is it reasonable that the given hypothesis will produce our sample? ‣ if not: we can reject the hypothesis ‣ if so: we cannot do anything • because: ‣ other untested hypotheses may also produce our sample

LOGIC OF HYPOTHESIS TESTING • Calculate how consistent hypothesis and sample are • if consistency is low: ‣ reject either hypothesis or sample ‣ sample is real, therefore reject hypothesis • otherwise: ‣ nothing can be done ‣ do not, ever, ever “accept” a hypothesis

LOGIC OF HYPOTHESIS TESTING • Which hypothesis? ‣ bearing in mind that all we can do is reject a hypothesis • our actual hypothesis ‣ we can try to reject our actual hypothesis • the opposite of our actual hypothesis ‣ we can try to reject the opposite of our actual hypothesis

NULL HYPOTHESIS TESTING • Is our sample effect size not consistent with the opposite of our actual hypothesis? • Opposite-hypothesis: ‣ there is no effect of IV on DV ‣ population effect size is zero • This is called the null hypothesis

NULL HYPOTHESIS TESTING Sample Effect Size 1 0. 5 Sampling Distribution This is the distribution of sample effect sizes produced by the null hypothesis 0 -0. 5 -1 frequency Null hypothesis: “There is no effect. ” This hypothesis is that the population effect size = 0

NULL HYPOTHESIS TESTING Samples from Null Hypothesis (sims=103570) Sample Effect Size 1 0. 5 97. 5% Sampling Distribution We mark out the central 95% of this. 0 2. 5% -0. 5 -1 frequency Null hypothesis: “There is no effect. ” This hypothesis is that the population effect size = 0 If the null hypothesis was true, then the sample effect size would be inside this range 95% of the time. And outside it just 5%.

NULL HYPOTHESIS TESTING Samples from Null Hypothesis (sims=103570) Sample Effect Size 1 0. 5 97. 5% Sampling Distribution If our sample is outside the 95% range, then we will say: 0 2. 5% -0. 5 -1 frequency Null hypothesis: “There is no effect. ” This hypothesis is that the population effect size = 0 the sample is not consistent with the null hypothesis.

NULL HYPOTHESIS TESTING Samples from Null Hypothesis (sims=103570) Sample Effect Size 1 0. 5 97. 5% If sample effect size is here, it could be consistent with the not-hypothesis. 0 2. 5% -0. 5 -1 frequency Null hypothesis: “There is no effect. ” This hypothesis is that the population effect size = 0 If it is anywhere else, it is not consistent with the not-hypothesis

LOGIC OF NULL HYPOTHESIS TESTING Hypothesis Null Hypothesis Sample is consistent We are Happy We are Sad Sample is not consistent We are Sad We are Happy • This table is how we might wish to view the result

LOGIC OF NULL HYPOTHESIS TESTING Hypothesis Null Hypothesis Sample is consistent We are Happy We are Sad Sample is not consistent We are Sad We are Happy • But we mustn’t ‣ recall that we can only reach a conclusion from an inconsistency ‣ so we cannot use the top row

LOGIC OF NULL HYPOTHESIS TESTING Hypothesis Null Hypothesis Sample is consistent We are Happy We are Sad Sample is not consistent We are Sad We are Happy • There’s another problem: ‣ we have only tested the null hypothesis ‣ so we cannot use the left column

LOGIC OF NULL HYPOTHESIS TESTING Consistent with sample Hypothesis Null Hypothesis We are Happy We are Sad Not consistent with sample We are Sad We are Happy This is what we look for This is called Null Hypothesis Testing

NULL HYPOTHESIS TESTING OUTCOMES • For any null hypothesis test, there are just two outcomes: ‣ we reject the null hypothesis the result is statistically significant ‣ we fail to reject the null hypothesis the result is not statistically significant

Contents Null Hypothesis Testing Uncertainty as a Probability The Logic of Inference DETAILS OF NULL HYPOTHESIS TESTING Type I errors Type II errors Statistical Inference

NULL HYPOTHESIS TESTING • Null hypothesis testing is used to determine the statistical significance of any relationship between variables ‣ does the uncertainty about the sample effect allow a conclusion about the population? • It does not tell you the size of any of these effects – an effect size is a different type of calculation

NULL HYPOTHESIS TESTING • The null hypothesis is the prediction that no effect exists ‣ Written down as H 0 • The hypothesis which states our prediction is called the alternative hypothesis

UNCERTAINTY AS A PROBABILITY • If any given hypothesis is specific enough, we can estimate : ‣ probability that hypothesis will produce any specified range of effect sizes • For mathematical convenience, we estimate this probability: ‣ probability that null hypothesis produces a sample with effect size at least as large as our sample • if that probability is low ‣ then the sample is judged to be inconsistent with the hypothesis

NULL HYPOTHESIS TESTING • Null hypothesis testing assumes that the null hypothesis is correct. ‣ and the sample effect size obtained is due to chance sampling from a population with zero effect size • Find the probability that the null hypothesis population would produce ‣ your effect size or more extreme • If that probability is low ‣ the null hypothesis and your sample cannot both be correct ‣ reject the null hypothesis

THE LOGIC • We can calculate p-value: ‣ the probability of getting our sample or more extreme given that the null hypothesis is true • If p-value is very small, then ‣ either our sample is wrong ‣ or the null hypothesis is wrong • We know that the sample isn’t wrong ‣ therefore the null hypothesis must be wrong

THE LOGIC • It is widely agreed that ‣ if a given outcome would occur by chance less than 5% of the time ‣ it is reasonable to say it didn’t occur by chance • So, when p-value is less than 0. 05 ‣ we reject the null hypothesis • the arbitrary value of 0. 05 is called alpha

INFERENCE Null Hypothesis p>0. 05 Fail to reject the null hypothesis Result is not statistically significant p<0. 05 Reject the Null Hypothesis Result is statistically significant Alternative Hypothesis p>0. 05 No conclusion can be reached uncertainty still exists p<0. 05 No definite conclusion should be reached uncertainty still exists

TYPE I ERROR • When a null hypothesis is rejected, there is a 5% likelihood that is was true ‣ any significant result may be in this 5% ‣ it is impossible to know whether any specific result is • Rejecting a null hypothesis when it should not is a Type 1 error ‘False positive’

TYPE II ERROR • When we fail to reject a null hypothesis but should have rejected it, this is a Type II error ‣ any non-significant result may be a Type II error ‣ it is impossible to know whether any specific result is ‘False negative’

STATISTICAL INFERENCE • Recall that this process is inference • Therefore it can lead to the wrong conclusion: Null Hypothesis FALSE Null Hypothesis TRUE p>0. 05 Type II error Correct p<0. 05 Correct Type I error • Before applying a null hypothesis test, any of these 4 outcomes is possible: ‣ the future decision could hold Type I error ‣ the future decision could hold Type II error

STATISTICAL INFERENCE • Reaching a conclusion after a null hypothesis test places the result in one of the two rows Null Hypothesis FALSE Null Hypothesis TRUE p>0. 05 Type II error Correct p<0. 05 Correct Type I error • After applying a null hypothesis test, only two outcomes are possible: ‣ if you found p<0. 05, that past decision could hold Type I error ‣ if you found p>0. 05, that past decision could hold Type II error

STATISTICAL INFERENCE Null Hypothesis FALSE Null Hypothesis TRUE p>0. 05 Type II error Correct p<0. 05 Correct Type I error • Before testing: ‣ all four outcomes are possible ‣ there is no way of knowing whether you will make an error • After testing: ‣ only one row applies ‣ there is no way of knowing whether you have made an error

OVERVIEW • Samples are uncertain guides to populations ‣ amount of uncertainty is known ‣ uncertainty limits how much we can claim to know • Null hypothesis testing ‣ converts amount of uncertainty into a p-value ‣ uses p-value to reach binary decision • Uncertainty still exists ‣ it is now the unknown possibility of a Type I or II error

THE BENEFITS OF NULL-HYPOTHESIS TESTING This slide is intentionally blank

THE PROBLEMS OF NULL-HYPOTHESIS TESTING • p-values are often misunderstood ‣ smaller p-value does not mean stronger outcome • alternative hypothesis is not tested ‣ no conclusion can be reached about the alternative hypothesis • if reject null hypothesis ‣ alternative hypothesis is not “proved” • if failure to reject null hypothesis ‣ no conclusion can be reached about either null or alternative hypothesis