Reporting differences and effect size Lecture 10 Becoming

Reporting differences and effect size Lecture 10 Becoming a Researcher: Using Data Dr Suzanna Forwood Suzanna. Forwood@Anglia. ac. uk 10/30/2021 Becoming a Researcher: Using Data 1

Learning Outcomes • FAQ for Research Report • Reminder – probability and significance • Able to say how confident we are that a significant difference is reliable • Confidence interval • Able to interpret the importance of a significant result, beyond what a p-value tells us • Effect size 10/30/2021 Becoming a Researcher: Using Data 2

Research Report FAQ 10/30/2021 Becoming a Researcher: Using Data 3

Assessment 011 Research Report (60%) Submission: 1800 words. Turnitin in 11 MAY, 2 pm Friday Marking – focus on results 10% introduction 80% results 10% discussion 10/30/2021 Becoming a Researcher: Using Data 4

011 Research Report (60%) • Students are asked to produce a research report addressing research questions of their choice using the dataset collected in practical classes. • The assessment requires students to demonstrate skills of formulating 2 -3 research questions that the report will address. These should be around a single theme, and should require the inclusion of at least one t-test and one correlation. 10/30/2021 Becoming a Researcher: Using Data 5

011 Research Report (60%) For each research question, students should separately be: 1. describing the appropriate of research design, and selecting a statistical test. 2. summarising the relevant data using descriptive numbers, tables and plots 3. conducting and reporting the appropriate statistical test. 4. reaching a conclusion with respect to the research question. 10/30/2021 Becoming a Researcher: Using Data 6

Resources • Getting Started • Data & SPSS • References • Writing up – including examples • Ask for help https: //canvas. anglia. ac. uk/courses/14110/modules/126807 10/30/2021 Becoming a Researcher: Using Data 7

011 Research Report (60%) 10/30/2021 Becoming a Researcher: Using Data 8

Questions answered • Practical Discussion – any time this week • Live Webinar with me - 10. 30 am Wednesday/Thursday/Friday 10/30/2021 Becoming a Researcher: Using Data 9

Probability and significance 10/30/2021 Becoming a Researcher: Using Data 10

Null Hypothesis Significance Testing • H 0 Null Hypothesis The data we are seeing are entirely explained by error variability / chance. 10/30/2021 • H 1 Alternative Hypothesis The data we are seeing reflect an important difference that is more than error variability / chance. 11

Null Hypothesis Significance Testing • H 0 Null Hypothesis The data we are seeing are entirely explained by error variability / chance • H 1 Alternative Hypothesis The data we are seeing reflect an important difference that is more than error variability / chance. H 0: Examples: H 1: Examples: • This score is no different from the class • The two groups are no different • The drug had no impact on performance 10/30/2021 • This score is larger than the rest of the class • The two groups are different • The drug improved performance 12

p values • Statistical tests used for deciding whether a difference is significant are based on probability. • They all involve calculating a statistic (e. g. z-score, t-score, Pearson's coefficient, Chi-squared), and then the probability associated with that statistic. 10/30/2021 Becoming a Researcher: Using Data 13

Null Hypothesis Sign Testing • Null Hypothesis The data we are seeing are entirely explained by error variability / chance • Alternative Hypothesis The data we are seeing reflect an important difference that is more than error variability / chance. H 0: The DEFAULT position H 1: The ALTERNATIVE position The test statistic that we calculate from our data tells us what is the probability that our data fit with the null hypothesis. 10/30/2021 14

p values • The p value tells you how likely it is that the results we obtained are due to error factors or chance • If this probability is very low (p < 0. 05), then we can reject the null hypothesis • This means we can infer support for the experimental hypothesis • It is more likely that the effects are due to some other factor, such as the independent variable 10/30/2021 Becoming a Researcher: Using Data 15

Null Hypothesis Significance Testing • H 0 Null Hypothesis The data we are seeing are entirely explained by error variability / chance H 0: The DEFAULT position The test statistic tells us the probability that our data fit with the null hypothesis. When p < 0. 05 the probability is LOW, this suggests that the data is UNLIKELY to be the result of error variability. 10/30/2021 • H 1 Alternative Hypothesis The data we are seeing reflect an important difference that is more than error variability / chance. H 1: The ALTERNATIVE position When the probability is LOW, we reject H 0 and accept H 1. Our finding is significant 16

Null Hypothesis Significance Testing • H 0 Null Hypothesis The data we are seeing are entirely explained by error variability / chance • H 1 Alternative Hypothesis The data we are seeing reflect an important difference that is more than error variability / chance. H 1: The ALTERNATIVE position H 0: The DEFAULT position The test statistic tells us the probability that our data fit with the null hypothesis. When p>0. 05 the probability is HIGH, this suggests that the data is LIKELY to be the result of error variability. 10/30/2021 Both hypotheses therefore fit with the data. Good practice in science is to opt for the simplest explanation, so we accept H 0. Our finding is NOT significant 17

Reporting p values If you are working it out by hand: • Use a critical value to decide significance • Report p<0. 05 or p>0. 05. If you are using SPSS: • It will give the exact value of p. You can therefore be more precise. • Report exact p value, so p=0. 004 or p=0. 674. • If SPSS says p= 0. 000. This looks like p=0, but that is not true or possible, so we report p<0. 001. 10/30/2021 Becoming a Researcher: Using Data 18

But • P values are not the only option. • We can get a bit obsessed with searching for statistically significant results. • Disappointment on finding p >0. 05 – our results aren’t significant • Excitement on finding p < 0. 05 - WOOHOO • Other options are in practical terms more useful. I will tell you about two. [You don’t HAVE TO use these in your report, but they are useful to know about, and consider using them when you can]. 10/30/2021 Becoming a Researcher: Using Data 19

10/30/2021 Becoming a Researcher: Using Data 20

Reporting differences with confidence intervals 10/30/2021 Becoming a Researcher: Using Data 21

Example – Stroop task • In the Stroop task, participants are asked to name the colour of the ink that each stimulus word is written in. • The stimulus words are all colour names. • There are two kinds of trial: • MATCH: the ink colour and the stimulus word are the same, e. g. RED • MISMATCH: the ink colour and the stimulus word are different, e. g. BLUE 10/30/2021 Becoming a Researcher: Using Data 22

Example – Stroop task • We record the number of correct answers to name the ink colour. • Results • Difference: 44. 78 - 28. 01 = 16. 77 10/30/2021 Becoming a Researcher: Using Data 23

Example – Stroop task • 10/30/2021 Becoming a Researcher: Using Data 24

Confidence Intervals • When we collect data from samples of participants, we are not certain they are a true representation of the population data • We might expect to find values close to the ones we observed, but not exactly the same This is because of Sampling error • We want to know how close our observed sample means are likely to be to the true population values 10/30/2021 Becoming a Researcher: Using Data 25

Confidence Interval • There will always be sampling error • A Confidence Interval (CI) gives us a range of values that the true population difference is likely to lie within. • A 95% Confidence Interval (CI) gives us a range of values that we can be 95% certain that the true population difference lies within. 10/30/2021 Becoming a Researcher: Using Data 26

Confidence Interval • When we carry out a t-test, we compare two means • Example: Stroop experiment • Difference: 10/30/2021 44. 78 - 28. 01 = 16. 77 Becoming a Researcher: Using Data 27

Confidence Interval • SPSS provides us with 95% confidence intervals Difference between conditions: 44. 78 - 28. 01 = 16. 77 t(168)=11. 08, p<0. 001 10/30/2021 Reporting confidence intervals: 95% CI [13. 78, 19. 75] Becoming a Researcher: Using Data 28

Confidence Interval • This means that, considering the difference between condition means we observed from our sample (16. 77), and based on the sample size and spread of our data, we can be 95% confident that the true difference (if we tested the entire population) would lie somewhere between 13. 78 and 19. 75 10/30/2021 Becoming a Researcher: Using Data 29

Confidence Interval • Why is 95% CI useful? • It provides us with the same result as t-score and p, • If the CI range contains zero, e. g. 95% CI [-1. 4, 2. 1], Then there is no significant difference • It provides us with more information than the p value. The CI range can help us see how big an effect there is likely to be in practical terms • If the CI is significant but small e. g. a training course improves end-of-year exam scores by 1. 12% [0. 29, 1. 71] Then although this is a statistically significant result, it is of limited practical value (at most you benefit by 1. 71 percent – which isn’t that great). 10/30/2021 Becoming a Researcher: Using Data 30

10/30/2021 Becoming a Researcher: Using Data 31

10/30/2021 Becoming a Researcher: Using Data 32

Effect Sizes 10/30/2021 Becoming a Researcher: Using Data 33

Effect size • 10/30/2021 Becoming a Researcher: Using Data 34

Effect size • 10/30/2021 Becoming a Researcher: Using Data 35

Effect size • Sometimes, with very large N’s, we find a significant result that represents a very small effect size • Represents a valid and meaningful difference between groups, in that the observed difference is greater than the difference expected due to error • But in reality, the effect of that difference is small • Interpretation danger with large N’s 10/30/2021 Becoming a Researcher: Using Data 36

Effect size • Example: Imagine we measure IQ for 100, 000 children. We find a significant effect of gender on IQ, with girls on average scoring 1 IQ points higher than boys. 10/30/2021 Becoming a Researcher: Using Data 37

Effect size • When testing 100, 000 people, the difference seen is very close to population difference. • It is very unlikely that a 1 point difference is due to chance, so p value is very small. 10/30/2021 Becoming a Researcher: Using Data 38

Effect size A small p value does not necessarily mean a big effect 1. P-value reflects the likelihood that your result was due to chance 2. When you’ve tested 100, 000 participants: • Your observed difference is likely close to the population difference • Your observed difference is unlikely to be due to noise 3. It is statistically significant, but not practically significant 10/30/2021 Becoming a Researcher: Using Data 39

Effect size • It would be useful to have something to report when we’re reporting p values and CI’s that tells us the size of the effect we’re dealing with. • Cohen’s d • • • Small effect size: d = 0. 2 Medium effect size: d = 0. 5 Large effect size: d = 0. 8 Imaginary IQ example: d = 0. 07 Our Stroop experiment: d = 1. 70 10/30/2021 Becoming a Researcher: Using Data 40

10/30/2021 Becoming a Researcher: Using Data 41

10/30/2021 Becoming a Researcher: Using Data 42