Metaanalysis Types of reviews Narrative review vs metaanalysis

Meta-analysis

Types of reviews �Narrative review vs. meta-analysis vs. integrative data analysis �Univariate vs. bivariate vs. multivariate vs. SEM �When was the first meta-analysis? �When was the term first used? �What are the advantages of quantitative reviews? �What are problems with them?

Steps to meta-analysis

1. Define your variables/question � 1 df contrasts (or multivariate meta-analysis or SEMbased meta-analysis) �What is a contrast?

2. Decide on inclusion/exclusion criteria �What factors do you want to consider here?

3. Collect studies systematically �Where do you find studies? �File drawer problem �Gray lit

4. Code your studies �What should you code? �Inter-rater reliability �If there is more than 1 effect per study, what do you do? �What does the sign mean on an effect size? �What are small, medium, and large effects? �How can you convert from one to another? �r or d? �http: //www. soph. uab. edu/Statgenetics/People/MBeas ley/Courses/Effect. Size. Conversion. pdf

Families of effect sizes—d family � 2 group comparisons (difference between the means) �Cohen’s d (with various subscripts) �Hedge’s g �Glass’s d or delta �Point biserial r �Within vs. between-participants designs �https: //www. frontiersin. org/articles/10. 3389/fpsyg. 2013. 00863/full Lakens, 2013 (Table 1) �Probability of superiority or common language ES

Families of effect sizes—R family �Continuous or multi-group (proportion of variability) �η 2 �ηp 2 �ηG 2 �ω2 and its parts �r, fisher’s z, R 2, adjusted R 2 �difference between η 2 and R 2 family �https: //www. frontiersin. org/articles/10. 3389/fpsyg. 2013. 00863/full Lakens, 2013 (Table 2)

Calculate for PO effect

Other effect sizes �Nonparametric effect sizes �Nonnormal data: convert z to r or d �Categorical data: � Rho � Cramer’s V � Goodman-Kruskal’s Lambda �How can you increase your effect sizes?

CIs �How can you calculate confidence intervals around your effect sizes? �http: //daniellakens. blogspot. com/2014/06/calculatingconfidence-intervals-for. html �https: //thenewstatistics. com/itns/esci/ �http: //www. cem. org/effect-size-calculator �https: //www. aggieerin. com/shiny-server/ �What type (e. g. , %) CI should you report?

Interpretation of effect sizes �Recommended for at least most important findings �Benchmarks? �SD units �Practical or clinical significance and compare to lit �PS or common language effect size �U �Binomial effect size display �Relative risk �Odds ratio �Risk difference

5. Combine effect sizes �When should you do fixed vs. random effects? �Should you weight effect sizes, and if so, on what? �How can you deal with dependent effect sizes? �Hunter and Schmidt method vs. Hedges et al. method

6. Calculate confidence intervals �Credibility intervals vs. confidence intervals

7. Check for/correct for biases �m-a effect = True effect + effect of pub and exp bias �Outliers �Correct for unreliability

Publication bias assessments �Rosenthal’s fail-safe N �# studies needed at p <. 05= (K/2. 706) (K(mean Z squared) = 2. 706) �Z = Z for that level of p � K = number of studies in meta-analysis �Funnel plot (Egger’s test) �Rank correlation test for pub bias �Correlation between n and ES �Cumulative meta-analysis �Sensitivity analyses

Fig. 3. Funnel plots of 11 (subsets of) meta-analyses from 2011 and Greenwald, Poehlman, Uhlman, and Banaij (2009). Marjan Bakker et al. Perspectives on Psychological Science 2012; 7: 543 -554 Copyright © by Association for Psychological Science

Publication bias corrections �Trim and fill �Sensitivity analysis �WAPP-WLS �PET-PEESE (Figure 1; van Elk et al. , 2015) � Critiques of PET-PEESE, http: //datacolada. org/59 �p-uniform � 3 PSM �Cumulative meta-analysis �Bayesian approaches (e. g. , BALM)

p-curves �p-curve analysis (Figure 1; Simmons & Simonsohn, 2017) � Critiques of p-curves �www. p-curve. com �Which do Carter et al. recommend? �What effects did QRPs have on meta-analytic estimates?

8. Look at heterogeneity of effect sizes �Chi-square test �I 2 (measure based on Chi-square) �Cochran’s Q or Hedges Q �Standard deviations of effect sizes �Stem and leaf plot �Box plot �Forest plot

Forest plot

9. Look for moderators �What are common moderators you might test? �How do you compare moderators?

“little ‘m’ meta-analysis” �Comparing and combining effect sizes on a smaller level—when might you want to do this? �How would you do it? �Average within-cell r’s with fisher z transforms �To compare independent r’s: Z = z 1 -z 2/sqrt ((1/n-3) + (1/n-3)) �To combine independent r’s: z = z 1+z 2/2

Write-up �Inclusion criteria, search, what effect size �Which m-a tech and why �Graphs �Stem and leaf plots of effect sizes (and maybe mods) �Forest plots �Stats on variability of effect sizes, estimate of pop effect size and confidence/credibility intervals �Moderators �Publication bias analyses

Significance and effect sizes �What is the problem with just using p-levels to determine whether one variable has an effect on another? �Be careful with comparisons--sample results: �For boys, r (87) =. 31, p =. 03 �For girls, r (98) =. 24, p =. 14 �How does sample size affect effect size? Significance? �Why are effect sizes important? �What is the difference between statistical, practical, and clinical significance?

What should you report? � 2 group comparison—treatment vs. control on anxiety symptoms � 3 group comparison—positive prime vs. negative prime vs. no prime on number of problems solved � 2 continuous variables—relationship between neuroticism and goal directedness � 3 continuous variables—anxiety as a function of selfesteem and authoritarian parenting � 2 categorical variables—relationship between answers to 2 multiple choice questions

Coming up �Next class: �Replication readings (don’t get in the weeds of replicated studies) �Meta-analysis assignment (updated 3/23)

The next week: Presentations for proposal �Formal presentations—dress nice, stand up �No more than 12 minutes �I’ll take notes and send to you �Go through your FINAL plan for your study— background, method, expected results, and discussion �Graded on: �Presentation style �Slide quality �Ability to answer questions �Clarity and completeness