Modeling Study Results As noted before we may

Modeling Study Results As noted before, we may decide that studies do not share a common effect, that is, we decide that Ti = θi + ei. However, we may still not wish to adopt the simple random-effects model, particularly if we have ideas about why results from different studies may differ. In such cases we may wish to examine some predictive models for our parameters, and thus our study results. 1

Predictive or Explanatory Models We will consider the regression model θi = 0 + 1 X 1 i +. . . + p Xpi We’ll also examine an ANOVA-like model, where the subscript j shows that study i is from the jth group or category of studies θji = θj for study i in group j Both analyses will be weighted using w or another weight. 2

Mixed Effects Models If each population effect differs, and Ti = i + ei we may want to model or explain the variation in the population effects. For example we may have i = 0 + 1 X 1 i + u’i Substituting, we have Ti = 0 + 1 X 1 i + u’i + ei Observed Effect predicted + Unexplained between- + Sampling effect from X value studies variation error 3

Going back to the model for Ti, we see that for the regression model Ti = 0 + 1 X 1 i +. . . + p Xpi + ei and for the ANOVA model Tji = θj + eji For both models we will have tests of significance and tests and estimates of slopes, means, etc. Also we’ll have tests of model specification that tell us whether we have explained all between studies differences. 4

For instance, for the expectancy studies, we can analyze weeks before the expectancy induction occurred, both as a categorical factor, and as a regression predictor: Ti = 0 + 1 weeksi +ei. For the ANOVA-like model we can group studies by weeks: 0 weeks exposure 1 week exposure 2 weeks exposure 3 or more weeks of exposure 5

A key difference between what we do for metaanalysis and standard OLS regression and ANOVA is that meta-analysis data typically violate the assumption of homoscedasticity, because study sizes differ and Vi values are not equal. For example, in the meta-analysis regression Ti = 0 + 1 X 1 i +. . . + p Xpi + ei we know that v(ei) can be computed and is a function of ni. If the ni’s vary, we won’t have homoskedasticity. 6

Therefore, weighted analogues to the typical analyses are needed. Our models show the only source of variation to be sampling error (the ei terms). We begin by using the weights used before: wi = 1/vi. These are our fixed effects weights, and shortly we will have a third set of weights as well. 7

Technically, we use weighted least squares (WLS) regression and weighted ANOVAs. We do not use random-effects weights because we are trying to explain the between-studies differences that the RE model treats as “uncertainty. ” Using RE weights gives a very conservative analysis. Later we will learn about mixed models. We use these if our ANOVA or regression analyses do not explain all of the between-studies differences in our data. 8

Weighted Regression for Meta-analysis (AKA meta-regression) The estimation of regression slopes is done via weighted least squares (WLS) regression. We have a vector of outcomes T = (T 1, T 2, . . . Tk)’ and a matrix of p predictors, called X Denote the matrix of variances of the outcomes T 1, T 2, . . . Tk as Σ = diag[V 1, . . . , Vk ]. 9

Then the WLS gives us ^ = (X’ Σ-1 X)-1 X’ Σ-1 T the vector of slopes, and its variance is ^ V( ) = (X’ Σ-1 X)-1 Under H 0: β = 0, the p slope estimates are normally distributed, so the standard output is not correct. 10

The printed standard computer output from pulldown menus shows slopes that are correct BUT SEs and t tests that are INCORRECT!! If we use pullodwn menus we must correct the SEs via SE = Printed SE/ MSE ^ Also the tests can be computed as z = /SE( ) or as z = (Printed test ) * MSE But why do that when we can use SPSS macros to get the right results! 11

Also we have two tests that tell us about our models: QModel + QResidual = Q 2(p) 2(k-p-1) 2(k-1) In pulldown output we can look at SSModel + SSResidual = SSTotal We then ask a series of questions. 12

First we ask. . . 1. Is QModel significant? If yes, the predictors explain some (or all) of the population variation If not, we might look for another predictor or try another model We’ll look at the expectancy data with “weeks” as our predictor. 13

The scatterplot shows a somewhat nonlinear relationship with “weeks” as the predictor. Note that the “best fit” line that SPSS plots will not be correct unless all studies are equal in size. This occurs because we need to use a weighted regression and the plotted line is unweighted. 14

We can make a plot that shows the weight each study gets under the fixed model. The points vary in size. Large circles show the larger studies that have more weight. 15

The fitted equation is obtained from pull-downs using the regression menu with “w” clicked into the box for the “WLS weight”. The model is di = 0. 158 - 0. 013 weeksi + ei SSRegression = QReg = 8. 16 is significant with p = 1 df. 16

• The macro from Dave Wilson called Meta. Reg. sps gives this analysis with proper CIs. Do not use the R 2 -- it is too low. ***** Inverse Variance Weighted Regression ***** Fixed Effects Model via OLS ***** ------- Descriptives ------Mean ES R-Square k. 0603. 2278 19. 0000 ------- Homogeneity Analysis ------Q df p Model 8. 1609 1. 0000. 0043 Residual 27. 6645 17. 0000. 0490 Total 35. 8254 18. 0000. 0074 ------- Regression Coefficients ------B SE -95% CI +95% CI Z P Beta Constant. 1584. 0501. 0602. 2566 3. 1620. 0016. 0000 weeks -. 0132. 0046 -. 0223 -. 0041 -2. 8567. 0043 -. 4773 17

Here’s the regression line from the weighted analysis: di = 0. 158 - 0. 013 weeksi + ei Clearly the points with the large effects at weeks = 0 are not well explained by this line. 18

Next we check the specification of the model by asking 2. Is QResidual significant? If QResidual is small, then the predictors explain all between-studies differences If QResidual is big, then unexplained differences remain and the predictors do not give a full explanation. We may want to explore other predictors or run a mixed model. 19

• The Meta. Reg output shows that the Q for residual is just significant. Using weeks as a continuous predictor does not quite explain the variation in effects. Just significant. ------- Homogeneity Analysis ------Q df Model 8. 1609 1. 0000 Residual 27. 6645 17. 0000 Total 35. 8254 18. 0000 ------- Regression Coefficients ------B SE -95% CI +95% CI Constant. 1584. 0501. 0602. 2566 weeks -. 0132. 0046 -. 0223 -. 0041 p. 0043. 0490. 0074 Z 3. 1620 -2. 8567 With one X, the ps for the model and the slope are identical. P Beta. 0016. 0000. 0043 -. 4773 20

Mixed Regression Model If we have not explained all variation it means there is excess uncertainty to account for. That is, our model is Ti = 0 + 1 X 1 i + ei + uꞋi . so we need to estimate the variance of the uꞋi. We do this by using a mixed model. Let’s see what Meta. Reg tells us about how much uncertainty remains, and how it affects the model. 21

Meta. Reg shows that the Q for residual is no longer significant. However, this is not really a legitimate test, so we do not report it. ------- Homogeneity Analysis ------Q df Model 7. 9277 1. 0000 Residual 24. 2264 17. 0000 Total 32. 1541 18. 0000 p. 0049. 1134. 0211 We are more interested in the Model test. Weeks is still significant even with added uncertainty accounted for! 22

Meta. Reg shows the residual variance as . 0048, well below either of the RE variances. The model is Ti = 0. 18 -. 0146 + ei + uꞋi vs. the (wrong) FE model Ti = 0. 16 -. 0132 + ei ------- Regression Coefficients ------B SE -95% CI +95% CI Z Beta Constant. 1773. 0567. 0662. 2884 3. 13 weeks -. 0146. 0052 -. 0247 -. 0044 -2. 82 P. 0018. 0049 . 0000 -. 4965 --- Maximum Likelihood Random Effects Variance Component ---v =. 00485 se(v) =. 00888 23

ANOVA-Like Categorical Model Consider studies that fall into p a priori groups. Then we have Tji the outcome for study i in group or class j, for i = 1 to kj studies in j = 1 to p groups with k = Σ kj Denote _ Tj The estimated average effect in group j θj The true population effect in group j 24

We ask the question: Do all θj values agree? Our null hypothesis is like that for the betweengroups test in ANOVA: H 0: θ 1 = θ 2 =. . . = θp In words H 0 says: All p sets of studies show the same population effects, or, The mean effects in the p groups are equal. 25

We also ask the question: Do all θji values in the jth group agree? This null hypothesis is like that for the overall Q test, but it is done within each of the p groups. For example, for group 1 we have H 01: θ 11 = θ 12 =. . . = θ 1 k 1 = θ 1 In words H 01 says “All studies in group 1 show the same population effect. ” 26

So for each of the groups (j=1 to p) we test this H 0 j: θj 1 = θj 2 =. . . = θj kj = θj For each group we get a Q value within the group (we’ll call it Qwithin j). Then we can get an overall test of within-group homogeneity across the p groups. We get it as Qwithin = Qwithin j 27

We will partition our total Q into two chi-square tests Q = QBetween + QWithin 2(k-1) 2(p-1) Total homogeneity test Tests differences between the p θj values 2(k-p) Tests for remaining variation in results In standard SPSS pulldown output we’d examine SSTotal = SSBetween + SSWithin 28

So we have parallel quantities Q = QBetween + QWithin SSTotal = SSBetween + SSWithin Q = SSTotal = All between-studies variation + sampling error QBetween = SSBetween = Between-groups variation + sampling error QWithin = SSWithin = Unexplained between-studies differences + sampling error 29

30

_ Tj is computed similarly, but within group j _ Tj = wji Tji / wji for wji = 1/Vji i i We can also get variances and SEs for these means (more on this below). QWj is a weighted variance of the effects within group j and QWithin is the sum of those variances p kj _ _ QWithin = wji (Tji - Tj )2 = QWj ~ 2(k-p) j i Each QWj is a chi-square test with df = kj - 1 31

We ask a series of questions about these tests: 1. Is QBetween significant? If yes, the predictor explains some (or all) of the population variation If not, we decide that the grouping variable that was tested does not explain between-studies differences. We might look for another predictor for ANOVA or try another kind of model. This factor, though, is not useful if Qbetween is not significant. 32

Example: Teacher expectancy effects Here we’ll look at exposure in terms of number of weeks the teachers had known the students (weekcat in our data). The means appear to be lower with more weeks of exposure. Let us examine the hypothesis of no differences. 33

• We can use a macro called Meta. F. sps developed by Dave Wilson. The output from his macro looks like this: ***** Inverse Variance Weighted Oneway ANOVA Fixed Effects Model via OLS ***** ------- Analog ANOVA table (Homogeneity Q) ------Q df p Between 20. 3788 3. 0000. 0001 Within 15. 4466 15. 0000. 4198 Total 35. 8254 18. 0000. 0074 The correct p values for the chi-square tests are given. Differences exist between groups. 34

Next (if QBetween is significant) we ask 2. Is QWithin significant and are any QWj values significant? If the QWj’s are small and QWithin is small, then the predictor explains all between-studies differences. If a few QWj’s are big and QWithin is small, then the predictor explains most between-studies differences -- outliers may be a possibility or we may need a mixed model. 35

The output showed significant between-groups differences and, considering all groups, we do not see heterogeneity within groups (p =. 42 which is quite large). ***** Inverse Variance Weighted Oneway ANOVA Fixed Effects Model via OLS ***** ------- Analog ANOVA table (Homogeneity Q) ------Q df p Between 20. 3788 3. 0000. 0001 Within 15. 4466 15. 0000. 4198 Total 35. 8254 18. 0000. 0074 The ANOVA model appears to fit perfectly! But let’s check each group. 36

Meta. F. sps produces this table: ------- Q by Group ------Group Qw df. 0000 6. 2140 4. 0000 1. 0000 4. 9825 2. 0000. 4045 2. 0000 3. 8456 7. 0000 p. 1837. 0828. 8169. 7974 Each line gives a Q for a subset of studies (here for weekcat groups). All p values are above. 05, suggesting each subset is homogeneous. If they were not, we would use a mixed model. 37

• The Meta. F. sps macro produces a table of means as well. The overall mean does not differ from 0. ------- Effect Size Results Total ------Mean ES SE -95%CI +95%CI Z Total. 0603. 0365 -. 0112. 1319 1. 6539 P. 0981 k 19. 00 ------- Effect Size Results by Group ------Group Mean ES SE -95%CI +95%CI Z. 0000. 3620. 1099. 1466. 5774 3. 2939 1. 0000. 3516. 1121. 1319. 5712 3. 1367 2. 0000. 0665. 0727 -. 0760. 2091. 9147 3. 0000 -. 0630. 0500 -. 1611. 0350 -1. 2605 P. 0010. 0017. 3604. 2075 k 5. 00 3. 00 8. 00 • Two group means differ from zero (weekcat = 0 and 1) and two do not. 38

Contrasts for ANOVA-like Models If we reject H 0: θ 1 = θ 2 =. . . = θp (i. e. , QBetween is significant) or if we have some groups we would like to compare based on a priori hunches, we can use the values of the means (the Tj ) to compute contrasts: C = Σ cj Tj where Σ cj = 0. We compute the SE for C, and the test statistic is z = C/SE(C). 39

The contrasts parallel traditional ANOVA contrasts and can be used, e. g. , to compare the mean of the combined TE weekcat groups 0 and 1 to that of groups 2 and 3. The hypothesis tested is H 0: (θ 0 + θ 1 )/2 =(θ 2 + θ 3 )/2 The contrast is C = (T 0 + T 1 )/2 -(T 2 + T 3 )/2 = Σ cj Tj 40

If the contrast is C = Σ cj Tj , then SE(C) = Σ c 2 j Var (Tj ) Here SE(C) = Often it is easier to use integer weights so we don’t have fractions in the variance For the TE data, we could use integer weights cj of 1 and -1, then C = (T 0 + T 1 ) - (T 2 + T 3 ) 41

For our data for the contrast using fractional weights C = (T 0 + T 1 )/2 -(T 2 + T 3 )/2 = Σ cj Tj = (. 362 +. 352)/2 - (. 067 + -. 063)/2 = . 357 -. 002 =. 355 _____________ SE (C) = . 012/4 +. 013/4 +. 005/4 +. 002/4 = 0. 089 42

The test of H 0 is z = C/SE(C) =. 355/. 089 = 3. 97 so we reject H 0: (θ 0 + θ 1 )/2 =(θ 2 + θ 3 )/2 It seems that students exposed to teachers for a week or less show significantly larger mean expectancy effects than students with 2 or more weeks exposure. 43