Estimation and Adjustment of Bias in Randomised Evidence
Estimation and Adjustment of Bias in Randomised Evidence Using Mixed Treatment Comparison Meta-analysis Sofia Dias, NJ Welton, AE Ades with Valeria Marinho, Georgia Salanti, Julian Higgins Avon RSS, May 2010 Department of Community Based Medicine
Overview • Motivation • Treatment networks and MTC • Adjusting for Bias in Mixed Treatment Comparisons Meta-analysis (MTC) • The MTC model • Example: Fluoride dataset • Probability of bias model • Results and Conclusions 2
Mixed Treatment Comparisons • Often more than two treatments for a given condition • Network of trials comparing different interventions for a condition • Direct and indirect evidence available on treatment effects • Because of the network structure, there is enough information to estimate and adjust for bias within the network • For bias adjustment, there is no need to rely on exchangeability assumption between meta-analyses in different fields 3
Example: The Fluoride Data • 6 different interventions for preventing dental caries in children and adolescents 1. 2. 3. 4. 5. 6. • No Treatment Placebo Fluoride in Toothpaste Fluoride in Rinse Fluoride in Gel Fluoride in Varnish Active Treatments From 6 Cochrane Reviews* 4 *Marinho et al. , 2002; 2003; 2004 (Cochrane Library)
Network and Number of trials 1 T Pl 3 1 69 1 4 6 13 3 NT • 130 trials 4 9 31 G 1 V 4 • eight 3 -arm trials • one 4 -arm trial • 150 pairwise comparisons R 5
Introduction to MTC 1. Six treatments 1, 2, 3, 4, 5, 6 2. Take treatment 1 (No Treatment) as reference 3. Then the treatment effects d 1 k of all other treatments relative to 1 are the basic parameters 4. Given them priors: d 1, 2, d 1, 3, …, d 1, 6~ N(0, 1002) 6
Functional parameters in MTC • The remaining contrasts are functional parameters d 2, 3 = d 1, 3 – d 1, 2 d 2, 4 = d 1, 4 – d 1, 2 CONSISTENCY assumptions … 1 2 3 d 4, 6 = d 1, 6 – d 1, 4 d 5, 6 = d 1, 6 – d 1, 5 • Any information on functional parameters tells us indirectly about basic parameters • Either FE or RE model satisfying these conditions 7
Notation • Data i = 1, …, 130 study index k = 1, 2, 3, …, 6 treatment index rik – number of caries occurring in trial i, treatment k, during the trial follow-up period Eik – exposure time in arm k of trial i (in person years)
Fluoride: Poisson MTC RE model i = 1, …, 130 Exposure time in person years rate at which events occur in arm k of trial i Priors MTC consistency equations 9
MTC results: LHR relative to No Treatment Residual deviance is 278. 6 (270 data points) 10
Posterior mean of residual deviances for each point 11
Check how evidence is combined in the network • Poor fit can indicate inconsistency in the network • For each pair, separate direct evidence from indirect evidence implied by the rest of the network* • Can see how evidence is combined in the network to give overall MTC estimate • Helpful to locate pairs of comparisons where there may be problems *Dias et al. , Stats in Med. 2010 12
LHR for Placebo v Toothpaste Direct MTC Bayesian p-value = 0. 32 Indirect 13
LHR for Placebo v Varnish Bayesian p-value = 0. 04 MTC Direct Indirect 14
LHR for Rinse v Varnish Bayesian p-value = 0. 02 MTC Indirect Direct 15
But we have additional information on the risk of bias of all included studies BIAS MODELS 16
Treatments NT P T R G V Total Allocation concealment No of studies Blinding adequate unclear inadequate Double Single ? 1 0 0 1 0 4 1 3 0 3 1 0 3 0 1 0 2 1 0 1 0 0 3 0 2 1 9 0 5 4 0 6 3 4 0 3 1 61 8 46 7 61 0 0 25 2 20 3 22 0 3 9 0 6 3 9 0 0 3 0 3 0 0 1 0 1 0 4 0 3 1 2 2 0 1 0 0 130 11 98 21 103 17 10 17
MTC RE model with bias Priors MTC consistency equations 18
MTC Bias Model • Assume non-zero mean bias, bi = b ≠ 0, in comparisons of NT or Pl with Active treatments • For Active-Active comparisons assume mean bias is zero • Expect bias to increase size of treatment effect: b<0 19
Fluoride: Risk of Bias indicators • Allocation concealment • • Best empirical evidence of bias But… 98/130 studies ‘unclear’ Only 11/130 studies ‘adequate’ Some comparisons have no adequately concealed trials • Blinding also available to inform risk of bias status • Used “Any bias” as a composite indicator of bias: 54/130 studies at risk of bias. 20
Probability of Bias Model • Any study with unclear allocation concealment has a probability p of being at risk of bias • Adequately concealed trials are not at risk of bias • Inadequately concealed trials are at risk of bias • Use only allocation concealment as bias indicator • Bias terms identifiable in this rich network 21
Probability of Bias Model 22
Comparing Model Fit Res. Dev* p. D MTC with no bias adjustment DIC Between trial heterogeneity 278. 6 259. 3 537. 9 0. 22 (0. 19, 0. 26) Any. Bias 277. 6 257. 9 535. 5 0. 15 (0. 12, 0. 18) Probability of bias 274. 6 253. 0 527. 6 0. 12 (0. 10, 0. 15) Bias adjustment * Compare with 270 data points 23
Posterior mean of residual deviances for each point: MTC and Prob of bias models Study 42: Placebo v Toothpaste (1 of 69 trials) Allocation concealment unclear Study 63: No Treat v Varnish (1 of 4 trials) Allocation concealment unclear and not “double blind” Study 102: Placebo v Varnish (1 of 3 trials) Allocation concealment unclear 24
Treatment effects relative to No Treatment (LHR) Unadjusted MTC (solid) and Probability of Bias model (dashed) 25
Varnish effects • Cochrane Review to assess efficacy of Fluoride Varnish (Marinho et al, 2004) • Noted that the small number and poor methodological quality of varnish trials might be overestimating the true effect of this intervention. • The results of the bias-adjusted analysis support this hypothesis. 26
Which treatment is best? Unadjusted MTC Bias-adjusted MTC Probability Best (%) Rank No Treatment 0 6 Placebo 0 5 Toothpaste 3. 6 2. 9 9. 3 2. 7 Rinse 4. 1 2. 8 53. 8 1. 6 Gel 3. 7 3. 2 12. 4 2. 9 Varnish 88. 5 1. 2 24. 6 2. 8 27
Results: Probability of Bias • posterior mean = -0. 19, Cr. I (-0. 36, -0. 02) • posterior sd = 0. 40, Cr. I (0. 29, 0. 55) • Trials with unclear allocation concealment are at risk of bias with probability p • Posterior mean of p = 0. 13 • Model identified 5 trials (with unclear allocation concealment) as having a high probability of bias 28
Prob of bias for studies with unclear allocation concealment o – unclear allocation concealment + – unclear allocation concealment and single blind ∆ – unclear allocation concealment and unclear blinding status 29
Other findings • Between trial heterogeneity in treatment effects reduced in bias-adjusted model • Model with Active-Active bias was also fitted with similar results: Active-Active bias had posterior mean of zero • But assumptions on direction of bias… • Assumed bias would favour the newest treatment (also the most intensive) 30
Conclusions • Bias estimation and adjustment possible within MTC because there is a degree of redundancy in the network • Assumption that study specific biases are exchangeable within the network • Uses only internal evidence • Weaker than required from using external evidence • Ideas extend to multiple bias indicators • But will need a very rich evidence structure 31
Consequences for Decision Modelling • Uses only internal evidence • May be more acceptable to patient groups, pharmaceutical industry… • Risk of bias indicator chosen based on empirical research • Results may change if different bias indicators chosen Again: • Assessment of model fit & sensitivity analysis crucial if decisions based on these models are to have credence 32
References • Our website: http: //bristol. ac. uk/cobm/research/mpes • Dias S, Welton NJ, Marinho VCC, Salanti G, Higgins JPT and Ades AE (2010) Estimation and adjustment of Bias in randomised evidence using Mixed Treatment Comparison Meta-analysis. Journal of the Royal Statistical Society A, to appear Vol 173 issue 4 (available online). • Dias S, Welton NJ, Caldwell DM and Ades AE (2010) Checking consistency in mixed treatment comparison meta-analysis. Statistics in Medicine, 29, 945 -955. • Schulz KF, Chalmers I, Hayes RJ and Altman DG (1995) Empirical Evidence of Bias. Dimensions of Methodological Quality Associated With Estimates of Treatment Effects in Controlled Trials. JAMA, 273, 408 -412. 33
- Slides: 33