Should FDA EMEA Health Canada CONSORT accept LastObservation

Should FDA, EMEA, Health Canada, CONSORT accept Last-Observation. Carried-Forward Analyses? A Systematic Review of Dementia Drug RCTs. Dr. Frank Molnar Associate Professor of Medicine, University of Ottawa, Canada Staff, The Ottawa Hospital Division of Geriatric Medicine Medical Director, The Ottawa Hospital Geriatric Day Hospital (focus on Dementia) Co-chair, Champlain Dementia Network ( www. champlaindementianetwork. org ) Medical Director, Regional Geriatric Program of Eastern Ontario ( www. rgpeo. com )

Conflict of Interest Disclosure Frank Molnar MSc, MDCM, FRCPC Has no real or apparent conflicts of interest to report.

Background – the basics of group-togroup comparisons in RCTs n n n The effectiveness of medications is tested via RCTs that follow response to new treatments and comparators over time. It is almost inevitable that some participants will drop out of such studies before trial completion. if analyses only include participants who remain in the trial, then study power is lost, benefit of randomization is lost and trials risk generating erroneous conclusions. Consequently, intention-to-treat (ITT) - the inclusion of all patients in the analysis according to the group determined at randomization - has become the accepted standard for the analysis of controlled clinical trials. The strength of ITT is that it not only preserves power but it also promotes balance between treatment groups for both known and unknown confounders, thereby preserving the benefits of randomization.

Background - dropouts n n In order for ITT approaches to analyze all patients randomized, including dropouts, several methods to estimate missing data have been developed Due to its simplicity (and possibly due to tradition and/or lack of motivation / impetus to change), a commonly employed technique to impute missing data is Last-Observation-Carried-Forward (LOCF), also known as ‘endpoint analysis’

Background - LOCF n Last-Observation-Carried-Forward replaces subjects’ missing outcomes with the last available measurement and requires that two basic conditions be met: n n that the subjects’ responses would have been constant (i. e. stable) from the last observed value (i. e. point of dropout) to the endpoint of the trial rather than declining or improving further that missing values are ‘missing completely at random’ or MCAR (i. e. that the probability of dropout is not related in any way to variables such as disease severity, symptoms, group assignment or drug side-effects).

The Concern in Dementia RCTs l LOCF ignores trajectory - whether the subject was improving or getting worse at the time of dropout. l LOCF freezes outcome values at the last observation, thereby creating an apparent stabilization of disease, symptoms and/or function in dropouts (problematic in progressive conditions)

Figure 1: Potential impact of last-observation-carried-forward analysis in longitudinal randomized controlled trials in chronic progressive diseases Molnar, F. J. et al. CMAJ 2008; 179: 751 -753 Copyright © 2008 Canadian Medical Association or its licensors

Differential LOCF bias if have greater or earlier dropouts in treatment group than control group (Effect measured by LOCF [c – d] > True effect [a – b] resulting in exaggerated positive effect - biased in favor of treatment) a = True treatment values b = True control values c = Treatment values altered by LOCF d = Control values altered by LOCF 100 Score on outcome variable (cognitive or functional scale) c } a LOCF bias b d 0 1 2 3 4 5 6 7 Time (visit #) 8 9 } LOCF bias 10

Differential LOCF bias if have greater or earlier dropouts in control group than treatment group (Effect measured by LOCF [c – d] < True effect [a – b] resulting in underestimate of effectiveness - biased against treatment ) a = True treatment values b = True control values c = Treatment values altered by LOCF d = Control values altered by LOCF Score on outcome variable (cognitive or functional scale) 100 c a } LOCF bias } d b 0 1 2 3 4 5 6 7 Time (visit #) 8 9 10 LOCF bias

Systematic Review Methods; Identification of Studies p p p An electronic literature search of Medline and the Cochrane Register of Controlled Trials from January 1984 (the year of publication of the Mc. Khann criteria for Alzheimer disease) to February 2008 This electronic search was supplemented by hand searching of the reference lists of selected articles, meta-analyses and review articles. The full text of RCT reports were then independently reviewed by two certified specialists in Geriatric Medicine with clinical expertise in dementia, with formal research methodological training.

Methods; Eligibility Criteria p Double-blinded, randomized controlled trials of cholinesterase inhibitors (donepezil, Aricept, rivastigmine, Exelon, galantamine, Reminyl) and N-methyl-D-aspartate (NMDA) receptor antagonists (Namenda, Ebixa, memantine) examining progressive symptoms (e. g. cognition, function) in Alzheimer disease, Vascular dementia, Mixed dementia or Mild Cognitive Impairment employing DSM or NINCDS-ADRDA criteria for Alzheimer disease or NINDS-AIREN criteria for Vascular Dementia were eligible for this review. p The systematic review was restricted to studies with full trial reports published in English language peer-reviewed journals.

Results; Selection of articles o Of the 1146 articles identified by the search strategies, 191 papers (including RCT reports, non-randomized trial reports, commentaries, systematic reviews and meta-analyses) were selected for full text and reference section review. Of these, 57 RCT reports met the eligibility criteria for systematic review

Results; Trial Characteristics of 57 studies o 45 Alzheimer disease o o o 8 vascular dementia or mixed dementia o o 21 donepezil 11 rivastigmine [1 donepezil-rivastigmine] 7 galantamine, 6 memantine), 3 donepezil 3 galantamine 2 memantine 4 MCI o o o 2 donepezil 1 rivastigmine 1 galantamine).

Results; Reporting of Dropouts o Cholinesterase inhibitor studies were more likely to demonstrate greater dropout in the treatment group (73% of studies) compared to memantine studies (25% of studies). n When cholinesterase inhibitor studies were combined, there was a higher dropout rate in the treatment group (23. 3%) than in the control group (16. 8%). o n LOCF would bias in favor of treatment When memantine trials were combined the opposite pattern was noted with fewer dropouts in the treatment group (14. 6%) than in the control group (18. 5%). o LOCF would bias against treatment

Results; Types of non-ITT Analyses o 35 employed Observed Case analysis (OCA) n o Fully Evaluable Population analysis (FEP) n o Described as completing study with a preset level of compliance [% compliance with meds, number of visits attended] – the compliance criteria may vary from study to study. Treatment Per Protocol analysis (TPP) n o Randomized patients with at least one evaluation while on study medication at designated assessment times – analyze data at that point but do not carry forward Similar to FEP as meet a preset level of compliance [% compliance with meds, number of visits attended] that varies from study to study but do not necessarily have to complete study Completer analysis (CA) n Analyze all patients that complete trial regardless of compliance

Results; Types of ITT Analyses o o o 12 (21%) of the 57 studies did not identify the type of ITT analysis performed or performed only non-ITT analysis. Of the remaining 45 studies that did perform an identifiable form of ITT analysis, 42 (93%) employed LOCF. 35 of the trials that performed ITT analysis (78 %) relied on LOCF as the only form of ITT analysis.

Results; Types of ITT Analyses o Of the 45 studies that employed LOCF, only 8 reported performing another type of ITT analysis to confirm the results of LOCF analyses. n n 3 of these 8 studies did not comment on the results of the alternate non- LOCF ITT analysis. Of the 5 studies that commented on the results of the alternate ITT analysis, 4 did not report the values of the outcomes calculated by the alternate ITT analysis but did indicate that the direction of results was unchanged. n It is uncertain if the values of the point estimates of the outcomes were similar when the alternate ITT analyses were performed.

Results; Types of ITT Analyses o Only 1 study verified the values of the point estimates of outcomes measuring drug efficacy generated by LOCF with point estimates of an alternate form of ITT analysis. n The study verified the values of 3 positive outcomes (out of hundreds published)

Results; Contraindications to the use of LOCF o Of the 35 studies employing LOCF as the only form of ITT analysis, 23 (66 %) explicitly demonstrated contraindications (factors that could introduce bias) to the use of LOCF that would exaggerate effectiveness. . n Greater dropout rate in treatment group n Earlier dropouts in treatment group More rapid progressors in treatment dropout group n o 66% is likely an underestimate n It was unclear if the remaining 12 studies were free of contraindications as most studies failed to report adequate data regarding the timing and severity of disease of dropouts to permit a precise estimate of the number of contraindications to the use of Last-Observation-Carried-Forward.

Reasonable Concerns n It is highly unlikely that, given the high prevalence of conditions promoting ‘LOCF bias’ uncovered in this study, the point estimates of some of the hundreds of outcomes generated in the reviewed trials that used LOCF have not been altered in some way. n The question is likely not ‘has bias been introduced’, but rather How many outcomes have been biased ? n To what degree have they been biased ? n

Why keep promoting LOCF? n LOCF provides no benefits over other forms of ITT analysis while introducing unnecessary risk of bias in progressive disease states. n Unless we have an interest in exaggerating the effectiveness of the drugs, why should we continue to use a technique such as LOCF if it provides no benefit and unnecessarily introduces bias? n LOCF is unnecessary – other ITT approaches that carry less risk of bias are readily available and have been used n time response relationship for ADAScog/11 analyzed using generalized linear modeling n mixed effects modeling n mixed models repeated measures (MMRM) n Sensitivity analyses of ITT analyses should be done to explore if one form is introducing bias. n Petersen RC, Thomas RG, Grundman M et al. Vitamin E and Donepezil for the Treatment of Mild Cognitive Impairment. NEJM 2005; 352: 2379 – 2388.

Barrier to optimal clinical care n The use of LOCF creates unnecessary risk of generating biased results that will tend to favor more toxic therapies (e. g. cholinesterase inhibitors) over less toxic therapies (e. g. memantine). n n Drug Toxicity →increase in measured efficacy When using LOCF, toxicity becomes a positive attribute of a drug. n This effect may create a barrier to the funding and use of current (memantine) and future less toxic therapies.

FINAL CONCLUSION: Analysis remains the weak link in proving efficacy / effectiveness in dementia drug RCTs (an unnecessary vulnerability that will prevent increased use of these medications).