Alpha Recycling in Confirmatory Clinical Trials Unifies Simplifies

Alpha Recycling in Confirmatory Clinical Trials Unifies, Simplifies, Extends many common MTPs Olivier Guilbaud Senior Principal Scientist Astra. Zeneca R&D, Södertälje, Sweden 1 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY

Outline • Background • Splitting, Recycling, Adding, (parts of alpha) • Graphs and Default Graphs for MTPs • Rejection Algorithm • Holm’s MTP for groups of Hs, and a problem to discuss • Improvements through extensions of the Default Graph • Summary, and further results: • (Simultaneous Confidence regions, CTP-formulation of MTPs) 2 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY

Background: Regulatory environment Citation from EMEA/CPMP’s (2002) Points to Consider on Multiplicity Issues in Clinical Trials : “ … multiplicity can have a substantial influence on the rate of false positive conclusions …” ”As a general rule it can be stated that control of the family-wise type-I error rate in the strong sense … is a minimal prerequisite for confirmatory claims” 3 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY

Background: Multiple Confirmatory Comparisons • Several Efficacy Variables • Several Tolerability Variables • Several Comparators • Several Doses • Several kinds of Administration (e. g. once daily, twice daily, …) • Delta-Noninferiority / Superiority / Delta-Superiority • Several Subgroups/Kinds of Subjects (e. g. Placebo, Active 1, Active 2, …) Confirmatory: Pre-specified family of multiple comparisons (i. e. statistical tests). Risk of getting any ”false positive” result/conclusion is (no matter how many or which H 0 s are true – Strong Control) 4 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY

Background: Abbreviations/Terminology • MTP : Multiple Testing Procedure • MCP : Multiple Comparison Procedure • FWER : Family-Wise Error Rate (type-I errors) = Pr[at least 1 true Hi in the Family is rejected by the MTP] (to be controlled to be in the strong sense) , no matter how many, or which, His in the family are true In contrast to the weak sense (old) : , if all His in the family are true 5 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY

Background: brief refresher about three basic MTPs Three basic MTPs for Family {H 1, H 2, H 3} based on raw p-values p 1, p 2, p 3 • Fixed-Sequence MTP (old) p 1 , p 2 , p 3 • Fallback MTP (2003/2005) weights w 1, w 2, w 3 p 1 w 1 , p 2 (w 2+) , p 3 (w 3+) • Holm MTP (1979) p(1) /3, p(2) /2, p(3) /1 Now, to the Alpha-Recycling framework 6 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY

Two articles in Statistics in Medicine, February 2009 Only this now 7 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY

Idea: Splitting and Recycling test-mass (= part of ) Throughout: Raw p-value available for each null hypothesis H Bonferroni: split of Fixed-Sequence: Combination of recycling of split & recycling of 8 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY

Idea: Adding test-mass from different paths Parallel-gatekeeping MTP: 2 graphs for same MTP Default graph Important for rejection Algorithm Its sequences reflect the possible rejection paths 9 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY

Idea: Adding test-mass from different paths Fallback MTP for 3 Hs: 2 graphs for same MTP Default graph Important for rejection Algorithm Its sequences reflect the possible rejection paths 10 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY

Idea: Adding test-mass from different paths Holm’s MTP for 3 Hs: 2 graphs for same MTP Default graph Important for rejection Algorithm Its sequences reflect the possible rejection paths 11 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY

Rejection Algorithm - based on Default-Graph Seqs Step 1. For each ”first” H in Sequences, (a) add-upp test mass from ”its” Sequences (b) test H at this added-up level If nothing is rejected, then Stop; otherwise: Reduce Sequences by deleting all rejected H from Sequences, and go to Step 2 with remaining reduced Sequences Step r = 2, 3, …. For each ”first” H in remaining reduced Sequences, (a) add-upp test mass from ”its” Sequences (b) test H at this added-up level If nothing is rejected, then Stop; otherwise: Reduce Sequences by deleting all rejected H from Sequences, and go to Step r + 1 with remaining reduced Sequences This MTP controls (in the strong sense) the FWER to be ! 12 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY

Comments • number of sequences can be increased (no restriction) • length of sequences can be increased (no restriction) • if not all sequences contain all Hs, then improvements are possible (by increasing length of relevant sequences) • interpretability of MTP is important (also when improvements are considered) 13 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY

A refresher – Bonferroni & Holm for groups of Hs Bonferroni for 2 groups of 2 Hs, with Fixed-Seq testing within group Holm for 2 groups of 2 Hs, with Fixed-Seq testing within group: Step 1: Bonferroni-version as above Step 2: If one group is entirely rejected, but not the other, then try again in the other at increased level 14 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY /2 1 2 3 4 /2 1 12 3 4

Problem to discuss with your neighbour Bonferroni for 2 groups of 2 Hs, with Fixed-Seq testing within group (is already in default-graph form): /2 1 2 3 4 Extend sequences of default graph to get /2 recycling improvement equivalent to 1 Holm for 2 groups of 2 Hs, with Fixed-Seq testing within group 3 15 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY /2 2 4

Recent Example discussed by FDA statisticians • Confirmatory family with 4 null hypotheses : H 1 w H 2 (1 -w) H 3 H 4 16 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY

Recent Example discussed by FDA statisticians Alpha-recycling applied to show Non-inf & Superiority for Primary & Secondary Variables Basic version w H 2 Reformulation in Recycling terms w (1 -w) H 1 H 1 H 2 H 3 (1 -w) H 3 H 4 Very easy to test with rejection algorithm ! But why not recycle also after the last H in each sequence ? 17 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY

Recent Example discussed by FDA statisticians Alpha-recycling applied to show Non-inf & Superiority for Primary & Secondary Variables - Simple improvement Improvement w H 2 Reformulation in Recycling terms w (1 -w) H 1 H 1 H 3 H 2 H 3 H 4 H 4 H 2 (1 -w) May be questioned by some (not me)! Very easy to test with rejection algorithm ! This improved MTP is ” -exhaustive” ! 18 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY May be questioned by some (not me)!

Sequences can be extended/added for more rejections Improved Fallback MTP for 3 Hs: 2 graphs for same MTP Default graph This improved MTP is ” -exhaustive” ! 19 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY

Sequences can be extended/added for more rejections Improved Parallel-gatekeeping : 2 graphs for same MTP Default graph This improved MTP is ” -exhaustive” ! 20 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY

XXX example: 2 primary & 2 second vars, 2 age stata Reformulation in Recycling terms Default graph Original proposal 1 2 4 3 5 7 8 /8 /8 1 1 4 4 2 3 5 6 7 7 7 7 8 8 8 8 6 9 11 9 9 11 11 10 12 10 10 12 12 to show strong control of FWER Simple improvement possible ( Holm instead of Bonf after H 8 ): Add sub-seq (11, 12) to Default-graph sequences ending with (9, 10) Add sub-seq (9, 10) to Default-graph sequences ending with (11, 12) 21 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY

Compact notation for MTP in XXX example Reformulation in Recycling terms Default graph Original proposal 1 2 4 3 5 7 8 /8 /8 1 1 4 4 2 3 5 6 7 7 7 7 8 8 8 8 6 9 11 9 9 11 11 10 12 10 10 12 12 Compact notation for this recycling MTP (B here is Bonf-operator): B(H 1 B(H 2, H 3), H 4 B(H 5, H 6)) H 7 H 8 B(H 9 H 10, H 11 H 12) Can be expanded (using certain rules) to 8 sequences of Hs above 22 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY

YYY example: 3 doses vs. 0, 1 primary & 1 second var Basic version Reformulation in Recycling terms Default graph /2 /4 /4 1 2 1 1 2 2 4 3 5 5 4 6 6 3 5 6 Idea: To give lowest dose a chance if at least 1 of the 2 larger doses works for the primary variable 23 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY

YYY example – Improvement 1 Reformulation in Recycling terms Default graph Improvement 1 /2 /4 /4 1 2 1 1 2 2 4 3 5 5 4 5 6 6 5 3 5 6 6 Increases level for H 5 and H 6 if H 3 and/or H 4 are rejected 24 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY 6

YYY example – Improvement 2 Reformulation in Recycling terms Default graph Improvement 2 1/2 1 2 3 5 /4 /8 /8 /4 1 1 1 2 2 2 3 5 5 4 5 6 6 5 6 2 2 1 1 6 2 3 4 4 3 1 4 4 3 3 4 6 Gives H 1 or H 2 a (Holm-type) second opportunity to be rejected if not rejected initially 25 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY

Summary, and further results • Easy to formulate (cf. Closed-Testing Pocedures) • Easy to perform (even manually) • Unifies many common MTPs • Easy to construct new MTPs (for new problems) • Easy to calculate multiplicity-adjusted p-values (algorithm similar to that for rejections) • Compact algebraic notation available (to describe and derive default graphs) • Easy to obtain weights of Bonferroni test for any intersection hypothesis HI of corresponding CTP (useful for contruction of Conf regions) 26 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY

Corresponding Simultaneous Confidence Regions ? ? ? Three basic MTPs for Family {H 1, H 2, H 3} based on raw p-values p 1, p 2, p 3 • Fixed-Sequence MTP (old) p 1 , p 2 , p 3 • Fallback MTP (2003/2005) weights w 1, w 2, w 3 p 1 w 1 , p 2 (w 2+) , p 3 (w 3+) • Holm MTP (1979) Open until p(1) /3, p(2) /2, p(3) /1 2007 27 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY Solution in JASA 1999

Corresponding Simultaneous Confidence Regions !!! Amazing: Simultanously presented at MCP 2007 in Vienna 28 years after Holm (1979) Extensions and Relations Confidence regions require weights that can be obtained directly from rejection algorithm of recycling approach 28 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY

Outline • Background • Splitting, Recycling, Adding, (parts of alpha) • Graphs and Default Graphs for MTPs • Rejection Algorithm • Holm’s MTP for groups of Hs, and a problem to discuss • Improvements through extensions of the Default Graph • Summary, and further results: • (Simultaneous Confidence regions, CTP-formulation of MTPs) 34 O Guilbaud, FMS+Cramér Society, AZ-Södertälje, 2010 -11 -29 Proprietary and Confidential © Astra. Zeneca 2009 FOR INTERNAL USE ONLY