Uses and Abuses of Adaptive Randomization An Industry

Uses and Abuses of (Adaptive) Randomization: An Industry Perspective Benjamin Lyons, Ph. D. and Akiko Okamoto, Sc. D. FDA/Industry Workshop September, 19, 2003 Johnson & Johnson Pharmaceutical Research and Development L. L. C. 1

Outline • • Adaptive vs. Static Randomization Implementation Challenges – Errors by Investigators – Errors in Algorithm – Errors related to Drug Supply • Conclusion FDA/Industry Workshop September, 19, 2003 Johnson & Johnson Pharmaceutical Research and Development L. L. C. 2

Adaptive vs. Static Randomization • Static randomization requires that one randomization list is generated at the start of the trial. • Adaptive (Dynamic) randomization algorithms (e. g. , Urn model) assign treatments based on patient characteristics and previous treatment assignments. FDA/Industry Workshop September, 19, 2003 Johnson & Johnson Pharmaceutical Research and Development L. L. C. 3

Covariate Adaptive Randomization • Treatment assignment of the (n+1)st patient may depend upon the previous first n patients. • Usual mechanism is a balance function that is minimized by assigning the new patient to a certain treatment. FDA/Industry Workshop September, 19, 2003 Johnson & Johnson Pharmaceutical Research and Development L. L. C. 4

Why use adaptive randomization? • Treatment balance “required” within each level of stratification factors. • For small trials with many stratification factors static-stratified randomization will not insure balance within each strata or overall. FDA/Industry Workshop September, 19, 2003 Johnson & Johnson Pharmaceutical Research and Development L. L. C. 5

Why avoid adaptive randomization • May be hard to interpret using standard theory (see recent CPMP guidelines on adjustments for baseline covariates). • Many chances to make errors. • Implications of some errors on inference are not easy to understand in the context of standard theory. • Some errors may put trial validity at risk. FDA/Industry Workshop September, 19, 2003 Johnson & Johnson Pharmaceutical Research and Development L. L. C. 6

Implementation Challenges Three types of errors: – Errors by investigators; – Errors in the algorithm; – Errors caused by a faulty drug supply method. FDA/Industry Workshop September, 19, 2003 Johnson & Johnson Pharmaceutical Research and Development L. L. C. 7

Example 1: Site Error • Site enters the wrong strata level for a patient. • Site assigns the wrong medication kit and perhaps treatment to patient. FDA/Industry Workshop September, 19, 2003 Johnson & Johnson Pharmaceutical Research and Development L. L. C. 8

Response • Do we update the balance function by altering the assignment weights to reflect error? • If corrected there are three categories of balance functions: – randomized before the error; – randomized after the error but before the correction; – randomized after the correction. • If not corrected there are only two. FDA/Industry Workshop September, 19, 2003 Johnson & Johnson Pharmaceutical Research and Development L. L. C. 9

Analysis • How do you report this? • Are the pre-specified test statistics asymptotically valid? • For stratification error is there a sensitivity analysis? • How should you incorporate into a permutation or or resampling procedure? FDA/Industry Workshop September, 19, 2003 Johnson & Johnson Pharmaceutical Research and Development L. L. C. 10

Prevention • Site training. • Train sponsor staff on how to react to the error. • Giving IVRS vendor staff explicit instructions on who decides to update the algorithm. • Is it sound to alter the algorithm for a few minor errors? FDA/Industry Workshop September, 19, 2003 Johnson & Johnson Pharmaceutical Research and Development L. L. C. 11

Example 2: Algorithm Error • Specification is correct for 1: 1 assignment as indicated by simulation in SAS. • Actual code to calculate assignment written in an SQL program. • Validation of SQL program did not include any simulation. FDA/Industry Workshop September, 19, 2003 Johnson & Johnson Pharmaceutical Research and Development L. L. C. 12

Result • Error in SQL program detected after 50% enrollment. Balance is 2: 1. • Program fixed so that the balance at the end of the trial is 1: 1. • Probability of treatment assignment correlated with date of trial entry. FDA/Industry Workshop September, 19, 2003 Johnson & Johnson Pharmaceutical Research and Development L. L. C. 13

Analysis • Is this trial randomized? • Are the standard test statistics asymptotically valid. • How should we account for the error in any permutation test? • Should the trial results be reported at all? • Could entry time be correlated with patient characteristics and hence outcome? FDA/Industry Workshop September, 19, 2003 Johnson & Johnson Pharmaceutical Research and Development L. L. C. 14

Prevention • Validate the actual software that produces the assignment through simulation prior to roll out. • Check balance results frequently during the trial. • Vendor must have a responsible/trained statistician who understands the issues. FDA/Industry Workshop September, 19, 2003 Johnson & Johnson Pharmaceutical Research and Development L. L. C. 15

Example 3: Drug Supply • Supply at sites is not adequate. – – Lack of study drug. Drug not re-supplied often enough. High enrollment in short periods. Uneven enrollment by site. • In some cases all treatment arms are not available when a subject is randomized. FDA/Industry Workshop September, 19, 2003 Johnson & Johnson Pharmaceutical Research and Development L. L. C. 16

Response • System provides “over rides” or “forced randomizations”: – the patient is assigned to available treatment regardless of what the algorithm says. • Adaptive algorithm is ignored for this patient. FDA/Industry Workshop September, 19, 2003 Johnson & Johnson Pharmaceutical Research and Development L. L. C. 17

Result • Trial should be balanced if only a few occurrences. • “Forced” assignment included in the balance function. • The algorithm has not been implemented as stated in the protocol and the report. • Are subsequent randomizations that used the faulty balance function valid? FDA/Industry Workshop September, 19, 2003 Johnson & Johnson Pharmaceutical Research and Development L. L. C. 18

Analysis • Are the standard test statistics asymptotically valid? • How does a permutation test account for the ‘over rides’? • How many “forced” assignments before the entire randomization is suspect? FDA/Industry Workshop September, 19, 2003 Johnson & Johnson Pharmaceutical Research and Development L. L. C. 19

Prevention • Supply trials with dynamic randomization centrally with one kit going to each site after each randomization. OR • Have abundant supply at all sites. OR • Do not allow forced randomization, turn patients away if all arms not available. FDA/Industry Workshop September, 19, 2003 Johnson & Johnson Pharmaceutical Research and Development L. L. C. 20

Simulation • 171 Subjects. • Two treatment Arms: A and B. • 4 Strata: Site (16) and three prognostic factors (2, 2, and 4 levels). • Randomization by Biased Coin. • Entry time , stratification and response based on CNS trial. • Assignment is altered in 10, 000 replications. FDA/Industry Workshop September, 19, 2003 Johnson & Johnson Pharmaceutical Research and Development L. L. C. 21

Supply Algorithm • Each site began with 4 kits: 2 A and 2 B. • Re-supplied in 1 “day” with two kits when one arm is empty. • Patients may enter with only 1 arm available. • If arm assigned by IVRS was missing then the remaining treatment was given. • Drug supply is part of the simulation. FDA/Industry Workshop September, 19, 2003 Johnson & Johnson Pharmaceutical Research and Development L. L. C. 22

Results • For 10000 “trial” simulations – – Average of 5 “forced randomization: per trial; T-statistic calculated for each “trial”; Distribution similar to theoretical. Supply error has no effect. FDA/Industry Workshop September, 19, 2003 Johnson & Johnson Pharmaceutical Research and Development L. L. C. 23

Conclusion • • • Adaptive Randomization is more difficult to execute then static randomization. There are several sources of error. Result of errors are poorly understood. Some errors may be “minor” errors. Using Adaptive randomization adds costs and risk to running a trial. FDA/Industry Workshop September, 19, 2003 Johnson & Johnson Pharmaceutical Research and Development L. L. C. 24