Application of the Win Ratio for BenefitRisk Analysis

Application of the Win Ratio for Benefit-Risk Analysis Paulette Ceesay, Shahrul Mt-Isa, and Joe Heyse Merck & Co. Joint Statistical Meetings, Vancouver August 1, 2018

Overview q Background q Definition of the win ratio (WR) method and the adaptation for benefit risk analysis (BRA) q Illustration of the features of WR for BRA using simulated data q Application to a Phase III trial for migraine headache q Comparison of WR and Desirability of Outcome Ranking (DOOR) methods q Concluding Remarks 2

Background q Historically, efficacy and safety analyses are performed separately when assessing the value of a new drug or dose. q Recently there is an increasing interest to assess the comparative benefit-risk (BR) of candidate drugs and vaccines. q Evans and Follmann (2016) recently advocated for a unified composite of benefit and risk as a pragmatic endpoint in a clinical trial setting, using the desirability of outcome ranking (DOOR, Evans et al. , 2015). 3

Background q The win ratio concept, developed to address composite endpoints, (Pocock et al. 2012) was adapted for a potential application in BR analysis. q We applied this approach retrospectively to evaluate the BR of a Phase III compound for treating migraine headache. 4

Definition of Win Ratio q Win ratio (Pocock et al. , 2012) considers the clinical importance order of multiple outcomes in a clinical trial q Compares each patient in the Treatment group with every patient in the Control group to determine the winner/loser/tie within each pair across the multiple outcomes. q Each paired comparison starts with the most important outcome, and moves to lower priority outcomes only if higher priority outcomes result in a tie. 5

Definition of Win Ratio (cont. ) q The proportion favorable (PF) is the difference in the proportion of wins minus the proportion of losses between treatments q The win ratio (WR) is the ratio of the numbers of wins in the two treatment groups. q Inference is based on U statistics which use indicator variables for winners and losers 6

Estimation

Win Ratio for Benefit Risk Analysis q The objective of BRA is to compare treatments integrating both efficacy outcomes and adverse events q The importance of efficacy determined by severity of disease and magnitude of the treatment effect q Adverse events can be mild, moderate, severe, or serious 8

Win Ratio for Benefit Risk Analysis (cont. ) q WR method can be applied by ordering efficacy variables and AEs by relative importance q Rule is required to judge win/lose/tie comparing each pair of patients q Method is highly applicable to active comparator clinical trials 9

Very modest simulations 1000 simulated trials using randomization test (NR=1000 resamples) 2 groups m=n=100 per group 3 variables (1) AE 1 (2) EFF (3) AE 2 Scenario (1) SAE (2) Efficacy (3) Mod AE Proportio n Favorable Rejects Win Ratio Rejects 1 Control Treat. 0. 05 0. 4 0. 1 0. 049 2 Control Treat. 0. 05 0. 4 0. 6 0. 1 0. 789 0. 786 3 Control Treat. 0. 05 0. 10 0. 4 0. 6 0. 1 0. 500 0. 490 10 10

Case Study q Phase III clinical trial for treatment of migraine headache q Adults with migraine with or without aura (International Headache Society criteria) treated a moderate or severe attack with q placebo q Drug A q Drug B q Active comparator 11

Case Study (cont. ) q Comparison of B-R profile using Proportion Favorable (PF) and Win-Ratio (WR) ◦ ◦ X 1: X 2: X 3: X 4: SEVERE AE PAIN FREE AT 2 HR Moderate AE Mild AE 12

Proportion with Response Placebo Drug A Drug B Active Cntrl. 328 317 338 328 (1) SEVERE 0. 018 0. 016 0. 006 0. 030 (2) PAIN FREE 0. 088 0. 177 0. 263 0. 308 (3) MODERATE 0. 064 0. 057 0. 109 0. 110 (4) MILD 0. 161 0. 199 0. 201 0. 314 SAMPLE SIZE 13

B-R Analysis Based on Ordered Multivariate U Statistics (Randomization Test with 2000 Resamples) Comparison A vs. B PF (One-sided P) PF is B - A WR (One-sided P) Ratio is B/A Placebo vs. Low Dose 0. 068 (P = 0. 033) 1. 285 (P = 0. 035) Placebo vs. High Dose 0. 140 (P = 0. 001) 1. 604 (P = 0. 001) Placebo vs. Active 0. 086 (P = 0. 020) 1. 294 (P = 0. 024) Active vs. Low Dose -0. 018 (P = 0. 651) 0. 950 (P = 0. 649) Active vs. High Dose 0. 046 (P = 0. 140) 1. 136 (P = 0. 141) Values of PF > 0 indicate a favorable BR profile for treatment relative to the control. q Values of WR > 1 indicate a favorable BR profile for treatment relative to the control. q 14

Comparing Randomization Test and Large Sample Approximation Comparison A vs. B Placebo vs. Drug A Placebo vs. Drug B Placebo vs. Active vs. Drug A Active vs. Drug B Parameter PF = B – A WR = B/A Randomization P-value LS Approx. P-value PF = 0. 068 0. 033 0. 024 WR = 1. 285 0. 035 0. 116 PF = 0. 140 0. 001 <0. 0001 WR = 1. 604 0. 001 0. 006 PF = 0. 086 0. 020 WR = 1. 294 0. 024 0. 107 PF = -0. 018 0. 651 0. 692 WR = 0. 950 0. 649 0. 621 PF = 0. 046 0. 140 0. 109 WR = 1. 136 0. 141 0. 226 15

Desirability of Outcome Ranking (DOOR, Evans et al, 2015) q Developed for an application to comparing strategies for optimal antibiotic use q Response adjusted for duration of antibiotic risk (RADAR) categorizes and ranks patients q(1) Overall clinical outcome q(2) Duration of treatment risk q DOOR is the rank for patients in the new strategy compared to the old strategy 16

Analysis of Headache Data Using DOOR Pain free AE SAE Description (1) X O O Pain-free, no AE, and no SAE (2) X X O Pain-free and no SAE, but with AE (3) O O O Not pain-free, no AE, and no SAE (4) O X O Not pain-free and no SAE, but with AE (5) X O X Pain-free and no AE, but with SAE (6) O O X Not pain-free and no AE, but with SAE q Used ranked categories for DOOR including tied ranks q Alternative analysis using ordered categorical data analysis (Chuang-Stein et al. , 1991, 1994 and Claggett et al. 2015) q Other categorizations are possible and have been recommended 17

Analysis of Headache Data Using DOOR Comparison DOOR Analysis Prob. Gt. Rank (95% C. I. ) P-value 1 -sided Proportion Favorable Win Ratio Placebo vs. Low Dose 0. 533 (0. 495, 0. 570) P = 0. 046 PF = 0. 068 WR = 1. 285 Placebo vs. High Dose 0. 572 (0. 533. 0. 610) P = 0. 00013 PF = 0. 140 WR = 1. 604 Placebo vs. Active 0. 542 (0. 502, 0. 582) P = 0. 021 PF= 0. 086 WR = 1. 294 Active vs. Low Dose 0. 491 (0. 449, 0. 532) P = 0. 674 PF = -0. 018 WR = 0. 950 Active vs. High Dose 0. 574 (0. 486, 0. 568) P = 0. 100 PF = 0. 046 WR = 1. 136 18

Concluding Remarks q Methods such as Multi-Criteria Decision Analysis (MCDA) have focused on utility measures using weighted combinations of the estimated treatment effects across the favorable and unfavorable outcomes q Categorical methods analyze an ordered grouping of patients based on efficacy with or without AE q DOOR and Win Ratio are based on comparing pairs of patients to determine a win/lose/tie outcome q Summary statistics use multivariate extensions of the well known U statistics and ranking methods q Design considerations for both methods can be developed. 19

Concluding Remarks (cont. ) q DOOR and Win Ratio have some important differences that need to be considered in applications q. DOOR requires an explicit ranking q. Ties: DOOR counts ties as ½; WR looks at either wins – losses or wins/losses and ties are not counted q. WR relies on an explicit ordering of the multivariate response collective over efficacy and adverse events q DOOR and win ratio have very rich applications beyond benefit risk analysis such as composite endpoints and personalized treatment choices 20

References q q q Bebu I, Lachin JM (2016). Large sample inference for a win ratio analysis of a composite outcome based on prioritized components. Biostatistics 17: 178 -187. Chuang-Stein, C (1994), “A New Proposal for Benefit-Less Risk Analysis in Clinical Trials, ” Control Clinical Trials, 15, 30– 43. Chuang-Stein C, Mohberg NR, and Sinkula M S (1991). “Three Measures for Simultaneously Evaluating Benefits and Risks Using Categorical Data from Clinical Trials, ” Statistics in Medicine, 10, 1349– 1359. Claggett B, Tian L, Castagno D, and Wei L J (2015). “Treatment Selections Using Risk–Benefit Profiles Based on Data From Comparative Randomized Clinical Trials with Multiple Endpoints, ” Biostatistics, 16, 60– 72. Evans SR, Follmann D (2016). Using Outcomes to Analyze Patients Rather than Patients to Analyze Outcomes: A Step Toward Pragmatism in Benefit: Risk Evaluation. Statistics in Biopharmaceutical Research 8: 386 -393. 21

References (cont. ) Evans SR, Rubin D, Follmann D, Pannello G, et al. (2015). Desirability of outcome ranking (DOOR) and response adjusted for duration of antibiotic risk. Healthcare Epidemiology CID 61: 800 -806. q Evans SR (2009). Noninferiority clinical trials. Chance 22: 53 -58. q Pocock SJ, Ariti CA, Collier TJ, Wang D (2012). The win ratio: a new approach to the analysis of composite endpoints in clinical trials based on clinical priorities. European Heart Journal (2012), 33: 176 -182. q Pocock SJ (1997). Clinical trials with multiple outcomes: a statistical perspective on their design, analysis, and interpretation. Controlled Clinical Trials, 18: 530 -545. q 22

Disclaimer The methods proposed to evaluate BR are solely the views of the authors and are not endorsed by Merck & Co. 23

Backup 24

Categories For Headache Data Categorization Placebo Low Dose High Dose Active 22 38 54 44 7 16 35 52 (3) Pain, No AE, No SAE 226 193 177 135 (4) Pain, AE, No SAE 67 65 70 87 (5) PF, SAE 0 2 0 5 (6) Pain, SAE 6 3 2 5 328 317 338 328 (1) PF, No AE, No SAE (2) PF, AE, No SAE Total N Key: PF = pain free at 2 hours; AE is mild or moderate AE, SAE is severe AE 25

Ordered Categorical Data Analysis Ordered Categories P-value (1 -sided) Rank Analysis P-value (1 sided) Placebo vs. Low Dose 0. 0151 0. 0455 Placebo vs. High Dose <0. 0001 Placebo vs. Active 0. 0056 0. 0206 Active vs. Low Dose 0. 6994 0. 6738 Active vs. High Dose 0. 0787 0. 0995 Comparison Note: Ordered categories are scored 1 through 6; Rank analysis using average ranks as scores 26