Partitioned survival analysis current use in NICE submissions

Partitioned survival analysis: current use in NICE submissions and critical review Marta Soares, Senior Research Fellow Centre for Health Economics marta. soares@york. ac. uk Kuopio, Finland, August 2018 (acknowledgement: Beth Woods)

Partitioned Survival Analyses Overview of today’s lecture: . Introduction. Implementing Part. SA. Review of NICE apprsals. Step by step guided exercise (simulated data). Features of Part. SA and considerations on its use

Partitioned Survival Analyses Introduction

What is decision modelling for health care decision making? ● Health systems need to make a number of decisions on health care resources: which, whom, when, where… ○ Typical example is reimbursement/access (NICE) ○ Other examples: individual funding requests (NHS England) ● Decisions informed by HTA including cost-effectiveness, the latter comparing interventions in terms of: ○ long-term effects on population health (typically measured in QALYs), and ○ overall cost implications for relevant stakeholders or individuals. ● Evaluations are often model-based, where evidence from multiple sources ● and on multiple aspects of disease and treatment are considered together, and the model used to extrapolate to the long term. Uncertainty in the evidence can result in uncertain cost-effectiveness estimates. 5

Decision modelling State transition models Dead Stable disease Progressed disease 6

Decision modelling Parameter space Evidence space Evidential uncertainty Transitions Relative effectiveness Breadth and quality of evidence Empirical evidence Less representative Sparse Health state costs Non high quality Health state HRQo. L weights Mechanistic evidence Structural uncertainties Judgements Indirectly relevant evidence Inconsistent evidence 7

Decision modelling for decision making ● Which intervention is worthwhile using ○ Based on expected values ○ Rejecting an intervention that is expected to be beneficial on the grounds of uncertainty solely is not ethically justified ● Under uncertainty, further timely research may be needed ○ To reverse initial guidance is proven wrong ○ Avoid the consequences of uncertainty ● Approval/rejection can be conditioned on research ○ Approval With Research (AWR) ○ Only in Research (Oi. R) 8

Partitioned Survival Analyses ● Decision models highly reliant on intervention-specific estimates of time spent in different health states ● Partitioned Survival Analysis (Part. SA) or “Area Under the Curve” modelling widely used in oncology P&R decisions ● Method not subject to formal evaluation ● NICE technical support document (TSD) commissioned to provide: ○ Description of method ○ Review of applications in NICE TAs ○ Critique of approach ○ Recommendations 9

Partitioned Survival Analyses Implementing Part. SA

Partitioned Survival Analyses Two steps: Survival analytic method: independent survival analyses for each health state n, of time from model entry to transiting to N>n Decision model: Basic manipulations used to derive proportion in each mutually exclusive health state over time Two modelled survival curves: -- PFS, Progression survival: Time to first STEP 1: free Inference of progression or death -- OS, Overall survival: Time to death STEP 2: Modelling 11

STEP 2: Modelling 12

Partitioned Survival Analyses, Breakdown of steps Steps: ● PFS 1. 2. Inference a) IPD or Kaplan-Meier (non-parametric) curves b) Fit a number of smooth distributions c) Choose one of the alternative distributions Modelling a) Read values at times of the cohort model b) Fill in the trace ● Repeat the same process for OS 13

Example 14

Partitioned Survival Analyses, Example 15

PFS 16

Fitting (TSD 14) • A number of smooth parametric functions available: Exponential, Weibull, Gompertz, Log. Normal, piecewise models • Assessing suitability ○ Visual inspection ○ Log-cumulative hazard plot ○ AIC/BIC tests Akaike and Bayesian information criteria – AIC and BIC Consider goodness of fit + parsimony (penalise the number of parameters) ○ Limitation : all based on relative fit to observed data; does not inform suitability for time beyond follow-up 17

18

19

STEP 1: Inference 20

21 21

STEP 2: Modelling 22

● Markov trace Time Pre- prog Pemb Progressed Pemb Dead Pemb Pre-prog SOC 0 1 1 1 0. 48 0. 18 2 0. 32 0. 04 3 0. 25 0. 01 4 0. 18 0. 00 5 0. 15 0. 00 Progressed SOC Dead SOC 23

24

25

26 26

27

● Markov trace Time Pre- prog Pemb 0 Progressed Pemb Dead Pemb Pre-prog SOC Progressed SOC Dead SOC 1 0 1 0. 48 0. 3 0. 18 0. 5 2 0. 32 0. 5 0. 04 0. 8 3 0. 25 0. 62 0. 01 0. 9 4 0. 18 0. 72 0. 00 0. 95 5 0. 15 0. 80 0. 00 0. 99 28

● Markov trace Time Pre- prog Pemb Progressed Pemb Dead Pemb Pre-prog SOC Progressed SOC Dead SOC 0 1 0. 48 0. 22 0. 3 0. 18 0. 32 0. 5 2 0. 32 0. 18 0. 5 0. 04 0. 16 0. 8 3 0. 25 0. 13 0. 62 0. 01 0. 09 0. 9 4 0. 18 0. 1 0. 72 0. 00 0. 05 0. 95 5 0. 15 0. 05 0. 80 0. 01 0. 99 29

External evidence on comparators ● External evidence typically uses an NMA on HR’s for PFS and OS; again, the NMA’ s are typically ran independently ● NMA on PFS 30

● NMA on OS 31

Partitioned Survival Analyses, Breakdown of steps 1. Inference a) IPD or Kaplan-Meier (non-parametric) curves b) Fit a number of smooth distributions Akaike's information criterion and Bayesian information criterion --------------------------------------Model | Obs ll(null) ll(model) df AIC BIC -------+-------------------------------exponential | 600 -503. 3867 -501. 2607 2 1006. 521 1015. 315 weibull | 600 -503. 1442 -501. 0364 3 1008. 073 1021. 264 gompertz | 600 -499. 8446 -497. 8114 3 1001. 623 1014. 813 lognormal | 600 -497. 1489 -496. 0835 3 998. 1669 1011. 358 loglogistic | 600 -501. 2529 -499. 458 3 1004. 916 1018. 107 ggamma | 600 -495. 2464 -494. 9719 4 997. 9439 1015. 532 --------------------------------------Note: N=Obs used in calculating BIC; see [R] BIC note. 32

Partitioned Survival Analyses Review of NICE appraisals

Review of NICE appraisals ● 30 most recent cancer appraisals (05. 2013 -02. 2016) ● Part. SA used in 22/30 (73%) ● Predominantly advanced/metastatic Reporting of TAs using the Part. SA method n/N (%) Correctly described by manufacturer (AG) 12/22 (55%) Correctly described by ERG/AG 12/22 (55%) Justification for method provided by manufacturer (AG) 7/22 (32%) Assumption of independence discussed by manufacturer/ERG/AG 4/22 (18%) 34

Review of NICE appraisals ● Modelled endpoints: PFS/TTD, OS ● For comparators included within pivotal trial(s): ● Individual patient data generally available ○ Survival curves modelled using range of methods ○ Almost always including parametric extrapolation ● For additional comparators ○ Only aggregate data available ○ Information from meta-analyses, indirect comparisons, network meta- analyses ● Treatment effects typically applied all endpoints for time horizon ● External data may also inform long-term extrapolations 35

Partitioned Survival Analyses Step by step exercise (simulated data) Software: . STATA for inference. Excel for modelling

Partitioned Survival Analyses, Data (complete) 37

Partitioned Survival Analyses, Data (censored) 38

Partitioned Survival Analyses, Data (complete) 39

Partitioned Survival Analyses, Data (censored) 40

Partitioned Survival Analyses PFS

Partitioned Survival Analyses Fitting STATA supported distributions (manual) 42

43

Partitioned Survival Analyses, Breakdown of steps PFS, Fitting Akaike's information criterion and Bayesian information criterion --------------------------------------Model | Obs ll(null) ll(model) df AIC BIC -------+-------------------------------exponential | 600 -772. 1269 -751. 7 2 1507. 4 1516. 194 weibull | 600 -772. 0307 -751. 2349 3 1508. 47 1521. 661 gompertz | 600 -772. 1081 -751. 6055 3 1509. 211 1522. 402 loglogistic | 600 -771. 4927 -754. 2902 3 1514. 58 1527. 771 --------------------------------------44

Partitioned Survival Analyses, Breakdown of steps PFS, Fitting 45

Partitioned Survival Analyses, Breakdown of steps PFS, extrapolation Life expectancy (predicted) C T Weibull 0. 90 1. 65 Log-logistic 3. 12 5. 96 Exponential 0. 90 1. 65 Gompertz 0. 92 1. 81 Dif 0. 75 2. 84 0. 75 0. 89 46

Partitioned Survival Analyses, Breakdown of steps Modelling, Exponential (best fitting) time 0 1 2 3 4 5 6 7 8 9 10 S(t) control 1. 00 0. 31 0. 09 0. 03 0. 01 0. 00 S(t) treat 1. 00 0. 54 0. 29 0. 15 0. 08 0. 04 0. 02 0. 01 0. 00 47

Partitioned Survival Analyses, Breakdown of steps PFS Modelling, Exponential. streg treat, d(e) nohr […] _t treat _cons Coef. -. 7148959. 2054434 Std. Err. . 1137388. 0707107 S(t)=exp(-l * time) l = exp(const + b * treatment) time 0 1 2 3 4 5 6 7 8 9 10 S(t) control 1. 00 0. 31 0. 09 0. 03 0. 01 0. 00 S(t) treat 1. 00 0. 54 0. 29 0. 15 0. 08 0. 04 0. 02 0. 01 0. 00 48

Partitioned Survival Analyses OS

Partitioned Survival Analyses, Breakdown of steps OS, Fitting Akaike's information criterion and Bayesian information criterion --------------------------------------Model | Obs ll(null) ll(model) df AIC BIC -------+-------------------------------exponential | 600 -449. 3264 -443. 0312 2 890. 0623 898. 8562 weibull | 600 -444. 4219 -437. 9418 3 881. 8837 895. 0744 gompertz | 600 -444. 2748 -437. 7265 3 881. 453 894. 6438 loglogistic | 600 -445. 1219 -439. 2173 3 884. 4345 897. 6253 ----------------------------------------------------------------------------50

Partitioned Survival Analyses, Breakdown of steps OS, Fitting 51

Partitioned Survival Analyses, Breakdown of steps PFS, extrapolation Life expectancy (predicted) C T Weibull 1. 83 2. 89 Log-logistic 6. 47 10. 60 Exponential 2. 47 4. 44 Gompertz 1. 41 1. 86 Dif 1. 06 4. 13 1. 97 0. 45 52

Partitioned Survival Analyses, Breakdown of steps PFS Modelling, Gompertz regression […] _t | Coef. Std. Err. treat | -. 60579. 1707958 _cons | -1. 3373. 1819161 /gamma |. 93202. 2865669 time 0 1 2 3 4 5 6 7 8 9 10 S(t) control 1. 00 0. 65 0. 22 0. 01 0. 00 0. 00 S(t) treat 1. 00 0. 79 0. 43 0. 09 0. 00 0. 00 53

Partitioned Survival Analyses Validity of predictions?

Partitioned Survival Analyses Features of Part. SA and considerations on its use

Advantages of Part. SA • Easy to construct and communicate • Generally validates well against trial data • Can use published summary data on PFS, OS ○ Less of a consideration for pivotal trial (IPD) ○ Major consideration for external data • Some analytic methods e. g. adjustments for treatment switching developed for OS endpoint 56

Disadvantages of Part. SA • Endpoints modelled independently despite ○ Including some of the same events ○ Events are structurally dependent ○ Intermediate events are often prognostic for risk and effects of treatment • For within-trial period, dependencies reflected in data • For analyses beyond the trial, dependency between endpoints ignored ○ OS determined only by time-to-death data ○ Extrapolations typically reflect within-trial trends in mortality rates only 57

Validity of extrapolations: baseline risk ● Assume transition probabilities constant ● Risk of death is elevated in progressed patients ● Risk of death depends on proportion of alive cohort who have progressed` 58

Validity of extrapolations: baseline risk 59

Validity of extrapolations: treatment effects ● Assume treatment modifies probability of progression (conforms to the ● proportional hazards assumption) Differences between treatments in mortality rates depend on differences in distribution across progression-free /progressed states 60

Validity of extrapolations: treatment effects ● Eventually population resides in progressed state ● Depending on when data is reviewed data may suggest HR decreasing/ ● stable/increasing in long term Ignoring information on progression and focusing on within-trial HR on OS problematic 61

Decision modelling ● State transition models (STMs) used extensively for modelling chronic diseases including cancer ● Describe biological/clinical processes and way interventions effect these processes ● Key role of decision models in NICE TAs ○ Synthesis of all relevant evidence ○ Comparison of all relevant interventions ○ Extrapolation of observed information over time ○ Characterise heterogeneity and uncertainty 62

Part. SA vs. state transition models (STMs) ● Part. SA: endpoints are modelled as independent OS determined only by time to death data ● STMs: clinical events explicitly and structurally related Differences across treatment arms ● OS reflects % in progressed state, and differences in mortality between progression-free and progressed patients i. e. OS depends on all three transitions 63

Is there really any difference? ● Part. SA and STM: Within-trial relationships between endpoints should be reflected in the data ● Extrapolation of OS in Part. SA ○ reflect within-trial trends in mortality rates, and ○ reflect the within-trial difference in hazard of death ● Extrapolation of OS in STM ○ reflect within trial estimates of each transition probability (conditioned by model structure), and ○ combined effect of treatment effects on individual transition probabilities ● Data requirements 64

State transition models, STMs ● Allow both the natural history of the disease and treatments effects on this to be reflected when extrapolating beyond the trial data ● Allows assumptions underpinning these extrapolations to be made explicit and therefore subject to scrutiny and sensitivity analyses 65

Non-Part. SA approaches used in NICE TAs ● 8/30 non-Part. SA models (27%) ● One used a response-stratified Part. SA (haematological cancer) ● Seven STMs. Vehicle to introduce assumptions (e. g PPS independent of treatment). Did not always use OS data that was available, or predict OS well Did not use established survival analysis methods for competing and sequential events Poor use of available data 66

Challenges in implementing STMs Differences across treatment arms (i) Estimation: when IPD available/recoverable (ii) Estimation: Data availability (iii) Implementation 67

Challenges in implementing STMs: (i) estimating transition probabilities with IPD ● Competing risks and multi-state modelling [tutorial in Williams MDM 2016] ● However: ○ Limited guidance on model selection and assessment ○ Challenges in modelling PPS (a ‘non-entry’ state): due to selection effects and dependent censoring , e. g. covariates/time dependency ● Naïve PPS analysis insufficient ○ More difficult to achieve reasonable fit to OS 68

Challenges in implementing STMs: (ii) Data availability ● Require data on individual transitions in aggregate or IPD form ● Unlikely to be available for external data ● Currently, no approach available ● Multi-parameter evidence synthesis / calibration may be feasible ● Trialists should be encouraged to report individual endpoint outcomes and share IPD 69

Challenges in implementing STMs: (iii) time-dependent transition probabilities ● Need for more advanced methods ○ Tunnel states ○ Semi-Markov approaches ○ Patient-level simulation ○ “Payoff” approaches (do not produce OS graphs) 70

Other major issues ● External data on long-term outcomes as a fix? ○ Not available for new interventions ● NICE methods guidance recommends assessing the “clinical and biological plausibility” of extrapolations and exploring alternative scenarios ○ Can’t scrutinise rates or treatment effects for individual transitions e. g. PPS ○ Can’t vary via sensitivity analyses ● PFS>OS ● PSA less meaningful 71

Summary ● Part. SA’s defining feature is that state membership is determined by independently modelled non-mutually exclusive survival curves ● No concerns where data is fully observed ● For extrapolation, however, Part. SA use prior trends in mortality, while STMs explicitly link mortality with intermediate prognostic events ● STMs allow assessing plausibility of extrapolations and meaningful sensitivity analyses (but are difficult to specify) 72

Partitioned Survival Analyses TSD recommendations

TSD recommendations (summary) ● Emphasis on importance of model conceptualisation and stakeholder recognition of limitations of Part. SA ● Survival curves corresponding to individual clinical events requested alongside Part. SA models ● Further research and guidance to support routine use of multi-state survival analysis ● State transition modeling recommended for use alongside the Part. SA approach (currently…) ○ assist in verifying the plausibility of Part. SA’s extrapolations ○ address further uncertainties in the extrapolation period ○ even if only for pivotal trial 74

TSD recommendations: Selecting, documenting and justifying the modelling approach Recommendation 1: The model conceptualisation process should be routinely reported and the rationale for the chosen modelling approach explicitly justified on the basis of theoretical and practical considerations. • Approach aligned with disease process and intervention effects • Appropriateness as vehicle for extrapolation • Evidential constraints Recommendation 2: Consistent and appropriate terminology should be applied in future appraisals when describing the Part. SA approach (e. g. use of the term “Partitioned survival analysis”). Recommendation 3: A summary of the main structural assumptions should be routinely reported and justified as required by the NICE guide to 75 the methods of technology appraisal.

TSD recommendations: Representing uncertainties associated with extrapolation Recommendation 4: All stakeholders should recognise the specific limitations of Part. SA for the purposes of extrapolation. • Limitations of prediction and limitations for exploring key uncertainties Recommendation 5: Modelling choices that influence outcomes in the extrapolation period should reflect all relevant evidence. • External data, within-trial data (recommendation 6), expert opinion Recommendation 6: Within-trial survival curves corresponding to individual clinical events should be supplied alongside Part. SA models. Recommendation 7: When extrapolation is required, Part. SA models should easily facilitate the investigation of alternative assumptions in accordance with current NICE methods guidance. 76

TSD recommendations: Use of alternative modelling approaches Recommendation 8: Transition probabilities within state transition models should be estimated using appropriate statistical methods and reflect all relevant evidence. • Competing risks and multi-state modelling, consideration of selection effects and dependent censoring, careful validation. • Care when incorporating external evidence given lack of established methods. • ERGs/AGs access to IPD may be necessary. Recommendation 9: Further research and guidance is required to support appropriate specification of state transition models using multi-state survival analysis. Recommendation 10: Further research is required to support incorporation of data external to the pivotal trial, and in particular data used to inform indirect comparisons, in to state transition models. 77

TSD recommendations: Use of alternative modelling approaches Recommendation 11: State transition modelling should be used alongside the Part. SA approach to assist in verifying the plausibility of Part. SA’s extrapolations and to address uncertainties in the extrapolation period, even if this is only plausible for the pivotal trial. Recommendation 12: Presentation of results from all Part. SA and state transition models should include tabulations showing the states in which life year and QALY differences between interventions accrue and a justification of why these differences should be considered plausible. Recommendation 13: Further research is required to identify the extent of possible biases associated with Part. SA and state transition models, and how this varies according to the context in which the approaches are used 78

Partitioned survival analysis: current use in NICE submissions and critical review Marta Soares, Senior Research Fellow Centre for Health Economics marta. soares@york. ac. uk Kuopio, Finland, August 2018 (acknowledgement: Beth Woods)