Advanced Sensitivity Analyses Probabilistic Correlated and Scenario July

Advanced Sensitivity Analyses: Probabilistic, Correlated and Scenario July 26, 2006 Mendel E. Singer, Ph. D. Case School of Medicine 216 -368 -1951 mendel@case. edu

Today’s Agenda Role of Sensitivity Analysis Different Types of Sensitivity Analysis 1 -way, 2 -way Scenario Correlated Probabilistic Why and when to do each type Examples How to’s – in theory and in Tree. Age

Structure of a CEA Paper Introduction/Background – Why the reader should continue reading Methods – Relevance Reference case, Strategies – Credibility Assumptions Results – Baseline Main message – Sensitivity Analysis Remove the reader’s doubts Discussion – If they are still reading, you have your chance

1 st Order Uncertainty Variability across patients – Not all patients alike Different characteristics lead to different results Different mean values for a parameter – All Patients Alike E. g. simulated cohort of identical patients or patient subgroups Probabilistic vs. Deterministic Variability in outcomes – results not always the same

2 nd Order Uncertainty Parameter value/distribution not known with certainty – Origin of the estimate Clinical trials Retrospective study Variation in results across published studies Expert opinion Variability around the parameter mean

Structural Uncertainty about model assumptions – Affects the structure of the tree – Examples Some possible but improbable events excluded Certain sequences not considered Various scenarios “averaged” – E. g. dose titration has many permutations No repeat surgery for a chronic illness Patients who stop treatment don’t restart treatment

Types of Sensitivity Analysis 1 -Way 2 -Way Scenario Correlated Probabilistic

1 -Way Sensitivity Analysis How to: – Reanalyze the model for a range of clinically plausible values to see if either extreme has an impact on the results Why? – Uncertainty in original estimate, e. g. 95%CI – Misspecification bias Problem definition doesn’t match source of parameter value Weakness in study that is source of parameter value 2 nd-order (parameter) uncertainty 1 st-order uncertainty not well addressed

1 -Way Sensitivity Analysis (2) Good for removing doubts about a “problem” parameter Good for identifying research priorities Easy to understand When? – Expected in every decision analysis/costeffectiveness analysis

2 -Way Sensitivity Analysis How to: – Select 2 parameters – Consider combinations of values for the 2 parameters, keeping the values for each within a clinically plausible range – For each combination determine the point of indifference between the strategies, i. e. threshold value – Graph the point of indifference for each combination – Are there regions where the conclusion changes?

2 -Way Sensitivity Analysis (2) Why? – May not be sensitive to 1 variable, but may be to a combination – Often thematic Different patient subgroup Same source for 2 parameters – may be systematic bias 2 components both poorly known or arbitrary – E. g. Vaccine efficacy vs Cost of vaccine for coming vaccine – Uncertainty in original estimates, e. g. 95%CI – Misspecification bias Problem definition doesn’t match source of parameter value Weakness in study that is source of parameter value

2 -Way Sensitivity Analysis (3) 2 nd-order (parameter) uncertainty 1 st-order uncertainty somewhat addressed Difficult to understand for oral presentations or for audience in experienced in methods When to do? – Usually only done when there is a strong reason (thematically linked parameters)

3 -way sensitivity analysis Can be done with 3 -D graphs, OR Done with animation on a computer to simulate the 3 rd dimension Tree. Age can do it Many people can grasp it, though the simulation won’t work in print I cannot grasp it, so learn it from someone else! Sorry, my brain only works in 2 dimensions

Scenario Sensitivity Analysis How to: – Create a descriptive scenario of interest that leads to changes in parameter values and/or changes in the structure of the model – Change the tree and parameter values as needed – Rerun the baseline analysis

Scenario Sensitivity Analysis (2) Why? – Alternative problem definition E. g. different setting or patient population – Need to change more than 2 parameters – Don’t need combinations of values of 2 parameters, just 1 specific combination – Important clinical scenario which either: 1 - and 2 -way sensitivity analyses won’t address Forms a valid, alternative baseline analysis

Scenario Sensitivity Analysis (3) 2 nd-order (parameter) uncertainty 1 st-order uncertainty to a degree Structural uncertainty Good for removing doubts about a small variation on the problem definition Useful when there are really several related problems to address Easy to understand When to do? – Only done when there is a clear reason

Scenario Sensitivity Analysis: Examples Patient with a common comorbid condition that was excluded in baseline – Can affect response rates, complication rates, surgical mortality rates, etc… Different setting multiple parameter changes – VA outpatient vs. managed care outpatient – Presentation to physician vs. emergency dept. – Different country Different practice patterns, costs, genetics

Scenario Sensitivity Analysis in Tree. Age No special techniques since we all have to do is change some default values and rerun the main cost-effectiveness analysis. Here’s what I suggest: – Save the tree under a new file name – Edit the values to reflect those of the scenario sensitivity analysis. – Run the cost-effectiveness analysis – Save your work.

Correlated Sensitivity Analysis How to: – Select >=2 parameters that are thematically linked such that as one parameter changes in one direction, the others are known to change in the same or opposite direction, but not stay the same – As the main variable changes within its plausible range, each correlated variable changes in the specified direction by the same proportionate amount in the specified range – Reanalyze like 1 -way sensitivity analysis on the main variable, with all correlated variables changing “behind the scenes” – E. g. main variable changes to halfway between baseline and upper limit, correlated variables also move halfway between its baseline value and the limit of the range.

Correlated Sensitivity Analysis (2) Why? – Possible systematic bias From same underlying study – May have had a healthier or sicker population – May have possible bias in methods leading to results that are biased in one direction Same method for each parameter, possible bias – E. g. utilities all done in providers as proxies for patients

Correlated Sensitivity Analysis (3) 2 nd-order (parameter) uncertainty 1 st-order uncertainty not very well Structural uncertainty – not really Good for dealing with cases where there is clear reason for concern that a systematic bias is present, or when there is clear reason to believe that some readers will think there is a systematic bias and this approach can alleviate those fears and restore reader confidence Easy to understand When to do? – Any time there is a valid or perceived fear of systematic bias

Correlated Sensitivity Analysis: Examples Utility values for stages of a disease – Answer what if the impact of the illness has systematically been under-or over-estimated Surgical mortality inherent in the procedure, not due to patient differences – Affects surgical mortality rates for different paths in tree, e. g. before or after some other intervention or complication

Probabilistic Sensitivity Analysis How to: – For each parameter’s point estimate, create a probability distribution for the mean value – Create any needed correlations – Create a random sample of parameter values using the distributions – Run an incremental cost-effectiveness analysis using this sample’s parameter values – Repeat for 1000 samples and analyze results Plot the individual samples showing the CE quadrants Proportion of time a strategy is cost-effective calculated at different values of willingness-to-pay

Probabilistic Sensitivity Analysis (2) Why? – Many parameters, all with uncertainty – Impact of variation simultaneously in all variables is unanswered by traditional methods of sensitivity analysis – Traditional incremental CEA based on mean values creates a single value, which can be misleading since it ignores the inherent variation due to uncertainty in all parameter estimates

Probabilistic Sensitivity Analysis (3) 2 nd-order uncertainty – does this well Does not address 1 st-order uncertainty and structural uncertainty Does not address misspecification Complements rather than replaces traditional 1 and 2 -way sensitivity analyses When to do? – Almost always. Can be exceptions, and sometimes it is obvious it won’t change anything. – Becoming expected, and some journals and many referees require it

Probability Distributions Uniform – Good for numbers based on arbitrary choice or expert opinion Normal – Good when based on solid estimates – Negative numbers allowed Beta – Restricts values to 0, 1 – Allows all kinds of shapes Log-Normal – – Good for skewed data Nonnegative Gamma – Nonnegative – Different shapes

WARNING!! PROCEED AT RISK! What follows is my own personal approach to conducting probabilistic sensitivity analysis. Others use different distributions. I am only showing you what I do, and will demonstrate the steps to actually doing it. Any dangers to your personal or professional health are not the responsibility of the presenter or the VA.

Key Idea For each parameter, take the baseline value as the mean for your distribution Use the upper and lower limits from your 1 -way sensitivity analysis to form the limits of an estimated 95% confidence interval Approximate the width of the confidence interval as mean +/- 2 standard deviations (thank you Andy Briggs).

Costs If based on solid estimates (e. g. Medicare reimbursement rates) for known specific health care utilizations, I used a Normal distribution with confidence interval +/25% of the mean value. 25% of mean = 2 standard deviations Standard deviation =. 125 * mean

Costs (2) If costs are based on studies, such as annual cost of care for a chronic illness, then I use wider confidence intervals and use a Gamma distribution. I often halve and double these kinds of costs for the high and low values for the range. The range (high – low) is about 4 std dev’s. Std dev = (high-low) / 4 Parameters of gamma distribution: Alpha = (mean/sd)2 Lambda = mean/sd 2

Probabilities and Utilities I use a beta distribution to keep it between 0 and 1. It is also flexible in shape. sd = (high – low) / 4 n = (mean * (1 -mean) / sd 2) – 1 r = mean * n Parameters of Beta distribution: alpha = r beta = n - r

Problem Definition Reference Case – 60 -year old male – 4 cm abdominal aortic aneurysm – Otherwise, patient is in good health Surgery vs Watchful Waiting Time horizon: 5 years Effectiveness Measure: Survival – Alive = 1 – Dead = 0

State Transition Diagram Acute Expansion Aneurysm Well Dead 1. If Death from other causes was considered, one might show all states with an arrow to death, or merely note this in text below the diagram. 2. Not shown are arrows from a state to itself, which is valid for all states except Acute Expansion.

Costs Cost of Annual Screening, $579 Normal distribution Mean = 579 sd =. 125 * mean =. 125*579 = 72 Cost of elective surgery, $49, 381 Normal distribution Mean = 49, 381 sd =. 125 * 49, 381 = 6, 173

Annual Discount Rate Baseline = 3% Range 1% - 5% Uniform distribution Low value = 0. 01 High Value = 0. 05

Probabilities Annual probability of expansion Baseline =. 033 Range =. 016 -. 049 sd = (high-low)/4 = (. 049 -. 016)/4 =. 00825 n = (mean*(1 -mean)/sd 2)– 1 ((. 033)(. 967)/. 008252)-1= 467. 85 r = mean * n =. 033 * 468 = 15. 44 Parameters of Beta distribution: alpha = r = 15. 44 beta = n – r = 467. 85 – 15. 44 = 452. 41

We first need to enter the distributions to be used in the PSA. Select Distributions from the Values menu.

Since there are no distributions yet, the box is blank. Click on New to create the first distribution.

You are shown the different distributions, each depicted with a typical shape. It defaults to the Normal dbn. This can be seen by the green shading in the diagram for Normal.

For the selected distribution, the pertinent parameters defining the distribution are shown on the right, where you can set these values.

For the cost of annual screening, the parameters of the Normal distribution were determined to be: mean = 579 and std deviation of 72, so this was entered. Then click OK. We will need to create a different distribution for each parameter.

Each distribution has an associated index. This number will show up when the distribution is used in the model. However, the name and description will show up in the list of distributions when you need to select which distribution to use. I like to name the distribution after the parameter it is associated with.

The name must be distinct from any name already used for a variable. But you don’t have to enter a name. I tend to not create a name at all, but instead, for the description I enter exactly the name of the variable I created in Tree. Age. Now click OK to finish creating this distribution, and go on to the other variables.

The distribution we just created now shows up in the list of distributions. We will need to enter distributions for each of the parameters. Let’s create one more together, but this time let’s do the probability of acute expansion. It requires a beta distribution.

Like before, the default distribution is the Normal dbn. We need to click on Beta to select the distribution type. Then the correct parameters will appear.

After selecting the Beta distribution, the Beta diagram is shaded green. The parameters have been changed to n and r. BUT…also note that the choice for integer parameters is selected, but we want real-numbered parameters.

After selecting real-numbered parameters, the parameters for the Beta distribution are changed to alpha and beta. We need to change the default values to 15. 44 and 452. 41.

Now we can click OK and then finish up creation of this distribution.

At this screen, the index was defaulted for me. I then typed in the description as the name of the corresponding variable. The order you enter the variables is the order for the index, which is used to list the distributions when selecting a distribution to use in the model. You might want the order to be alphabetical. In this case, that means this should be index 8. So, I’ll change it before clicking OK to be done.

Back to the tree. We need to change the values of the variables so that instead of a default value, we supply the distribution. The default values all show up in the variables window of the root node. First, we have to see the variables window. Select the root node and then…. .

Select Show variables window from the Values menu.

Now we can select each variable one at a time, and change the value assignment from a value to its corresponding distribution. Let’s select c. Annual. Screening.

The default baseline value of 579 is showing. We want to select the option on the right to Insert Distribution,

Select the distribution c. Annual. Screening, and click Use.

The distribution is now being used for the value of c. Annual. Screening. Note that the distribution is identified by the index number, and not the name or description for the distribution. Then click OK. We can then use the same technique to enter the rest of the distributions.

When all done, the variables window looks like this. Not very informative. After doing this, we may want to see the variables, so select Preferences from the Edit menu.

Select the option to Show definitions. This has the advantage of letting you see the variables, but it also means the tree takes up a lot more space. This means you don’t see as much of the tree at once.

Before running the analysis, we should check to see if the numeric formatting is OK. Select Numeric Formatting from the Edit menu.

For each category, Cost, Effectiveness and Cost/Eff, select it and set the number of decimal places to 3.

Before running the PSA, it is a good idea to first run the cost-effectiveness analysis again to see if you get the same answers as before. Tree. Age will use the mean values of the distributions for the parameter values if you do an ordinary cost-effectiveness analysis. Whew! Same answer!

To run the PSA, be sure the root node is selected and choose from the menus: Analysis/Monte Carlo Simulation/Sampling (PSA).

The option Sample all distributions was selected by default, and that IS what we want! The number of samples also defaulted to what we want – namely, 1000. This arbitrary, but commonly used. I then select More Options. This will allow me to set the seed for the random number generator.

I had to select the option at the top, Seed random number generator. This is the number used to start the algorithm Tree. Age uses to create random numbers. It is chosen randomly, sort of, each time. This causes a problem – we will get different answers each time. That can make it hard to detect if we have changed something inadvertently. Therefore, I always choose to fix the seed. I chose 1.

I had to select the option at the top, Seed random number generator. This is the number used to start the algorithm Tree. Age uses to create random numbers. It is chosen randomly, sort of, each time. This causes a problem – we will get different answers each time. That can make it hard to detect if we have changed something inadvertently. Therefore, I always choose to fix the seed. I chose 1. Easy to remember.

Now we are ready. Click on Begin.

At the bottom we are told that the statistics shown are for Elective Surgery. We can use the drop down list to see the same statistics for Watchful Waiting. Instead, lets try to see the Acceptability curve. Click on Graph.

Select Acceptability Curve from the drop-down list.

Here, click on No. We want to see the pairwise comparison.

Here I selected a range from 20, 000 to 100, 000 for willingness to pay. By using 8 intervals, we will get results in increments of 10, 000. For the baseline, I selected Watchful Waiting. For comparator(s) I selected Elective Surgery.

Whoops! Problem here. The Elective Surgery arm is dominated. PSA isn’t going to help. It will still lose even at the extreme end of the willingness to pay range. So, for a better example of what it should look like, here is a graph from an upcoming paper of mine. The problem is looking at whether to use growth factors as a first step before dose reduction of ribavirin in Hepatitis C patients experiencing anemia during combination therapy.

For each willingness to pay threshold, the graph plots 2 points: the proportion of the time each strategy is costeffective among the 1000 samples. These 3 values always add up to 1, so the 2 curves always intersect at p=. 5. At $50, 000/QALY or greater the darbopoetin alpha strategy is cost-effective in over 60% of the samples, and appears to be cost-effective compared to Dose Reduction.

To do correlated sensitivity analysis, we go to the properties window of one of the variables. I chose the probability of death from elective surgery. Click on Correlations.

We need to select all the variables to be correlated. I will choose just the p. Die. Surgery. No. Rupture, which is already shown above as selected. I then click on the >> button to add it to the list of correlated variables.

Tree. Age then prompts us to set whether the correlation is positive or negative. That is, when one variable changes, does the other change similarly or in the opposite direction. When done, click OK, and do so again when you return to the list (unless you want to add another correlated variable. That’s it! Now, when you do sensitivity analysis on one of these variables, the other one will automatically change, too. I’d like to do more, but tick, tick…. .

Questions or follow-up? Contact Mendel Singer at: 216 -368 -1951 or mendel@case. edu