Sensitivity analysis Marko Tainio Decision analysis and Risk

Sensitivity analysis Marko Tainio Decision analysis and Risk Management course in Kuopio 21. 3. 2011

Sensitivity analysis studies on what happens inside the Black Box Data Black Box (the model) Results

Outline of lecture • What is sensitivity analysis? – Why to use sensitivity analysis – What options there are? • Example of two sensitivity analysis methods – Nominal Range Sensitivity – Rank-order Correlation • Other sensitivity analysis methods

What is sensitivity analysis

Definition of sensitivity analysis • http: //en. wikipedia. org/wiki/Sensitivity_analysis • Sensitivity analysis (SA) is the study of how the variation (uncertainty) in the output of a mathematical model can be apportioned, qualitatively or quantitatively, to different sources of variation in the input of the model. • Put another way, it is a technique for systematically changing parameters in a model to determine the effects of such changes.

Main idea of SA • In sensitivity analysis you change the input parameters to see how the model results response to these changes • Thus, sensitivity analysis resembles laboratory research where you control input and measure the outcome – Same statistical methods are applied in laboratory studies and in sensitivity studies (correlation, regression analysis, ANOVA)!

Different sensitivity analysis methods Frey and Patil, 2002 divides sensitivity analyses to three broad categories: 1. Mathematical • Suitable for deterministic models 2. Statistical (or probabilistic) • Usually based on simulation and statistical parameters. 3. Graphical • Presenting of sensitivity with graphs, charts etc.

When to use sensitivity analysis • In simple, always when making risk or decision models! • Two main advantages: – You can guide your own modeling work by testing the sensitivity of the model while doing the assessment; – You can also communicate to possible users the main uncertainties related to assessment

Calculation of sensitivity analysis: - Nominal Range Sensitivity - Rank-order Correlation

Setting • How many square meters of tables we have in this building? • The model is simple: – (Number of tables) x (average width) x (average height) = n x w x h • Since we don’t know any of these parameters, we assume some distributions for them

Nominal Range Sensitivity Analysis Method • NRSA is used to evaluate the effect on model outputs of varying only one of the model inputs across its entire range of plausible values, while holding all other inputs at their nominal or base-case values • Equation: Page 14, Frey 2

NRSA sensitivity analysis Model is most sensitive to Number of tables parameter.

Qualities of NRSA analysis • Advantages: – Works with deterministic models (no need for Monte Carlo) – Easy to use and apply in number of models • Disadvantages: – Works only with linear models – Doesn’t take into account interactions/correlations between input parameters • NRSA is a good screening level sensitivity analysis tool

Sample and Rank Correlation Number of tables Coefficients 50 150 300 Height Result Model (aka. Black Box) 0. 5 1. 0 1. 2 Width 1 1. 5 2. 0 104 214 384

Sample and Rank Correlation Coefficients Two options for correlation analysis: 1. Parametric or Pearson – For linear models 2. Non-parametric or Spearman or rank – Also for non-linear models – Importance analysis • Correlation varies between -1 and 1 – The value of -1 represents a perfect negative correlation while a value of +1 represents a perfect positive correlation

In Monte Carlo, correlation is calculated between samples Correlation

Result of the rank-order correlation sensitivity analysis The uncertainty in the results correlates 80% with the uncertainty of „Number of tables” parameter.

Qualities of correlation sensitivity analysis • Advantage: – Easy to compute – Correlation available in most of the computer modeling tools (including Excel) • Disadvantage: – Correlation is not causation – Non-linear and non-monotonic models are problematic

Other sensitivity analysis methods

Research on sensitivity analysis • Frey et al. 2003 (Evaluation of Selected Sensitivity Analysis Methods Based Upon Applications to Two Food Safety Process Risk Models) lists 11 different sensitivity analysis methods • They also made recommendations on which sensitivity analysis to use in which situation • Report available: http: //www. ce. ncsu. edu/risk/Phase 2 Final. pdf

Different sensitivity analysis methods • Mathematical Methods for Sensitivity Analysis – Nominal Range Sensitivity Analysis Method – Differential Sensitivity Analysis (DSA) • Statistical Methods for Sensitivity Analysis – – – – Sample and Rank Correlation Coefficients Regression Analysis Rank Regression Analysis of Variance Classification and Regression Tree Sobol’s Indices Fourier Amplitude Sensitivity Test (FAST) • Graphical Methods for Sensitivity Analysis – Scatter Plots – Conditional Sensitivity Analysis

Selection of the sensitivity analysis (Frey et al. 2004) • Some selection criteria's: – What are the objectives of sensitivity analysis? – Based upon the objectives, what information is needed from sensitivity analysis? – What are the characteristics of the model that constrain or indicate preference regarding method selection? – How detailed is the analysis? – What are the characteristics of the software that may constrain selection of methods? – What are the specifications of the computing resources? – Can “push-button” methods adequately address characteristics of interest in the analysis? – Is the implementation of the selected sensitivity analysis method post-hoc? Frey et al. pages 46 -47, http: //www. ce. ncsu. edu/risk/Phase 3 Final. pdf

Some objectives of sensitivity analysis • Rank ordering the importance of model inputs (e. g. , critical control points); • Identifying combination of input values that contribute to high exposure and/or risk scenarios; • Identifying and prioritizing key sources of variability and uncertainty; • Identifying critical limits; • Evaluating the validity of the model.

What Information is Needed from Sensitivity Analysis? • Qualitative or quantitative ranking of inputs • Discrimination of the importance among different inputs • Grouping of inputs that are of comparable importance • Identification of inputs that are not important • Identification of critical limits • Identification of inputs and ranges that produce high exposure or risk • Identification of trends in the model response

Frey et al. Pages 58, http: //www. ce. ncsu. edu/risk/Phase 3 Final. pdf

Further reading • Frey et al. 2004. Recommended Practice Regarding Selection, Application, and Interpretation of Sensitivity Analysis Methods Applied to Food Safety Process Risk Models: http: //www. ce. ncsu. edu/risk/Phase 3 Final. pdf • Frey et al. 2003. Evaluation of Selected Sensitivity Analysis Methods Based Upon Applications to Two Food Safety Process Risk Models: http: //www. ce. ncsu. edu/risk/Phase 2 Final. pdf • Patil and Frey 2004. Comparison of sensitivity analysis methods based on applications to a food safety risk assessment model. Risk Analysis 24 (3): 573 -585