Copernicus Institute Uncertainty tools Sensitivity Analysis Error propagation

Copernicus Institute Uncertainty tools • • • Sensitivity Analysis Error propagation equations (TIER I) Monte Carlo analysis (TIER II) Expert Elicitation Scenario analysis NUSAP PRIMA Checklist model quality assistance Assumption analysis …. . . Universiteit Utrecht

Copernicus Institute Sensitivity analysis (SA) SA is the study of • The study of how the uncertainty in the output of a model (numerical or otherwise) can be apportioned to different sources of uncertainty in the model input • how a given model depends upon the information fed into it (Saltelli et al. , 2000). Universiteit Utrecht

Copernicus Institute Sensitivity analysis three types: • Screening • Local Sensitivity Analysis – Vary one parameter at a time over their range while keeping others at default value – Result: rate of change of the output relative to the rate of change of the input • Global Sensitivity Analysis – Vary all parameters over their ranges (dependencies!) – Result: contribution of parameters to the variance in the output Universiteit Utrecht

Copernicus Institute Uncertainty analysis = Mapping assumptions onto inferences Sensitivity analysis = The reverse process (slide borrowed from Andrea Saltelli) Universiteit Utrecht

Copernicus Institute Scenario analysis Example: IPCC TAR emission scenarios (IPCC, 2001) Universiteit Utrecht

Copernicus Institute Risks of climate change EU target Bron: IPCC, 2001 I Risks to unique and threatened species IV Aggregate impacts II Risks from extreme climatic events V Risks from future large-scale discontinuities III Distribution of impacts Universiteit Utrecht

Copernicus Institute Unexpected Discontinuities • undermine current trends. • create new futures. • influence our thinking about the future – and the past. • give rise to new concepts & perceptions. www. steinmuller. de Examples relevant to adaptation • Shut down of ocean circulation; • West Antartic Ice Sheet collapse; e. g. ATLANTIS study, Tol et al. 2006 Sudden events with: • unknown frequentist probability • low Bayesian probability • high impact • surprising character • Mega-outbreak of disease in agriculture; • Terrorist attack on Deltawerken during unprecedented storm tide; • Dengue epidemic in NL • Chemical accident upstream Rhine during period of extreme drought • . . Universiteit Utrecht

Copernicus Institute NUSAP Qualified Quantities • Numeral • Unit • Spread • Assessment • Pedigree (Funtowicz and Ravetz, 1990) Universiteit Utrecht

Copernicus Institute NUSAP in practice Case 1 VOC emissions from paint in the Netherlands Universiteit Utrecht

Copernicus Institute Universiteit Utrecht

Copernicus Institute How is VOC from paint monitored? VOC emission calculated from: • VVVF national sales statistics NL-paint in NL per sector • CBS paint import statistics • Estimates of paint-related thinner use • Assumption of VOC% imported paint • Attribution imported paint over sectors Universiteit Utrecht

Copernicus Institute Universiteit Utrecht

Copernicus Institute Universiteit Utrecht

Copernicus Institute Sources of error • • • • Definitional inconsistency Interpretation of definitions Boundaries between raw materials, products, assortment Miscategorization Misreporting via unit confusion Deliberate misreporting Miscoding Non-response Not counting small firms (reporting threshold CBS) Not counting non-VVVF members Firm dynamics Paint dynamics Computer code errors. . Universiteit Utrecht

Copernicus Institute Universiteit Utrecht

Copernicus Institute Universiteit Utrecht

Copernicus Institute Universiteit Utrecht

Copernicus Institute Universiteit Utrecht

Copernicus Institute Pedigree scores Trafic-light analogy <1. 4 red; 1. 4 -2. 6 amber; >2. 6 green Universiteit Utrecht

Copernicus Institute NUSAP Diagnostic Diagram high Danger zone Criticality Safe zone low strong Pedigree weak Universiteit Utrecht

Copernicus Institute NUSAP Diagnostic Diagram VOC% imp. paint Thin % Ind NS Decor Imp. Paint NS Ind Overlap VVVF/CBS imp Imp. Below threshold NS DIY NS Car Thin. % DIY-rest NS Ship Thin. % Car Gap VVVF-RNS Th. % decor Universiteit Utrecht

Copernicus Institute Case 2: Applying NUSAP to a complex model TIMER model: • 300 variables • 19 world regions • 5 economic sectors • 5 types of energy carriers • 2 forms of energy • some are time series about 160, 000 numbers Universiteit Utrecht

Copernicus Institute Morris (1991) • facilitates global sensitivity analysis in minimum number of model runs • covers entire range of possible values for each variable • parameters varied one step at a time in such a way that if sensitivity of one parameter is contingent on the values that other parameters may take, Morris captures such dependencies Universiteit Utrecht

Copernicus Institute NUSAP applied to TIMER energy model: Expert Elicitation Workshop • Focussed on 40 key uncertain parameters grouped in 18 clusters • 18 experts (in 3 parallel groups of 6) discussed parameters, one by one, using information & scoring cards • Individual expert judgements, informed by group discussion Universiteit Utrecht

Copernicus Institute Universiteit Utrecht

Copernicus Institute Instructions • Do the Pedigree assessment as an individual expert judgement, we do not want a group judgement • Main function of group discussion is clarification of concepts • Group works on one card at a time • If you feel you cannot judge the pedigree scores for a given parameter, leave it blank Universiteit Utrecht

Copernicus Institute Universiteit Utrecht

Copernicus Institute Example result gas depletion multiplier Radar diagram: Each coloured line represents scores given by one expert Same data represented as kite diagram: Green = min. scores, Amber= max scores, Light green = min. scores if outliers omitted (Traffic light analogy) Universiteit Utrecht

Copernicus Institute Universiteit Utrecht

Copernicus Institute Case 3 Chains of models EO 5 Environmental Indicators Universiteit Utrecht

Copernicus Institute RIVM Environmental Outlook • Scenario study issued every 4 years • hundreds of environmental indicators • basis for NL Environmental Policy Plan • Strongly based on chains of model calculations Universiteit Utrecht

Copernicus Institute Universiteit Utrecht

Copernicus Institute Calculation chain deaths and hospital admittances due to ozone 1 2 3 4 5 Societal/demographical developments VOC and NOx emissions in the Netherlands and abroad Ozone concentrations Potential exposure to ozone Number of deaths/hospital admittances due to exposure Universiteit Utrecht

Copernicus Institute Pedigree criteria for reviewing assumptions • • • Plausibility Inter-subjectivity peers Inter-subjectivity stakeholders Choice space Influence of situational restrictions (time, money, etc. ) • Sensitivity to view and preferences of analyst • Estimated influence on results Universiteit Utrecht

Copernicus Institute Workshop reviewing assumptions • • Completion of list of key assumptions Rank assumptions according to importance Elicit pedigree scores Evaluate method Universiteit Utrecht

Copernicus Institute Key assumptions deaths and hospital admittances due to ozone • Uncertainty mainly determined by uncertainty in Relative Risk (RR) • No differences in emissions abroad between the two scenarios • Ozone concentration homogeneously distributed in 50 x 50 km grid cells • Worst case meteo now = worst case future • RR constant over time (while air pollution mixture may change!) • Linear dose-effect relationship Universiteit Utrecht

Copernicus Institute Pedigree matrix for evaluating the tenability of a conceptual model Universiteit Utrecht

Copernicus Institute Model evaluation should focus on: • Purpose • Use • Quality • Transparency • Inclusiveness – A checklist tool can promote such a broader conception of model quality Universiteit Utrecht

Copernicus Institute Model Quality No simple solution for quality assessment of models. • Dense: modelling in ‘dense’ domains • Pitfalls: In such domains pitfalls are everywhere dense: some form of rigour is all that remains to yield quality • Craft: A modeller has to be a good craftsperson • System: discipline is maintained by controlling the introduction of assumptions into the model Universiteit Utrecht

Copernicus Institute Model Quality • Poor practice leads to wygiwyn: What You Get Is What You Need • Need heuristic that encourages self-evaluative systematization and refelxivity on pitfalls • Method of sytematization should not only provide guidance to how modellers are doing, • should also provide diagnostic help as to where problems may occur and why Universiteit Utrecht

Copernicus Institute Principles • Metric. There is no single metric for model performace • Truth. There is no such thing as a ‘correct’ model • Function. Models need to be assessed in relation to particular functions • Quality. Assessment is ultimately about quality to perform a given function Universiteit Utrecht

Copernicus Institute The point is: • . . . not that a model is good or bad but that there are ‘better’ and ‘worse’ forms of modelling practice • Models are ‘more’ or ‘less’ useful when applied to a particular problem. Objectives of a checklist: • Provide insurance against pitfalls in process • Provide insurance against irrelevance in application Universiteit Utrecht

Copernicus Institute Structure of checklist • Screening questions – Should you use this checklist at all? – Which parts of the checklist are potentially useful • Model and Problem domain – Intended function or application – Intended users – Problem domain Universiteit Utrecht

Copernicus Institute Structure of checklist - II • Assessment of internal strength – Parametric uncertainty and sensitivity – Structural uncertainty – Validation – Robustness – Model development practices Universiteit Utrecht

Copernicus Institute Structure of checklist - III • Interface with users – scale – choice of output metrics – tests for pseudo-precision/pseudoimprecision – management of anomalies – expertise Universiteit Utrecht

Copernicus Institute Structure of checklist - IV • Use in policy – incorporating stakeholders – translating results to broader domains – transparency in the policy process • Summary assessment – overall assessment – potential pitfalls Universiteit Utrecht

Copernicus Institute Example page from checklist Universiteit Utrecht