Uncertainty Assessment and Communication Jeroen van der Sluijs

Uncertainty Assessment and Communication Jeroen van der Sluijs Copernicus Institute for Sustainable Development and Innovation Utrecht University Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

Copernicus Institute Research topics • Energy and material demands and efficiencies (Dr. Martin Patel) • Possibilities for a more sustainable energy supply (Dr. Andre Faaij) • Land use, biodiversity and ecosystem functioning (Prof. Dr. Peter de Ruiter) • Innovation: systems, processes and policies (Prof. Dr. Ruud Smits) • Governance for sustainable development (Prof. Dr. Pieter Glasbergen) • Integrative models and methods and the management of risks and uncertainties (Dr. Jeroen van der Sluijs) Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

Complex environmental risks Typical characteristics (Funtowicz & Ravetz): • Decisions will need to be made before conclusive scientific evidence is available; • Decision stakes are high: potential error costs of wrong decisions can be huge • Values are in dispute • Knowledge base is mixture of knowledge and ignorance: Ø large (partly irreducible) uncertainties, knowledge gaps, and imperfect understanding; • Assessment dominated by models, scenarios, and assumptions • Many (hidden) value loadings in problem frames, indicators, assumptions Coping with uncertainty is essential Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

RIVM / De Kwaadsteniet (1999) “RIVM over-exact prognoses based on virtual reality of computer models” Newspaper headlines: • Environmental institute lies and deceits • Fuss in parliament after criticism on environmental numbers • The bankruptcy of the environmental numbers • Society has a right on fair information, RIVM does not provide it Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

Crossing the disciplinary boundaries Once environmental numbers are thrown over the disciplinary fence, important caveats tend to be ignored, uncertainties compressed and numbers used at face value e. g. Climate Sensitivity , see Van der Sluijs, Wynne, Shackley, 1998: Resulting misconception: ! 1. 5 -4. 5 °C Worst case = 4. 5°C Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 ?

The certainty trough (Mc. Kenzie, 1990) Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

In model based assessment and foresight of complex environmental problems, unquantifiableuncertainties dominate the quantifiable ones Unquantifiable uncertainties include those associated with • • • problem framings system boundaries model structures assumptions indeterminacies value ladenness Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

Insights on uncertainty • Uncertainty is partly socially constructed and its assessment always involves subjective judgement • Omitting uncertainty management can lead to scandals and crisis • More research does not necessarily reduce uncertainty – may reveal unforeseen complexities – irreducible uncertainty (intrinsic or practically) • High quality low uncertainty • Quality relates to fitness for function (robustness, PP) • Shift in focus needed from reducing uncertainty towards systematic attempts to explicitly cope with uncertainty Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

Dimensions of uncertainty • Technical (inexactness) • Methodological (unreliability) • Epistemological (ignorance) • Societal (limited social robustness) Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

RIVM Uncertainty Guidance Goals: · Structurean approach to environmental assessment that facilitates an awareness, identification and incorporation of uncertainty · Specifically address and relate the role of uncertainties in the context of policy advice · May not reduce uncertainties, but provides means to assess their potential consequences and avoid pitfalls associated with ignoring or ignorance of uncertainties · Guidancefor use and help against misuse of uncertainty tools · Provide useful uncertainty assessments · Promote the adoption of uncertainty methods · Facilitate design of effective strategies for communicating uncertainty. Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

Steps in uncertainty management Communication Problem framing & context analysis Process assessment Reporting Review & Evaluation Uncertainty Analysis Uncertainty Management Environmental Assessment methods Uncertainty identification & prioritization Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

Systematic reflection on uncertainty issues in: • Problem framing • Involvement of stakeholders • Selection of indicators • Appraisal of knowledge base • Mapping and assessment of relevant uncertainties • Reporting of uncertainty information Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

RIVM-MNP Uncertainty Guidance Reminder list Invokes Reflection Portal to QS Mini-Checklist Quickscan Questionnaire Further Guidance Advice Hints & Implications Tool Catalogue for Uncertainty Assessment Advice on Quantitative + Copernicus Institute Qualitative tools for UA Quickscan Hints & Actions List Detailed Guidance Download all volumes: www. nusap. net Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

Problem framing and context • • • Explore rival problem frames Relevant aspects / system boundary Typify problem structure Problem lifecycle / maturity Role of study in policy process Uncertainty in socio-political context Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

Type-III error: Assessing the wrong problem by incorrectly accepting the false meta-hypothesis that there is no difference between the boundaries of a problem, as defined by the analyst, and the actual boundaries of the problem (Dunn, 1997). Context validation (Dunn, 1999). The validity of inferences that we have estimated the proximal range of rival hypotheses. Context validation can be performed by a participatory bottom-up process to elicit from scientists and stakeholders rival hypotheses on causal relations underlying a problem and rival problem definitions. Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

In different phases of problem lifecycle, different uncertainties are salient Different problem-types need different uncertainty management strategies Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

Involvement of stakeholders • Identify relevant stakeholders. • Identification of areas of agreement and disagreement among stakeholders on value dimensions of the problem. • Recommendations on when to involve different stakeholders in the assessment process. Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

Indicators • How well do indicators used address key aspects of the problem? • Alternative indicators? • Limitations of indicators used? • Controversies in science and society about these indicators? Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

Adequacy of available knowledge base? • What are strong and weak points in the knowledgebase? – Use of proxies, empirical basis, theoretical understanding, methodological rigor, validation ÞNUSAP Pedigree analysis • What parts of the knowledge are contested (scientific and societal controversies)? • Is the assessment feasible in view of available resources? (limitations implied) Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

Example Pedigree matrix parameter strength Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

Example Pedigree results Trafic-light analogy <1. 4 red; 1. 4 -2. 6 amber; >2. 6 green This example is the case of VOC emissions from paint in the Netherlands, calculated from national sales statistics (NS) in 5 sectors (Ship, Building & Steel, Do It Yourself, Car refinishing and Industry) and assumptions on additional thinner use ( Th%) and a lump sum for imported paint and an assumption for its VOC percentage. See full research report on www. nusap. net for details. Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

NUSAP Diagnostic Diagram high Danger zone Combine results from sensitivity analysis and pedigree analysis by plotting each mode parameter in this diagram This reveals the parameters whose uncertainty is most problematic. Sensitivity low Safe zone strong Pedigree Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 weak

NUSAP Diagnostic Diagram VOC% imp. paint Thin % Ind NS Decor Imp. Paint NS Ind Overlap VVVF/CBS imp Imp. Below threshold NS DIY NS Car Thin. % DIY-rest NS Ship Example result from the case of emission monitoring of VOC from paint in NL. Thin. % Car Gap VVVF-RNS Th. % decor Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004 The most problematic parameter is the assumed VOC percentage in imported paint.

Mapping and prioritization of relevant uncertainties • Highlight uncertainties in typology relevant to this problem • Set priorities for uncertainty assessment • Select uncertainty assessment tools from the tool catalogue Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

Typology of uncertainties • Location • Level of uncertainty statistical uncertainty, scenario uncertainty, recognised ignorance • Nature of uncertainty knowledge-related uncertainty, variability-related uncertainty • Qualification of knowledge base (backing) weak, fair, strong • Value-ladenness of choices small, medium, large Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

Locations of uncertainties: • Context ecological, technological, economic, social and political representation • Expert judgement narratives, storylines, advices • Model model structure, technical model, model parameters, model inputs • Data measurements, monitoring data, survey data • Outputs indicators, statements Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

Tool catalogue For each tool: • Description • Goals and use • What sorts and locations of uncertainty does this tool address? • What resources are required to use it? • Strengths and limitations • guidance on application & complementarity • Typical pitfalls of each tool • References to handbooks, example case studies, web-sites, experts etc. Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

Tool catalogue • • • Sensitivity Analysis Error propagation Monte Carlo NUSAP Expert Elicitation Scenario analysis PRIMA Checklist model quality assistance Assumption analysis …. . . Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

Reporting • Make uncertainties explicit • Assess robustness of results • Discuss implications of uncertainty findings for different settings of burden of proof • Relevance of results to the problem • Progressive disclosure of information -> traceability and backing Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

IPCC WGI Proposal for Interpretation and Use of Probabilistic Terms Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

Weiss 2003 uncertainty scale 10. Beyond any doubt 9. Beyond a reasonable doubt 8. Clear and Convincing Evidence 7. Clear Showing 6. Substantial and credible evidence 5. Preponderance of the Evidence 4. Clear indication 3. Probable cause: reasonable grounds for belief 2. Reasonable, articulable grounds for suspicion 1. No reasonable grounds for suspicion 0. Impossible Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

Triggers that increase policy relevance of uncertainty • Close to a norm or target • Near a threshold of severe impact • On steep part of cost-curve or impact curve • Low pedigree • High (potential) valueladenness • ‘temperatute’ of scientific or societal controversies Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

More information: www. nusap. net Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004

References: Funtowicz, S. O. and J. R. Ravetz, Uncertainty and Quality in Science for Policy. Kluwer, 229 pp. , Dordrecht, 1990. James Risbey, J. P. van der Sluijs and J. Ravetz, Quality assistance in environmental modelling, Environmental Modelling and Assessment, forthcoming Van der Sluijs, Jeroen P. , A way out of the credibility crisis of models used in integrated environmental assessment, Futures, Vol. 34, 2002, pp. 133 -146. Jeroen van der Sluijs, Matthieu Craye, Silvio Funtowicz, Penny Kloprogge, Jerry Ravetz, and James Risbey Combining Quantitative and Qualitative Measures of Uncertainty in Model based Environmental Assessment: the NUSAP System, Risk Analysis, forthcoming Walker, W. , Harremoës, P. , Rotmans, J. , Van der Sluijs, J. , Van Asselt, M. , Jansen, P. , Krayer von Krauss, M. P. , 2003. Defining Uncertainty: A Conceptual Basis for Uncertainty Management in Model-Based Decision Support. Journal of Integrated Assessment, 4 (1) 5 -17. Weiss, C. , Scientific Uncertainty and Science Based Precaution. International Environmental Agreements: Politics, Law and Economics 3, 2003, p. 137 -166 Weiss, C. , Expressing Scientific Uncertainty. Law, probability and risk, 2003, 2, p. 25 -46 The four volumes of the RIVM uncertainty guidance: J. P. van der Sluijs, J. S. Risbey, P. Kloprogge, J. R. Ravetz, S. O. Funtowicz, S. Corral Quintana, Guimarães Pereira, B. De Marchi, A. C. Petersen, P. H. M. Janssen, R. Hoppe, and S. W. F. Huijs. RIVM/MNP Guidance for Uncertainty Assessment and Communication: Detailed Guidance Utrecht University & RIVM, 2003 (available from: www. nusap. net). A. C. Petersen, P. H. M. Janssen, J. P. van der Sluijs, J. S. Risbey, J. R. Ravetz RIVM/MNP Guidance for Uncertainty Assessment and Communication: Mini-Checklist & Quickscan Questionairre. . RIVM/MNP; ISBN 90 -6960 -105 -1, 2003 (available from: www. nusap. net). P. H. M. Janssen, A. C. Petersen, J. P. van der Sluijs, J. S. Risbey, J. R. Ravetz. RIVM/MNP Guidance for Uncertainty Assessment and Communication: Quickscan Hints & Actions List. RIVM/MNP, ISBN 90 -6960 -105 -2, 2003 (available from: www. nusap. net). J. P. van der Sluijs, P. H. M. Janssen, A. C. Petersen, P. Kloprogge, J. S. Risbey, W. Tuinstra, M. B. A. van Asselt, J. R. Ravetz, RIVM/MNP Guidance for Uncertainty Assessment and Communication: Tool Catalogue for Uncertainty Assessment Utrecht University & RIVM, , 2004 (available from: www. nusap. net). Most refs available on request as pdf file, just mail me at j. p. vandersluijs @chem. uu. nl Copernicus Institute Uncertainty Assessment - Flood Risk Management, Nottingham, 6 Oct 2004