Communicating uncertainty in seasonal climate forecasts to stakeholders

Communicating uncertainty in seasonal climate forecasts to stakeholders Andrea L. Taylor 1, 2 Suraje Dessai 2 Wändi Bruine de Bruin 2, 3 1 Sustainability Research Institute, University of Leeds 2 Centre for Decision Research, University of Leeds 3 Department for Engineering and Public Policy, Carnegie Mellon University

Paper presented at the Centre for Climate and Energy Decision Making (CEDM) funded by the US National Science Foundation. We thank CEDM and the US National Science Foundation for the opportunity to share this work with the wider CEDM community. EUPORIAS European Provision Of Regional Impacts Assessments on Seasonal and Decadal Timescales This project has received funding from the European Union’s Seventh Framework Programme for research, technological development and demonstration under grant agreement no 308291.

Dr Andrea Taylor Prof Suraje Dessai Prof Wӓndi Bruine de Bruin Taylor, A. L. , Dessai, S. , & de Bruin, W. B. (2015). Communicating uncertainty in seasonal and interannual climate forecasts in Europe. Phil. Trans. R. Soc. A, 373(2055), 20140454.

Seasonal Forecasts: Sources of Uncertainty • Uncertainty in seasonal climate predictions comes from multiple sources. • Forecasts are probabilistic. • No Forecast perfectly captures the world’s climate system. Example of ensemble seasonal temperature forecast (ECMWF – provided by Jean Pierre Ceron, Meteo France)

Seasonal Forecasts: The challenge of communicating uncertainty • Forecast distribution used to obtain measures of ‘likelihood’. • Forecast performance measured by comparing predictions to observations. – Reliability: How much better does the forecast perform than chance? – Skill: To what extent does the forecast perform better (or worse) than historical data at predicting future conditions?

Different ways of representing seasonal forecasts…. Top left example wind forecast (IC 3). Top middle seasonal temperature forecast (Meteo. Swiss), Top right precipitation forecast (Met Office). Bottom left Temperature anomaly forecast (ECMWF) … but relatively little testing with users

Reviewing the broader literature on communicating uncertainty • Interpretation and use of uncertain information may be influenced by a number of factors: – – – ‘ambiguity aversion’ (e. g. Ellsberg, 1961) institutional protocol (e. g. Demeritt et al. , 2010; Ramos et al. , 2010) technical and statistical expertise (e. g. Gregory et al. , 2012) cognitive biases (e. g. Kahneman, 2011) use of colour in visualisations (e. g. Kaye et al. , 2012) • Tradeoffs between detail, robustness and understandability (Stephenson et al. , 2012)

Communicating confidence and uncertainty in seasonal forecasts: Surveying user needs Sample 50 respondents from stakeholder organisations started the survey. 45 provided full completions. Use weather forecasts Use seasonal forecasts Use interannual forecast n = 45 n = 33 n = 18

Communicating confidence and uncertainty in seasonal forecasts: Surveying user needs Key questions 1. How accessible, understandable and useful do users perceived seasonal forecasts to be? 2. What information about uncertainty in seasonal forecasts do users currently receive, and what would they like to receive? 3. When it comes to representing likelihood what are participants’ preferences?

How accessible, understandable and useful do users perceived forecasts to be? 5 4, 5 4 3, 5 3 2, 5 Weather forecast 2 Seasonal forecast Interannual forecast 1, 5 1 0, 5 0 "Easy to find" "Easy to understand" "Useful" Mean ratings of accessibility, understandability and usefulness amongst survey respondents.

What information about uncertainty in seasonal forecasts do users currently receive, and what would they like to receive? Proportion of current users (%) 60 50 40 30 Yes No 20 No, but we would like to Don't know 10 0 Ranges of values Confidence intervals Verbal descriptions of likelihood Raw data Probability distributions Information Indicators of about how signal possible well earlier strength sources of forecasts error have matched observed climate

When it comes to representing likelihood what are respondents’ preferences?

When it comes to representing uncertainty what are respondents’ preferences? Mean ratings of preference and familiarity and correlation between preference, familiarity and statistical comfort Correlations (Spearman’s ρ) Map Fan chart Error bar Bar graph: Distribution Pie chart Spaghetti plot Bar graph: Tercile Preference Mean (SD) 3. 9 (0. 7) 3. 7 (0. 7) 3. 4 (0. 8) 3. 2 (0. 9) 3. 1 (1. 0) 3. 0 (0. 9) Familiarity Mean (SD) 3. 3(1. 1) 3. 4(1. 1) 3. 5(1. 1) 3. 0(1. 1) 2. 9(1. 1) 3. 0(1. 2) 2. 5(1. 1) Preference with Familiarity ρ. 60***. 52***. 74***. 52***. 71***. 69***. 76*** Preference with statistical comfort ρ -. 22. 24. 38**. 26† -. 21. 02. 08 Familiarity with statistical comfort ρ -. 06. 18. 34*. 19 -. 17. 13. 26† †Marginally significant at p ≤. 10 *Significant at p ≤. 05 **Significant at p ≤. 01 ***Significant at p ≤. 001

Conclusions and practical implications • Seasonal predictions are not judged to be as understandable as more familiar forecasts. • Information about forecast performance (skill) is not being clearly communicated to many current users. • Preference for different types of visualisation influenced by both familiarity and statistical expertise.

…. Follow up stakeholder Decision Lab • Preference for particular communication formats related to perceived familiarity, but not objective understanding. • Evidence that perceived familiarity can sometimes suppress objective understanding. • Even when forecasts do not perform well enough to provide useful information, “likelihoods” still affect judgement. • Highlights the importance of testing user understanding. • Should forecasts “without skill” be provided at all?

Centre for Decision Research Left to right: Prof Barbara Summers, Dr Andrea Taylor, Dr Yasmina Okan, Dr Simon Mc. Nair, Prof Wӓndi Bruine de Bruin, Dr Richard Hodgett, Prof Alan Pearman, Dr Gulbanu Kaplan

Climate Change Adaptation Group Left to right: Prof Suraje Dessai, Dr Marta Bruno Soares, Dr Susanne Lorenz, Dr Paola Sakai, Dr Andrea Taylor.