Verification of probabilistic forecasts Simon Mason simoniri columbia

Verification of probabilistic forecasts Simon Mason simon@iri. columbia. edu Seasonal Forecasting Using the Climate Predictability Tool

Who is this? Park Jae-sang, aka “Psy” 2 Seasonal Forecasting Using the Climate Predictability Tool

Probabilistic Forecasts Why do we issue forecasts probabilistically? • We cannot be certain what is going to happen • The probabilities try to give an indication of how confident we are that the specified outcome will occur. Odds are an alternative to probabilities; they indicate how much more (or less) likely we think the specified outcome will occur than not occur. If the probabilities are “correct”, or, equivalently, the odds are fair, then we can calculate how frequently the specified outcome should occur. . . 3 Seasonal Forecasting Using the Climate Predictability Tool

What makes a good forecast? Forecast: The odds of Prince William and Prince Kate’s baby being called Psy were 5000 to 1 against. Verification: Were those odds fair? 4 Seasonal Forecasting Using the Climate Predictability Tool

Reliability Give 90% confidence limits for the mean annual rainfall in New Delhi. 706 mm (27. 8 inches). How many of these confidence intervals contain the observed annual rainfall? Differences between how many of these confidence intervals do contain the observed annual rainfall and how many should contain the observation are the subject of questions about reliability? 5 Seasonal Forecasting Using the Climate Predictability Tool

Forecast “goodness” What makes a “good” forecast? 1. Consistency 2. Quality 3. Value Murphy AH 1993; Wea. Forecasting 8, 281 6 Seasonal Forecasting Using the Climate Predictability Tool

Consistent forecasts? All regions have highest probability on the normal category. Did we genuinely think that normal was the most likely category everywhere, or did we think it was the safest forecast everywhere? 70 – 80% of all the African RCOF forecasts have highest probability on normal. Are we really forecasting what we think, or are we playing safe? 7 Seasonal Forecasting Using the Climate Predictability Tool

Unconditional bias • Are probabilities consistently too high or too low? 8 Seasonal Forecasting Using the Climate Predictability Tool

Forecast “goodness” What makes a “good” forecast? 1. Consistency 2. Quality 3. Value Murphy AH 1993; Wea. Forecasting 8, 281 9 Seasonal Forecasting Using the Climate Predictability Tool

What is “skill”? 10 Seasonal Forecasting Using the Climate Predictability Tool 10

Skill Is one set of forecasts better than another? • Skillful forecasts are not necessarily good; both sets of forecasts may be really bad. • Unskillful forecasts are not necessarily bad: both sets of forecasts may be really good. “Skill” is poorly defined. What do we mean by “better”? 11 Seasonal Forecasting Using the Climate Predictability Tool 11

Skill Imagine a set of forecasts that indicates probabilities of rainfall (which has a climatological probability of 30%): 01 May 60% 02 May 60% 03 May 60% 04 May 60% 05 May 60% 06 May 10% 07 May 10% 08 May 10% 09 May 10% 10 May 10% 12 Suppose that rainfall occurs on 40% of the green forecasts, and 20% of the brown. The forecasts correctly indicate times with increased and decreased chances of rainfall, but do so overconfidently. The Brier skill score is -7%. Seasonal Forecasting Using the Climate Predictability Tool 12

Reliability • When we say 60% chance of above-normal, we expect a forecast of above-normal to be correct 60% of the time. • If we take all our forecasts for a location when we said 60% chance of above-normal, 60% of them (not more or less) should be above-normal. 13 Seasonal Forecasting Using the Climate Predictability Tool

Resolution Does the outcome change when the forecast changes? 01 May 60% 02 May 60% 03 May 60% 04 May 60% 05 May 60% 06 May 10% 07 May 10% 08 May 10% 09 May 10% 10 May 10% 14 When the forecast is 10% rain occurs 20% of the time. When the forecast is 60% rain occurs 40% of the time. Rain becomes more frequent when the forecast probability increases – there is resolution. Seasonal Forecasting Using the Climate Predictability Tool 14

Resolution • Does the outcome change when the forecast changes? • Example: does above-normal rainfall become more frequent when its probability increases? • Resolution is the crucial attribute of a good forecast. If the outcome differs depending on the forecast then the forecasts have useful information. If the outcome is the same regardless of the forecaster can be ignored. 15 Seasonal Forecasting Using the Climate Predictability Tool

Discrimination Does the forecast change when the outcome changes? 01 May 60% 02 May 60% 03 May 60% 04 May 60% 05 May 60% 06 May 10% 07 May 10% 08 May 10% 09 May 10% 10 May 10% 16 When rain occurs the average forecast probability is 43%. When it is dry the average forecast probability is 31%. The forecast probability for rain is higher when it does rain – there is some discrimination. Seasonal Forecasting Using the Climate Predictability Tool 16

Discrimination • Does the forecast differ when the outcome differs? • Example: is the probability on above-normal rainfall higher when above-normal rainfall occurs compared to when rainfall is normal or below-normal? • Discrimination is an alternative perspective to resolution. If the forecast differs given different outcomes then the forecasts have useful information. If the forecast is the same regardless of the outcome the forecaster can be ignored. 17 Seasonal Forecasting Using the Climate Predictability Tool

What makes a “good” probabilistic forecast? Reliability Sharpness Resolution Discrimination 18 the event occurs as frequently as implied by the forecasts frequently have probabilities that differ from climatology considerably the outcome differs when the forecast differs the forecasts differ when the outcome differs Seasonal Forecasting Using the Climate Predictability Tool 18

Verification in CPT • In CPT, “verification” relates to the assessment of probabilistic predictions: – As retroactive predictions in CCA, PCR, MLR or GCM; – As inputs in PFV. 19 Seasonal Forecasting Using the Climate Predictability Tool

Retroactive forecasting 1981 1982 Training period (1961 – 1980) Predict 1981 Training period (1961 – 1981) Omit 1982+ Predict 1982 Training period (1961 – 1982) 1983 Omit 1983+ Predict 1983 Training period (1961 – 1983) 1984 Training period (1961 – 1984) 1985 Omit 1984+ Predict 1984 Omit 1985+ Predict 1985 Given data for 1961 – date, it is possible to calculate a retroactive set of probabilistic forecasts. CPT will use an initial training period to cross -validate a model and make predictions for the subsequent year(s), then update the training period and predict additional years, repeating until all possible years have been predicted. 20 Seasonal Forecasting Using the Climate Predictability Tool

Probabilistic forecast input files INDEX and STATION files cpt: ncats (the number of categories; must be 3) cpt: C (start with category 1, i. e. below-normal, then repeat for category 2, i. e. normal; complete for all 3 categories, but make sure the probabilities add to 100) Date (the period for which the forecast applies, not the date the forecast was made) cpt: clim_prob (indicate the climatological probability of each category) 21 Seasonal Forecasting Using the Climate Predictability Tool

Verification of probabilistic forecasts Attributes Diagrams: graphs reliability, resolution, sharpness ROC Diagrams: graphs showing discrimination Scores: a table of scores for probabilistic forecasts Skill Maps: maps of scores for probabilistic forecasts Tendency Diagram: graphs showing unconditional biases Ranked Hits Diagram: graphs showing frequencies of observed categories having the highest probability Weather Roulette: graphs showing estimates of forecast value 22 Seasonal Forecasting Using the Climate Predictability Tool

Attributes diagrams The histograms show the sharpness. The vertical and horizontal lines show the observed climatology and indicate the forecast bias. The diagonal lines show reliability and “skill”. The coloured line shows the reliability and resolution of the forecasts. The dashed line shows a smoothed fit. 23 Seasonal Forecasting Using the Climate Predictability Tool

ROC diagrams Retroactive forecasts of MAM 1986 – 2010 Thailand rainfall using February Pacific SSTs 24 ROC areas: do we issue a higher probability when the category occurs? Graph bottom left: when the probabilities are high, does the category occur? Graph top right: when the probabilities are low, does the category not occur? Seasonal Forecasting Using the Climate Predictability Tool

Tendency diagrams Retroactive forecasts of MAM 1986 – 2010 Thailand rainfall using February Pacific SSTs. Shift towards abovenormal was successfully predicted. 25 Seasonal Forecasting Using the Climate Predictability Tool

Ranked Hits diagrams highest probability second highest probability lowest probability Retroactive forecasts of MAM 1986 – 2010 Thailand rainfall using February Pacific SSTs. Category with highest probability is occurring most frequently. 26 Seasonal Forecasting Using the Climate Predictability Tool

Probabilistic scores Scores per category Brier score: probability (assuming insquared error mean that the probability should be 100% if the category occurs and 0% if it does not occur) 27 Seasonal Forecasting Using the Climate Predictability Tool

Probabilistic scores Overall scores Ranked prob score: mean squared error in cumulative probabilities RPSS: % improvement over RPS using climatology forecasts (often pessimistic because of strict requirement for reliability) 2 AFC score: probability successfully of discriminating the wetter or warmer category in Resolution increase slope: % forecast probability Effective interest: % return given fair odds Linear prob score: average probability on the category that occurs 28 Seasonal Forecasting Using the Climate Predictability Tool

Forecast “goodness” What makes a “good” forecast? 1. Consistency 2. Quality 3. Value Murphy AH 1993; Wea. Forecasting 8, 281 29 Seasonal Forecasting Using the Climate Predictability Tool

Weather roulette – profits diagram Given fair odds: profit = 1 ÷ odds Multiply the investment by the profit (or loss) to indicate how much money would be made (or lost). Average over all locations. 30 Seasonal Forecasting Using the Climate Predictability Tool

Weather roulette – cumulative profits diagram Multiply the initial investment by the profit (or loss) carried over each year to indicate how much money would be made (or lost). 31 Seasonal Forecasting Using the Climate Predictability Tool

Weather roulette – effective interest rate diagram Multiply the initial investment by the profit (or loss) carried over each year, and calculate the effective interest rate. 32 Seasonal Forecasting Using the Climate Predictability Tool

Exercises • Generate some retroactive forecasts. • How do the retroactive validation results compare to the cross-validated? • Do the forecasts perform as well as you might have expected given the cross-validated skill measures? • Explore the various verification options. 33 Seasonal Forecasting Using the Climate Predictability Tool

web: iri. columbia. edu/cpt/ @climatesociety …/climatesociety CPT Help Desk cpt@iri. columbia. edu