Can we distinguish wet years from dry years

Can we distinguish wet years from dry years? Simon Mason simon@iri. columbia. edu Seasonal Forecasting Using the Climate Predictability Tool Bangkok, Thailand, 12 – 16 January 2015

The ROC • The ROC answers the question: Can the forecasts distinguish an event from a non-event? • Are we more confident it will be dry when it is dry compared to when it is not? – Do we forecast less rain when it is dry compared to when it is not dry? – Do we issue a higher forecast probability for below-normal when it is below-normal compared to when it is not? 2 Seasonal Forecasting Using the Climate Predictability Tool

ROC • Retroactive forecasts of MAM rainfall for Thailand. • Which year are you most confident is a dry year? 3 Seasonal Forecasting Using the Climate Predictability Tool

ROC • The most sensible strategy would be to list the years in order of increasing forecast rainfall. • If the forecasts are good, the “dry” years should be at the top of the list. 4 Seasonal Forecasting Using the Climate Predictability Tool

ROC For the first guess: Repeat for all forecasts. 5 Seasonal Forecasting Using the Climate Predictability Tool

ROC 6 Seasonal Forecasting Using the Climate Predictability Tool

ROC Plot the correct scores (hit -rate) against the incorrect (false-alarm rate) scores. We want the correct scores to be larger than the incorrect scores, i. e. , for the graph to be above the diagonal. 7 Seasonal Forecasting Using the Climate Predictability Tool

ROC Cross-validation of MAM rainfall for Thailand, using Feb NIÑO 4. 8 What is the year we are most confident is “below”? Was it “below”? If so score a hit; if not score a false-alarm. What is the year we are next most confident is “below”? Seasonal Forecasting Using the Climate Predictability Tool

ROC Cross-validation of MAM rainfall for Thailand, using Feb NIÑO 4. 9 What is the year we are most confident is not “below”? Was it “below”? If not we scored a false-alarm; if so we scored a hit. What is the year we are next most confident is not “below”? Seasonal Forecasting Using the Climate Predictability Tool

ROC The bottom left indicates whether the forecasts with strong indications of dry (or wet) are good. Can they indicate that an event will occur? The top right indicates whether the forecasts with strong indications of not dry (or not wet) are good. Can they indicate that an event will not occur? 10 Seasonal Forecasting Using the Climate Predictability Tool

Relative Operating Characteristics 11 Seasonal Forecasting Using the Climate Predictability Tool

Two-Alternative Forced Choice Test In which of these two Januaries did El Niño occur (Niño 3. 4 index >27°C)? What is the probability of getting the answer correct? 50% (assuming that you do not have inside information about ENSO). 12 Seasonal Forecasting Using the Climate Predictability Tool

Two-Alternative Forced Choice Test In which of these two Januaries did El Niño occur (Niño 3. 4 index >27°C)? What is the probability of getting the answer correct? That depends on whether we can believe the forecasts. Select the forecast with the highest temperature. 13 Seasonal Forecasting Using the Climate Predictability Tool

Two-Alternative Forced Choice Test We can ask the same question if the forecasts are probabilistic: In which of these two Januaries did El Niño occur (Niño 3. 4 index >27°C)? What is the probability of getting the answer correct? That depends on whether we can believe the forecasts. Select the forecast with the higher probability. 14 Seasonal Forecasting Using the Climate Predictability Tool

Two-Alternative Forced Choice Test • Retroactive forecasts of MAM rainfall for Thailand. • How well do the forecasts distinguish “dry” years (driest 20%) from other years? • Do we forecast less rain when it is dry compared to other years? 15 Seasonal Forecasting Using the Climate Predictability Tool

Two-Alternative Forced Choice Test • It is easier to calculate by sorting the forecasts so the driest forecast are at the top. We can then count how many of the non-dry years are lower in the table than the dry years. • For 2010: 14 of the 15 non-dry years have wetter forecasts. • For 1998: 14 of 15 • For 1995: 14 of 15 • For 2005: 14 of 15 • For 1992: 12 of 15 • In total: 68 of 75 ≈ 91%. 16 Seasonal Forecasting Using the Climate Predictability Tool

Two-Alternative Forced Choice Test • If the forecasts could perfectly discriminate the dry years, the forecasts would be drier than for all the non-dry years, and the dry years would be listed at the top of the table. • If the forecasts could not discriminate the dry years at all, they would be randomly distributed through the table, and there would be a 50% chance of the forecast being drier than on a non-dry year. 17 Seasonal Forecasting Using the Climate Predictability Tool

ROC The area beneath the red curve, 0. 91, gives us the probability that we will successfully discriminate a “dry” year from a nondry year. The area beneath the blue curve, 0. 85, gives us the probability that we will successfully discriminate a “wet” year from a nonwet year. 18 Seasonal Forecasting Using the Climate Predictability Tool

ROC score • The ROC score indicates how successfully we can distinguish an event (e. g. , “below-normal”) from a non-event (“normal” or “above -normal”). – How often do we predict less rain when we observe “belownormal” compared to when we observe “normal” or “abovenormal”? • But the predictions could be in probabilities: – How often do we predict a higher chance of below-normal when we observe “below-normal” compared to when we observe “normal” or “above-normal”? • … or in categories: – How often do we predict a drier category when we observe “below-normal” compared to when we observe “normal” or “above-normal”? 19 Seasonal Forecasting Using the Climate Predictability Tool

Summary • The ROC answers the question: Can the forecasts distinguish an event from a non-event? • The graph can help identify conditional skill (e. g. , can we forecast wet conditions better than dry conditions? ) • It can be used to verify deterministic (discrete and continuous) and probabilistic forecasts. 20 Seasonal Forecasting Using the Climate Predictability Tool

Exercises • Diagnose the quality of your forecast models by analysing the ROC graphs. 21 Seasonal Forecasting Using the Climate Predictability Tool

web: iri. columbia. edu/cpt/ @climatesociety …/climatesociety CPT Help Desk cpt@iri. columbia. edu