Forecast Verification Assessing the Quality of Forecasts Simon

Forecast Verification – Assessing the Quality of Forecasts Simon Mason simon@iri. columbia. edu Cari. COF 2019 -20 Dry Season – Seasonal Forecast Training Workshop Port of Spain, Trinidad and Tobago, 25 – 26 November 2019

Probabilistic Forecast Verification (PFV) 2 Seasonal Forecast Training Workshop 2019 Nov 25 -26

Probabilistic forecast input files INDEX and STATION files cpt: ncats (the number of categories; must be 3) cpt: C (start with category 1, i. e. below-normal, then repeat for category 2, i. e. normal; complete for all 3 categories, but make sure the probabilities add to 100) Date (the period for which the forecast applies, not the date the forecast was made) cpt: clim_prob (indicate the climatological probability of each category) 3 Seasonal Forecast Training Workshop 2019 Nov 25 -26

Climatological Period Options ~ Climatological Period 4 Seasonal Forecast Training Workshop 2019 Nov 25 -26

Missing values Options ~ Data ~ Transform Y Data? Options ~ Data ~ Missing Values Actions ~ Calculate ~ Verify 5 Seasonal Forecast Training Workshop 2019 Nov 25 -26

Verification of probabilistic forecasts Attributes Diagrams: graphs reliability, resolution, sharpness ROC Diagrams: graphs showing discrimination Scores: a table of scores for probabilistic forecasts Skill Maps: maps of scores for probabilistic forecasts Tendency Diagram: graphs showing unconditional biases Ranked Hits Diagram: graphs showing frequencies of observed categories having the highest probability Weather Roulette: graphs showing estimates of forecast value 6 Seasonal Forecast Training Workshop 2019 Nov 25 -26

Ranked Hits diagrams highest probability second highest probability lowest probability IRI forecasts of Caribbean JAS rainfall 1998 – 2015. Category with highest probability is occurring more frequently with time. 7 Seasonal Forecast Training Workshop 2019 Nov 25 -26

ROC diagrams IRI forecasts of JAS 1998 – 2015 Caribbean rainfall 8 ROC areas: do we issue a higher probability when the category occurs? Graph bottom left: when the probabilities are high, does the category occur? Graph top right: when the probabilities are low, does the category not occur? Seasonal Forecast Training Workshop 2019 Nov 25 -26

Weather roulette – profits diagram Given fair odds: profit = 1 ÷ odds Multiply the investment by the profit (or loss) to indicate how much money would be made (or lost). Average over all locations. 9 Seasonal Forecast Training Workshop 2019 Nov 25 -26

Weather roulette – cumulative profits diagram Multiply the initial investment by the profit (or loss) carried over each year to indicate how much money would be made (or lost). 10 Seasonal Forecast Training Workshop 2019 Nov 25 -26

web: iri. columbia. edu/cpt/ @climatesociety …/climatesociety CPT Help Desk cpt@iri. columbia. edu