Verification in CPT Simon J Mason simoniri columbia
Verification in CPT Simon J. Mason simon@iri. columbia. edu International Research Institute for Climate and Society The Earth Institute of Columbia University CIMH Workshop on the Climate Predictability Tool Bridgetown, Barbados, July / August 2013
Verification • In CPT, “verification” relates to the assessment of probabilistic predictions: – As retroactive predictions in CCA, PCR, MLR or GCM; – As inputs in PFV.
Retroactive forecasting Given data for 1951 -2000, it is possible to calculate a retroactive set of probabilistic forecasts. CPT will use an initial training period to cross-validate a model and make predictions for the subsequent year(s), then update the training period and predict additional years, repeating until all possible years have been predicted.
Station Data INDEX and STATION files cpt: ncats (the number of categories; must be 3) cpt: C (start with category 1, i. e. below-normal, then repeat for category 2, i. e. normal; complete for all 3 categories, but make sure the probabilities add to 100) Date (the period for which the forecast applies, not the date the forecast was made) cpt: clim_prob (indicate the climatological probability of each category)
Probabilistic Forecast Verification (PFV)
Probabilistic Forecast Verification (PFV) Using Tools ~ Climatological period, set the first and last year of the period used for defining above- and below-normal. Make sure that the box to allow the climatological period to extend beyond the training period is checked.
Probabilistic Forecast Verification (PFV) To see the results go to the menu Tools ~ Verification: Attributes Diagrams: one for each category, and one for all categories combined ROC Diagrams: one curve for each category Scores: a table of scores for probabilistic forecasts Skill Maps: maps of scores for probabilistic forecasts Tendency Diagram: graphs showing unconditional biases Ranked Hits Diagram: graphs showing frequencies of observed categories having the highest probability Weather Roulette: graphs showing estimates of forecast value
Attributes diagrams The histograms show the sharpness. The vertical and horizontal lines show the observed climatology and indicate the forecast bias. The diagonal lines show reliability and “skill”. The coloured line shows the reliability of the forecasts. The dashed line shows a smoothed fit.
ROC diagrams
Tendency diagrams
Ranked Hits diagrams highest probability second highest probability lowest probability
Weather roulette – profits diagram Multiply the investment by the profit (or loss) to indicate how much money would be made (or lost) in each year.
Weather roulette – cumulative profits diagram Multiply the initial investment by the profit (or loss) carried over each year to indicate how much money would be made (or lost).
Weather roulette – effective interest rate diagram Multiply the initial investment by the profit (or loss) carried over each year, and calculate the effective interest rate.
Exercises • Verify the IRI JAS rainfall forecasts for the Caribbean.
- Slides: 15