Probability Forecasts Verification using CPT Run CPT in
Probability Forecasts Verification using CPT
Run CPT in retroactive mode to create the forecasts data: Action -> Calculate -> Retroactive
CPT will calculate a training period from half of data: Length of initial train period: -> 18 (1981 -1998) next upcoming forecast will be 1999, 2000, 2001, … by update interval -> 1
You probably set the initial period for last 5 -year forecasts: Length of initial train period: -> 30 (1981 -2010) next upcoming forecast will be 2011, 2012, …, 2015 by update interval -> 1
Running in progress will show the info. of calculation: Length of initial train period: -> 30 so the training/forecast are: 1 st (1981 -2010)/2011, 2 nd (1981 -2011)/2012, 3 rd (1981/2012)/2013, …
After the calculating has done and to display skill map: Tools -> Validation -> Retroactive -> Skill Maps
After the calculating has done and to verification: Tools -> Verification -> Attributes Diagrams -> ROC Diagrams -> Scores and etc.
Relative Operating Characteristic (ROC) Complete all plots of the ROC curve and compute the ROC area ROC curves 1 0, 9 0, 8 0, 7 Hit rate 0, 6 0, 5 0, 4 B = 0. 4925 N = 0. 6380 0, 3 A = 0. 6497 0, 2 no skill 0, 1 0 0, 2 0, 4 0, 6 False-alarm rate 0, 8 1, 0
ROC curves – curve shapes
Interpretation of ROC curves (1) i. e. below-normal rainfall category (dry / non-dry years) 1 The forecasts are perfectly by discriminating dry years from non-dry years. 0, 9 0, 8 0, 7 Hit Rate 0, 6 The forecast probabilities for all the dry years are higher than the forecast probabilities for all the non-dry years. 0, 5 0, 4 0, 3 0, 2 0, 1 0 0 0, 1 0, 2 0, 3 0, 4 0, 5 0, 6 0, 7 False-alarm Rate 0, 8 0, 9 1
Interpretation of ROC curves (2) i. e. below-normal rainfall category (dry / non-dry years) 1 The forecasts are no better by discriminating dry years from non-dry years than by guessing. 0, 9 0, 8 0, 7 Hit Rate 0, 6 There are just two possibilities, guessing has 50% chance of discriminating correctly. 0, 5 0, 4 0, 3 0, 2 0, 1 0 0 0, 1 0, 2 0, 3 0, 4 0, 5 0, 6 0, 7 False-alarm Rate 0, 8 0, 9 1
Interpretation of ROC curves (3) i. e. below-normal rainfall category (dry / non-dry years) 1 The forecasts are very well at discriminating dry years from non-dry years. (ROC area is about 95%) 0, 9 0, 8 0, 7 Hit Rate 0, 6 0, 5 0, 4 0, 3 0, 2 0, 1 0 0 0, 1 0, 2 0, 3 0, 4 0, 5 0, 6 0, 7 False-alarm Rate 0, 8 0, 9 1
Interpretation of ROC curves (4) i. e. below-normal rainfall category (dry / non-dry years) 1 The forecasts are good by discriminating dry years from non-dry years. (ROC area about 75%) 0, 9 0, 8 0, 7 Hit Rate 0, 6 0, 5 0, 4 0, 3 0, 2 0, 1 0 0 0, 1 0, 2 0, 3 0, 4 0, 5 0, 6 0, 7 False-alarm Rate 0, 8 0, 9 1
Interpretation of ROC curves (5) i. e. below-normal rainfall category (dry / non-dry years) 1 The forecasts can discriminate dry years from non-dry years , but most of the time they indicate the incorrect year as dry. 0, 9 0, 8 0, 7 Hit Rate 0, 6 0, 5 0, 4 0, 3 0, 2 0, 1 0 0 0, 1 0, 2 0, 3 0, 4 0, 5 0, 6 0, 7 False-alarm Rate 0, 8 0, 9 1
Interpretation of ROC curves (6) i. e. below-normal rainfall category (dry / non-dry years) 1 0, 9 0, 8 0, 7 Hit Rate 0, 6 0, 5 0, 4 0, 3 0, 2 0, 1 0 0 0, 1 0, 2 0, 3 0, 4 0, 5 0, 6 0, 7 False-alarm Rate 0, 8 0, 9 1 • The forecasts are fairly by discriminating dry years from non-dry years, better than just by guessing (ROC area about 60%). • The forecasts are performing well at identifying dry year occurs. The forecast probabilities are high when a dry year occurs. • The forecasts are performing poorly at identifying non-dry years.
Interpretation of ROC curves (7) i. e. below-normal rainfall category (dry / non-dry years) 1 0, 9 0, 8 0, 7 Hit Rate 0, 6 0, 5 0, 4 0, 3 0, 2 0, 1 0 0 0, 1 0, 2 0, 3 0, 4 0, 5 0, 6 0, 7 False-alarm Rate 0, 8 0, 9 1 • The forecasts are fairly by discriminating dry years from non-dry years, better than just by guessing (ROC area about 60%). • The forecasts are performing well at identifying non-dry years. The forecast probabilities are low when a dry year does not occur. • The forecasts are performing poorly at identifying dry years.
Interpretation of ROC curves (8) i. e. below-normal rainfall category (dry / non-dry years) 1 0, 9 0, 8 0, 7 Hit Rate 0, 6 0, 5 0, 4 0, 3 0, 2 0, 1 0 0 0, 1 0, 2 0, 3 0, 4 0, 5 0, 6 0, 7 False-alarm Rate 0, 8 0, 9 1 • The forecasts are fairly by discriminating dry years from nondry years, better than just by guessing (ROC area about 60%). • The forecasts are performing well at identifying dry years. The forecast probabilities are high when a dry year occurs. • The forecasts are performing well at identifying non-dry years. The forecast probabilities are low when a dry year does not occur. • The forecasts are performing poorly when the probabilities are close to climatology.
Reliability Diagrams with probability bin 10% The Excel on Reliability (2) will automatically compute the reliability diagrams using bin size 10%.
Reliability Diagrams with probability bin 5% The Excel on Reliability (3) will automatically compute the reliability diagrams using bin size 5%.
Interpretation of reliability diagrams
Interpretation of reliability diagrams
Interpretation of sharpness Sharpness histogram: the frequency of forecasts in each probability bin shows the sharpness of the forecast. Over-dispersed 40% Biased 30% 20% 10% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% 30% 40% Calibrated 30% 20% 10% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% 40% Under-dispersed
- Slides: 22