Probabilistic Forecast Verification PFV view option Simon J

Probabilistic Forecast Verification (PFV) view option Simon J. Mason cpt@iri. columbia. edu International Research Institute for Climate and Society The Earth Institute of Columbia University

OVERVIEW The Probabilistic Forecast Verification (PFV) view option within the Climate Predictability Tool (CPT) uses CPT’s retroactive verification tools to assess the quality of past probabilistic forecasts, such as those issued through the Regional Climate Outlook Forums. The PFV view option works similarly to the other options (CCA, PCR, etc. ), but does not construct a statistical prediction model since the inputs are the historical forecasts. The Y input file is the same as for the other view options: a historical set of predictands. The X input file, instead of being a historical set of predictors, is a historical set of probabilistic forecasts …

SELECTING PFV Choose PFV from the View menu option

INPUT DATASETS Read in the X variables as a set of historical probabilistic forecasts.

CPT INPUT FILE FORMATS INDEX and STATION files cpt: ncats (the number of categories; must be 3) cpt: C (start with category 1, i. e. below-normal, then repeat for category 2, i. e. normal; complete for all 3 categories, but make sure the probabilities add to 100) Date (the period for which the forecast applies, not the date the forecast was made) cpt: clim_prob (indicate the climatological probability of each category)

CPT INPUT FILE FORMATS GRIDDED files This file-type contains : cpt: ncats (the number of categories; must be 3) cpt: C (start with category 1, i. e. below-normal, then repeat for category 2, i. e. normal; complete for all 3 categories before proceeding to any other fields or the next time step)

CPT INPUT FILE FORMATS In Excel the file should be saved as “Text (Tab delimited) (*. txt)” Afterwards, check that Excel has not changed the date format in station or index files.

INPUT DATASETS Read in the Y variables as for the other view options. The dataset should include: • the climatological period, and • the verification period (the period for which historical forecasts are to be assessed).

SETTING THE CLIMATOLOGICAL PERIOD Using Tools ~ Climatological period, set the first and last year of the period used for defining above- and below-normal. Make sure that the box to allow the climatological period to extend beyond the training period is checked.

SETTING THE VERIFICATION PERIOD In PFV, the training period becomes the verification period. Make sure that the start dates for the X and Y files match.

SETTING THE VERIFICATION PERIOD Set the length of the training period to the number of forecasts to be verified. The cross-validation window is not used, so that can be ignored (but in version 12, it is best to set this to 1).

SETTING THE VERIFICATION PERIOD You may need to adjust the missing value settings: Options ~ Data ~ Missing Values Consider setting the %s of missing values and missing stations or gridpoints high so that locations with only a few forecasts are still verified, and that years are included even if there are lots of missing values.

RUNNING CPT Then you can perform the verification: Actions ~ Calculate ~ Verify

RESULTS To see the results go to the menu Tools ~ Verification: Attributes Diagrams: one for each category, and one for all categories combined ROC Diagrams: one curve for each category Scores: a table of scores for probabilistic forecasts Skill Maps: maps of scores for probabilistic forecasts Tendency Diagram: graphs showing unconditional biases Ranked Hits Diagram: graphs showing frequencies of observed categories having the highest probability Weather Roulette: graphs showing estimates of forecast value

SCORES To calculate confidence intervals for the probabilistic scores go to: Options ~ Resampling Settings Check the box to calculate the confidence intervals. The intervals are calculated using bootstrapping. Set the size of the bootstrap sample (a large number gives more robust results, but will require more computation time), and the confidence level for the intervals.

PROBABILITY BINS For some verification procedures the probabilities are binned. The default is to round the probabilities to the nearest 10% (in line with the WMO SVSSLRF), but you may prefer to round the probabilities to the nearest 5% (which is standard in the RCOFs). To do so, go to: Options ~ Verification Settings and select 21 – (5% bins).