A Superior Alternative to the Modified Heidke Skill

A Superior Alternative to the Modified Heidke Skill Score for Verification of Categorical Versions of CPC Outlooks Bob Livezey Climate Services Division/OCWWS/NWS 28 th Climate Diagnostics and Prediction Workshop Reno, October 20, 2003

Outline 1. 2. 3. 4. 5. 6. Introduction Contingency Tables & Notation Common Scores & Score Attributes Gandin & Murphy Equitable Scores Gerrity Scores Recommendations

Contingency Tables and Notation Table 4. 2. Contingency table giving pij in percent (total sample size n=788) for U. S. mean temperature forecasts for June through August 1983 -90. pij: pi: : qi: pi*: Joint relative frequencies Observed relative frequencies Forecast relative frequencies Prescribed relative observed frequencies (climatology)

Some Simple Categorical Skill Scores: Heidke, CPC Heidke, and Pierce

Some Simple Categorical Skill Scores: Heidke, CPC Heidke, and Pierce Score Inequitable HSS PSS GS FMA 0. 09 0. 10 0. 17 JJA 0. 12 0. 05 0. 08 Skill scores for U. S. mean temperature forecasts in three categories for February through April and June through August 1983 -90.

Desirable Attributes of Scores • Equitable; – Equitable without dependence on the forecast distribution; • Rewards for correct forecasts inversely proportional to their event frequencies; • Penalties for incorrect forecasts directly proportional to their event frequencies; – Penalties for incorrect ordinal forecasts with equal event frequencies proportional to degree of miss; • Consistent with an underlying linear association and insensitive to type or number of categories used. Note: 2 &3 imply that all information in the contingency table is taken into account.

Figure 4. 1

Gandin and Murphy Equitable Scores

Gerrity Scores

Figure 4. 4

Recommendations • CPC use the Gerrity score for ordinal multicategorical verification – Forecast history is digitized so skill history can be constructed – Clueless audience remains clueless – Score now equitably accounts for all facets of forecast performance • CPC use actual frequencies • CPC routinely determine confidence limits of scores • Reference Jolliffe and Stephenson (2003; Wiley)