Comparison of Scores in Model Comparison MSSS COR

Comparison of Scores in Model Comparison (MSSS, COR, RMSE and ROC) Tomoaki OSE (MRI/JMA) With help of Yukiko NARUSE (CPD/JMA) and use of JMA/MRI CGCM and AGCM Hindcasts

MSSS, COR and RMSE (from SVS_LRF of WMO manual)

Dependence of MODEL comparisons on SCORES FOR RAIN ANOMALY OF NH SUMMER PREDICTION FROM NH WINTER

Originally defined MSSS is not useful at current Seasonal Forecasts CGCM MSSS AGCM MSSS CGCM BIAS AGCM BIAS

Better COR does not necessarily correspond to better MSSS. CGCM MSSS AGCM MSSS CGCM COR AGCM COR

MSSS is largely influenced by model SDs. CGCM MSSS AGCM MSSS CGCM SDmdl/ SDanal AGCM SDmdl/ SDanal

The change of RMSE is more similar to the change of MSSS than COR. CGCM RMSE AGCM RMSE CGCM COR AGCM COR

Comparison of SCOREs: ROC map seems to be more similar to COR map. CGCM ROC CGCM MSSS CGCM COR CGCM RMSE

Summary • It is confirmed that better COR does not necessarily indicate better MSSS in model comparison. • Because MSSS is definitely dependent on model SD (and/or BIAS) in addition to COR. • The change of RMSE shows similar characteristics to the change of MSSS. • Small RMSE regions do not correspond to the regions for good seasonal forecasts. • COR map seems to show more similar distribution to ROC map than MSSS map.

Why COR for SIF modelers? • Temporal correlation is the most fundamental skill measure for SI forecasts predictability. The square of correlation is the ratio of forecast signal variance to the total variance. • BIAS, SD, MSSS and RMSE can be corrected by linear transform using OBS while correlation is invariant. • Correlation is widely used and most users are familiar with it and easy to understand. • In order to evaluate the value of probability forecast for particular user’s application, more sophisticated skill measure may be needed anyway for each.

Dependence of MODEL comparisons on SCORES FOR T 2 M ANOMALY OF NH WINTER PREDICTION FROM NH SUMMER

Comparison of SCOREs: ROC map seems to be more similar to COR map. CGCM ROC CGCM MSSS CGCM COR CGCM RMSE

Dependence of MODEL comparisons on SCORES FOR RAIN ANOMALY OF NH WINTER PREDICTION FROM NH SUMMER

Comparison of SCOREs: ROC map seems to be more similar to COR map. CGCM ROC CGCM MSSS CGCM COR CGCM RMSE

Dependence of MODEL comparisons on SCORES FOR T 2 M ANOMALY OF NH SUMMER PREDICTION FROM NH WINTER

Comparison of SCOREs: ROC map seems to be more similar to COR map. CGCM ROC CGCM MSSS CGCM COR CGCM RMSE