The Assessment of Seasonal Forecast Skills from the

The Assessment of Seasonal Forecast Skills from the Canadian HFP Q. Teng 1, S. Kharin 1, F. Zwiers 1 and X. Zhang 2 1 Canadian Centre for Climate Modelling and Analysis, Meteorological Service of Canada Climate Research Branch, Meteorological Service of Canada 2 CCRM, Outline 1. Introduction 2. Model and Data 3. Results ● Deterministic skills ● Probabilistic skills 4. Summary

Introduction ● Basic premise: Slow variations in lower-boundary forcing (e. g. , SST; sea-ice cover and temperature; land surface). ● Main sources of error: ► initial conditions; ensemble prediction ► model error multi-model prediction multi-model ensemble prediction ● Forecast verification: ► deterministic measures: e. g. , mean square error (MSE); root mean square error (RMSE); root mean square skill score (RMSSS); anomaly correlation coefficient (ACC) etc. ► probabilistic measures: e. g. , Brier score (BS); Brier skill score (BSS); attributes (or reliability) diagram; relative operating characteristic (ROC) and its skill score etc.

Model and Data ● Model Output Historical Forecasting Project HFP 1 (Derome et al. 2001) HFP 2 Models Resolution GCM 2 (Mc. Farlane et al. 1992) T 32 L 10 SEF (e. g. , Ritchie 1991) T 63 L 23 GEM (Côté et al. 1998) 1. 875°L 50 GCM 3 (Mc. Farlane et al. 2001) T 63 L 31 Predictions MAM; JJA; SON; DJF; (1969 - 1995) JFMA; FMAM; MAMJ; AMJJ; MJJA; JJAS; JASO; ASON; SOND; ONDJ; NDJF; DJFM; (1969 - 2001) Ensemble Members 6 6 10 10 ● Verification datasets: NCEP/NCAR (Kalnay et al. 1996) and ERA-40 (Simmons and Gibson 2000) reanalyses fields ● Variables: Z 500 , T 700 & T 2 m for MAM, JJA, SON, DJF; (1969 -1995)

Bias -- Z 500 (DJF) GCM 2 SEF GEM GCM 3

RMSE -- Z 500 NEX TR GL ERA-40 NCEP NAM North America (NAM): 20° - 80°N, 150° - 45°W; Northern Extra-tropics (NEX): 30° - 87. 5°N, 180°E - 180°W; Tropics (TR): 30°S - 30°N, 180°E - 180°W; Globe (GL): 87. 5°S - 87. 5°N, 180°E - 180°W. Canada (CA): 40° - 87. 5°N, 150° - 45°W; Pacific-North-America (PNA): 20° - 80°N, 180° - 60°W; Northern Hemisphere (NH): 0° - 87. 5°N, 180°E - 180°W; purple: GCM 2 green: SEF yellow: GEM red: GCM 3

RMSE NEX TR GL T 2 m T 700 Z 500 NAM GCM 2; SEF; GEM; GCM 3

Brier Score (BS) and Brier Skill Score (BSS) ● BS is the MSE of probability forecast; where is the forecast probability; is the event variable: ● BSS

Brier Skill Score (BSS) -- Cont’d ● BSS Where ● Three methods to estimate P (Kharin and Zwiers 2003):

BSS -- Z 500 & T 700

BSS -- Z 500 North America DJF SON JJA MAM BN NN AN

BSS -- Z 500 Tropics DJF SON JJA MAM BN NN AN

BSS -- Time Series (DJF Tropics) Years The year refers to the Dec. of the corresponding winter.

Decomposition of BS Following Murphy (1973), the BS can be expressed as: reliability Where resolution uncertainty is the number of probability bins; denotes the total number of forecast/event pairs; represents the number of times each probability forecast is used; is the observed sample relative frequency; is the observed overall relative frequency or sample climatology.

Improved Gaussian Count Attributes Diagram -- T 2 m Tropics (DJF)

Improved Gaussian Count ROC -- Z 500 North America (DJF)

ROCSS Z 500 T 700 T 2 m

Summary ● The identity of the best single model varies; ● The multi-model ensemble is superior to the single model ensembles; ● The raw Gaussian fit method is generally better than the count method, while the statistically improved technique may not always be beneficial.

Acknowledgments Juan-Sebastian Fontecilla, Normand Gagnon, Fouad Majaess and Steve Lambert.