Forecast calibration and combination Bayesian assimilation of seasonal
Forecast calibration and combination: Bayesian assimilation of seasonal climate predictions Caio A. S. Coelho Department of Meteorology University of Reading c. a. d. s. coelho@reading. ac. uk Thanks to: David B. Stephenson, Magdalena Balmaseda, Francisco J. Doblas-Reyes and Sergio Pezzulli PLAN OF TALK Calibration and combination issues Conceptual framework forecasting Forecast Assimilation: • Example 1: Nino-3. 4 index forecasts • Example 2: Equatorial Pacific SST forecasts • Example 3: S. American rainfall forecasts • Example 4: Regional rainfall downscaling EUROBRISA project Met Office, Exeter (U. K. ), 20 February 2006 •
This talk is based on the following work: Coelho C. A. S. 2005: “Forecast Calibration and Combination: Bayesian Assimilation of S Predictions”. Ph. D Thesis. University of Reading. 178 pp. Coelho C. A. S. , D. B. Stephenson, M. Balmaseda, F. J. Doblas-Reyes and G. J. van Old Towards an integrated seasonal forecasting system for South America. ECMWF Technical Memorandum No. 461, 26 pp. Also in press in the J. Climate. Coelho C. A. S. , D. B. Stephenson, F. J. Doblas-Reyes, M. Balmaseda, A. Guetter and G Oldenborgh, 2006: A Bayesian approach for multi-model downscaling: Seasonal forecas rainfall and river flows in South America. Meteorological Applications, 13, 1 -10. Stephenson, D. B. , Coelho, C. A. S. , Doblas-Reyes, F. J. and Balmaseda, M. , 2005: “Forecast Assimilation: A Unified Framework for the Combination of Multi-Model Weather and Climate Predictions. ” Tellus A, Vol. 57, 253 -264. Coelho C. A. S. , S. Pezzulli, M. Balmaseda, F. J. Doblas-Reyes and D. B. Stephenson, 20 “Forecast Calibration and Combination: A Simple Bayesian Approach for ENSO”. Journal of Climate. Vol. 17, No. 7, 1504 -1516. Coelho C. A. S. , S. Pezzulli, M. Balmaseda, F. J. Doblas-Reyes and D. B. Stephenson, 20 “Skill of Coupled Model Seasonal Forecasts: A Bayesian Assessment of ECMWF ENSO Forecasts”. ECMWF Technical Memorandum No. 426, 16 pp. Available from: http: //www. met. rdg. ac. uk/~swr 01 cac
Calibration and combination issues Calibration • • Why do forecasts need it? Which are the best ways to calibrate? How to get good probability estimates? Who should do it? Combination • • Why combine forecasts? Should model predictions be weighted or selected? How best to combine? Who should do it?
Conceptual framework Data Assimilation “Forecast Assimilation”
Multi-model ensemble approach Errors: Solution: Model formulation Initial conditions Multi-model Ensemble DEMETER Development of a European Multi-Model Ensemble System for Seasonal to Interannual Prediction http: //www. ecmwf. int/research/demeter
DEMETER Multi-model ensemble system 7 coupled global circulation models Model ECMWF. LODYC CNRM. Country International France CERFACS INGV MPI UKMO. France Italy Germany U. K. 9 member ensembles ERA-40 initial conditions SST and wind perturbations 4 start dates per year (Feb, May, Aug and Nov) 6 month hindcasts Hindcast period: 1980 -2001 (1959 -2001).
Examples of application • • 0 -d: Niño-3. 4 index • 1 -d: Equatorial Pacific SST • 2 -d: South American rainfall
Example 1: Empirical Niño-3. 4 forecasts 95% P. I. Well-calibrated: Most observations in the 95% prediction interval (P. I. )
ECMWF coupled model ensemble forecasts m=9 DEMETER: 5 -month lead Observations not within the 95% prediction interval! Coupled model forecasts need calibration
Univariate X and Y Prior: Likelihood: Posterior: Bayes’ theorem:
Likelihood modelling: y
Combined forecasts Note: most observations within the 95% prediction interval!
Comparison of the forecasts Empirical Combined Coupled SUMMARY Combined forecasts: • are better calibrated than coupled • have less spread than empirical • match obs better than either Blue dots = observations Red dots = mean forecast Grey shade = 95% prediction interval
Some verification statistics Mean Absolute Error (MAE) defined as: The Brier score (BS) is a simple quadratic score for probability forecasts of binary events (e. g. whether SST anomaly < 0). It is defined as: Forecast Climatology Empirical MAE ( C) 1. 16 0. 53 Brier score 0. 25 0. 05 Spread ( C) 1. 19 0. 61 Coupled Combined 0. 57 0. 31 0. 18 0. 04 0. 33 0. 32 Combined forecasts have smallest MAE, BS, and spread
Multivariate X and Y: More than one Normal variable Prior: Likelihood: Posterior: Matrices
Example 2: Equatorial Pacific SST DEMETER: 7 coupled models; 6 -month lead SST anomalies: Y (°C) Forecast Brier Score Climatol p=0. 5 0. 25 Multi-model 0. 19 FA 58 -01 0. 17 Forecast probabilities: p
Brier Score as a function of longitude Brier Score=0. 25 for p=0. 5 climatology Brier Score<0. 25 more skilful than climatology Forecast assimilation reduces (i. e. improves) the Brier score in the eastern and western equatorial Pacific
Brier Score decomposition reliability resolution uncertainty
Reliability as a function of longitude Forecast assimilation improves reliability in the western Pacific
Resolution as a function of longitude Forecast assimilation improves resolution in the eastern Pacific
Why South America? Seasonal climate potentially predictable DEMETER El Niño (DJF) Multi-model La Niña (DJF) Source: Climate Prediction Center (http: //www. cpc. ncep. noaa. gov) Correlation of ensemble mean DJF rainfall forecasts with PREC/L observations
Why South American rainfall? Agriculture Electricity: More than 90% produced by hydropower stations e. g. Itaipu (Brazil/Paraguay): • World largest hydropower plant • Installed power: 12600 MW • 18 generation units (700 MW each) • ~25% electricity consumed in Brazil • ~95% electricity consumed in Paraguay
Itaipu
Example 3: S. American rainfall anomaly composites DEMETER: 3 coupled models Obs Forecast Multi-model Assimilation (ECMWF, CNRM, UKMO) 1 -month lead Start: Nov DJF ENSO composites: 19592001 ACC=1. 00 ACC=0. 51 ACC=0. 97 ACC=1. 00 ACC=0. 28 ACC=0. 82 • 16 El Nino years ACC=Anomaly Correlation Coefficient • 13 La Nina years Spatial correlation of map with obs map (mm/day)
DJF rainfall anomalies for 1975/76 and 1982/83 Obs Multi-model Forecast Assimilation La Nina 1975/76 ACC=-0. 09 ACC=0. 59 ACC=0. 32 ACC=0. 56 El Nino 1982/83 (mm/day)
DJF rainfall anomalies for 1991/92 and 1998/99 Obs Multi-model Forecast Assimilation ACC=0. 04 ACC=0. 32 ACC=0. 08 ACC=0. 38 (mm/day)
Brier Skill Score for S. American rainfall Forecast assimilation improves the Brier Skill Score (BSS) in the tropics
Reliability component of the BSS Forecast assimilation improves reliability over many regions
Resolution component of the BSS Forecast assimilation improves resolution in the tropics
Empirical model for South American rainfall Y: DJF rainfall Z: ASO SST Matrices
Correlation maps: DJF rainfall anomalies Empirical Multi-model Integrated Comparable level of determinist skill Better skill in tropical and southeastern South America
Mean Anomaly Correlation Coefficient Empirical Multi-model Integrated Most skill in ENSO years and forecast assimilation can improve skill
Brier Skill Score for S. American rainfall Empirical ENS Multi-model Integrated Forecast assimilation improved Brier Skill Score (BSS) in the tropics
Reliability component of the BSS Empirical Multi-model Integrated Forecast assimilation improved reliability in many regions
Resolution component of the BSS Empirical Multi-model Integrated Forecast assimilation improved resolution in the tropics
Example 4: regional rainfall downscaling Multi-model ensemble 3 DEMETER coupled models ECMWF, CNRM, UKMO 3 -month lead Start: Aug NDJ Period: 1959 -2001
South box: NDJ rainfall anomaly Multi-model - - - Observation Forecast assimilation Correlati on Brier Score Multimodel 0. 57 0. 22 FA 0. 74 0. 17 Forecast assimilation improves skill substantially
North box: NDJ rainfall anomaly Multi-model - - - Observation Forecast assimilation Correlatio n Brier Score Multimodel 0. 62 0. 21 FA 0. 63 0. 18 Forecast assimilation improved skill marginally
Summary • Forecasts can be improved both by calibration and by combination • Statistical calibration and combination is analogous to data assimilation and is a fundamental and essential part of the forecasting process (forecast assimilation) • Forecast assimilation is easy to do for normally distributed predictands such as monthly mean temperatures and seasonal rainfall: • • • Nino-3 probability forecasts improved – less biased and smaller spread Equatorial SST forecasts improved in eastern and western Pacific S. American rainfall forecasts improved in Equatorial and Southern regions • Combination can improve the resolution of the forecasts (the ability to discriminate between different observed situations) whereas calibration can improve the reliability of the forecasts • First steps towards an integrated seasonal forecasting system for South America including both empirical and coupled model predictions • EUROBRISA project will implement this system at CPTEC - Brazil
The EUROBRISA Project Lead Investigator: Caio A. S. Coelho Key Idea: To improve seasonal forecasts in S. America: a region where there is seasonal forecast skill and useful value. http: //www. met. rdg. ac. uk/~swr 01 cac/EUROBRISA Aims • Strengthen collaboration and promote exchange of expertise and information between European and S. American seasonal forecasters • Produce improved well-calibrated real-time probabilistic seasonal forecasts for South America • Develop real-time forecast products for non-profitable governmental use (e. g. reservoir management, hydropower production, and agriculture) Institutions Country Partners CPTEC Brazil Coelho, Cavalcanti, Silva Dias, Pezzi ECMWF EU Anderson, Balmaseda, Doblas-Reyes, Stockdale INMET Brazil Moura, Silveira Met Office UK Graham, Davey, Colman Météo France Déqué SIMEPAR Brazil Guetter Uni. of Reading UK Stephenson Uni. of Sao Paulo Brazil Ambrizzi, Silva Dias CIIFEN Ecuador EUROBRISA was approved by ECMWF council in June 2005 Camacho, Santos
(oi) Reliability diagram (Multi-model) o (pi)
Direct and inverse regression Regression of obs on forecasts Regression of forecasts on obs y More natural to model uncertainty in forecasts for a given observation (ensemble spread of dots) than to model uncertainty in observations for a given ensemble forecast. so we model the likelihood on right rather than the more common forecast calibration (MOS) approach on the left.
(oi) Reliability diagram (FA 58 -01) o (pi)
Moment measure of skewness Measure of asymmetry of the distribution
- Slides: 44