Deutscher Wetterdienst COSMODEEPS Susanne Theis Christoph Gebhardt Zied
Deutscher Wetterdienst COSMO-DE-EPS Susanne Theis Christoph Gebhardt, Zied Ben Bouallègue, Michael Buchhold
Presentation Overview è setup of COSMO-DE-EPS è first results of pre-operational phase è verification è forecasters‘ feedback è enlarging the sample at low cost è looking into past production cycles COSMO GM – September 2011
Presentation Overview è setup of COSMO-DE-EPS è first results of pre-operational phase è verification è forecasters‘ feedback new è enlarging the sample at low cost è looking into past production cycles COSMO GM – September 2011 new
Deutscher Wetterdienst Setup of COSMO-DE-EPS
COSMO-DE-EPS status è pre-operational phase has started: Dec 9 th, 2010 è pre-operational setup: è 20 members è grid size: 2. 8 km convection-permitting è lead time: 0 -21 hours, 8 starts per day (00, 03, 06, . . . UTC) è variations in physics, initial conditions, lateral boundaries COSMO GM – September 2011 model domain
C G O M S E , M O ID 7 F k. E S , m E P G S F S , 2. 8 G k S m M Generation of Ensemble Members Ensemble Chain BC-EPS COSMO GM – September 2011 plus variations of • initial conditions • model physics
C G O M S E , M O ID 7 F k. E S , m E P G S F S , 2. 8 G k S m M Generation of Ensemble Members plus variations of • initial conditions • model physics Ensemble Chain BC-EPS is running as a time-critical application at ECMWF COSMO GM – September 2011
Generation of Ensemble Members Perturbation Methods Peralta, C. , Ben Bouallègue, Z. , Theis, S. E. , Gebhardt, C. and M. Buchhold, 2011: Accounting for initial condition uncertainties in COSMO-DE-EPS. Submitted to Journal of Geophysical Research. Peralta, C. and M. Buchhold, 2011: Initial condition perturbations for the COSMO-DE-EPS, COSMO Newsletter 11, 115– 123. Gebhardt, C. , Theis, S. E. , Paulat, M. and Z. Ben Bouallègue, 2011: Uncertainties in COSMO-DE precipitation forecasts introduced by model perturbations and variation of lateral boundaries. Atmospheric Research 100, 168 -177. (contains status of 2009) COSMO GM – September 2011
COSMO-DE-EPS plans (2011 -2014) è upgrade to 40 members, redesign 2011 reach operational status 2012 è statistical postprocessing è initial conditions by LETKF è lateral boundary conditions by ICON EPS COSMO GM – September 2011 2013
Deutscher Wetterdienst First Results of Pre-operational Phase - verification - forecasters‘ feedback new
Deutscher Wetterdienst First Results of Pre-operational Phase - verification - forecasters‘ feedback new
Verification Method SYNOP RADAR § Ensemble Members § Probabilities of Precipitation COSMO GM – September 2011
PREC 1 h accumulation, threshold: 0. 1 mm DETERMINISTIC SCORES for Individual Members 0. 5 Equitable Threat Score 0. 4 JUNE 2011 (multi-model / multi-configuration) Are there many cases with the same „best member“ or „wettest member“? 0. 3 0. 2 IFS GME GFS GSM 0. 1 0. 0 0 5 } 20 members 10 15 Forecast Time [h] COSMO GM – September 2011 20 Do the ensemble members have different long-term statistics? - look at Equitable Threat Score - look at Frequency Bias Index (results similar, not shown)
PREC 1 h accumulation, threshold: 0. 1 mm DETERMINISTIC SCORES for Individual Members 0. 5 Equitable Threat Score 0. 4 JUNE 2011 (multi-model / multi-configuration) Are there many cases with the same „best member“ or „wettest member“? 0. 3 0. 2 IFS GME GFS GSM 0. 1 0. 0 0 5 } 20 members 10 15 20 Do the ensemble members have different long-term statistics? - look at Equitable Threat Score - look at Frequency Bias Index (results similar, not shown) Forecast Time [h] Only small differences in long-term statistics Members may be treated as equally probable COSMO GM – September 2011
PREC 1 h accumulation RANK HISTOGRAM observation. . . - …treated as „Ensemble Member“ Frequency JUNE 2011 - …ranked according to prec amount at each grid point and forecast hour How frequent is each rank? 0. 05 0. 00 If ensemble underdispersive U-shaped rank histogram 1 6 11 16 Rank COSMO GM – September 2011 21 Observation
PREC 1 h accumulation observation. . . RANK HISTOGRAM - …treated as „Ensemble Member“ - …ranked according to prec amount at each grid point and forecast hour Frequency JANUARY 2011 How frequent is each rank? 0. 05 0. 00 If ensemble underdispersive U-shaped rank histogram 1 6 11 16 Rank COSMO GM – September 2011 21 Observation
PREC 1 h accumulation observation. . . RANK HISTOGRAM - …treated as „Ensemble Member“ - …ranked according to prec amount at each grid point and forecast hour Frequency JANUARY 2011 How frequent is each rank? 0. 05 0. 00 If ensemble underdispersive U-shaped rank histogram 1 6 11 16 Rank 21 Observation a) Underdispersiveness relatively small b) Four groups Many cases with large influence by global models COSMO GM – September 2011
PREC 1 h accumulation BRIER SKILL SCORE JANUARY 2011 How good are the probabilities derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) > 0. 1 mm > 2 mm 0 5 10 15 Forecast Time [h] COSMO GM – September 2011 20 - for different precipitation thresholds (colors) (probabilites of exceeding a certain threshold) - for different forecast lead times (x-axis)
PREC 1 h accumulation BRIER SKILL SCORE JANUARY 2011 How good are the probabilities derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) > 0. 1 mm > 2 mm 0 5 10 15 20 - for different precipitation thresholds (colors) (probabilites of exceeding a certain threshold) - for different forecast lead times (x-axis) Forecast Time [h] Always positive! Ensemble provides additional value to COSMO-DE Additional value grows with lead time (less deterministic predictability) COSMO GM – September 2011
PREC 1 h accumulation BRIER SKILL SCORE JUNE 2011 How good are the probabilities derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) > 0. 1 mm > 2 mm 0 5 10 15 20 - for different precipitation thresholds (colors) (probabilites of exceeding a certain threshold) - for different forecast lead times (x-axis) Forecast Time [h] Always positive! Ensemble provides additional value to COSMO-DE Additional value grows with lead time (less deterministic predictability) COSMO GM – September 2011
PREC 1 h accumulation BRIER SKILL SCORE MAY - JULY 2011 How good are the probabilities derived from the ensemble? compared to the deterministic COSMO-DE (always forecasting 0% or 100%) Look at Brier Skill Score (no skill: zero) - for different precipitation thresholds (x-axis) (probabilites of exceeding a certain threshold) 0. 1 1 2 5 10 20 Threshold [mm/h] - for all foreast lead times For larger precipitation amounts (summer): even more additional value COSMO GM – September 2011
PREC 1 h accumulation RELIABILITY DIAGRAM JUNE 2011 log (# fcst) > 0. 1 mm > 2 mm COSMO GM – September 2011 Are the probabilities already well calibrated? (without extra calibration) If we isolate all cases with a forecast probability of -say- 75 -85% … did the event occur in 80% of these cases? diagonal line: optimal - for different prec thresholds (colors) (probs of exceeding a threshold)
PREC 1 h accumulation RELIABILITY DIAGRAM JUNE 2011 log (# fcst) > 0. 1 mm > 2 mm Are the probabilities already well calibrated? (without extra calibration) If we isolate all cases with a forecast probability of -say- 75 -85% … did the event occur in 80% of these cases? diagonal line: optimal - for different prec thresholds (colors) (probs of exceeding a threshold) Reliability diagram shows some bias and underdispersiveness Lines are not flat additional calibration has good potential COSMO GM – September 2011
Summary of Verification (Precipitation) è Ensemble provides additional value to COSMO-DE (for all accumulations, lead times, precipitation thresholds, …) è Ensemble underdispersiveness is relatively small è Ensemble members may be treated as equally probable è Additional calibration has good potential COSMO GM – September 2011
Summary of Verification (Precipitation) è Ensemble provides additional value to COSMO-DE (for all accumulations, lead times, precipitation thresholds, …) è Ensemble underdispersiveness is relatively small è Ensemble members may be treated as equally probable è Additional calibration has good potential Pre-operational COSMO-DE ensemble prediction system already meets fundamental quality requirements for precipitation COSMO GM – September 2011
Other Variables è T_2 M and VMAX have been verified è ensemble spread is far too small è nevertheless, ensemble provides additional value to COSMO-DE COSMO GM – September 2011
Other Variables è T_2 M and VMAX have been verified è ensemble spread is far too small è nevertheless, ensemble provides additional value to COSMO-DE ensemble prediction system has been developed with focus on precipitation Upgrade to 40 members will also look at other variables COSMO GM – September 2011
Deutscher Wetterdienst First Results of Pre-operational Phase - verification - forecasters‘ feedback new
Forecasters‘ Feedback è available products: see figure precipitation, snow, wind gusts, T_2 m probability thresholds: warning criteria è all products on grid-scale (2. 8 km) è in addition: precipitation probabilities for larger areas (10 x 10 grid boxes) „probability that the precipitation event will occur anywhere within the region“ COSMO GM – September 2011 probabilities, quantiles, ensemble mean, spread, min, max, …
Forecasters‘ Feedback è evaluate „full package“ - including the visualization tool - consistency of products è select interesting cases è consider forecasters‘ interpretation - perception as intended? - is there any value in the forecast, additional to forecasters‘ knowledge? COSMO GM – September 2011
Forecasters‘ Feedback è what they prefer to use: è 90%-quantile of precipitation è precipitation probabilities for an area (10 x 10 grid points) COSMO GM – September 2011
Forecasters‘ Feedback è what they prefer to use: è 90%-quantile of precipitation è precipitation probabilities for an area (10 x 10 grid points) è what they appreciate: è early signals for heavy precipitation è indication that deterministic run may be wrong COSMO GM – September 2011
Forecasters‘ Feedback è what they prefer to use: è 90%-quantile of precipitation è precipitation probabilities for an area (10 x 10 grid points) è what they appreciate: è early signals for heavy precipitation è indication that deterministic run may be wrong è what they criticize: è jumpiness between subsequent runs è lack of spread in T_2 M and VMAX COSMO GM – September 2011
Forecasters‘ Feedback è what they prefer to use: è 90%-quantile of precipitation è precipitation probabilities for an area (10 x 10 grid points) è what they appreciate: è early signals for heavy precipitation è indication that deterministic run may be wrong è what they criticize: è jumpiness between subsequent runs è lack of spread in T_2 M and VMAX è what they are learning: è dealing with low probabilities (10% probability for extreme weather issue a warning? ) COSMO GM – September 2011
Deutscher Wetterdienst Enlarging the Sample at Low Cost - looking into past production cycles new
Deutscher Wetterdienst Enlarging the Sample at Low Cost - looking into past production cycles - Introduction - Verification Study - Recommendation new
COSMO-DE-EPS Overview è start of pre-operational phase / evaluation è 20 members probabilities, quantiles, etc è runs at 00 UTC, 03 UTC, 06 UTC, … COSMO GM – September 2011 December 2010
COSMO-DE-EPS Overview è start of pre-operational phase / evaluation December 2010 è 20 members probabilities, quantiles, etc è runs at 00 UTC, 03 UTC, 06 UTC, … è start of pre-operational phase Q 1 2012 with 40 ensemble members è reach operational status End of 2012 è statistical postprocessing 2013 of probabilities of precipitation COSMO GM – September 2011
COSMO-DE-EPS Overview è start of pre-operational phase / evaluation è 20 members probabilities, quantiles, etc è runs at 00 UTC, 03 UTC, 06 UTC, … è start of pre-operational phase December 2010 further improvements ? Q 1 2012 with 40 ensemble members è reach operational status End of 2012 è statistical postprocessing 2013 of probabilities of precipitation COSMO GM – September 2011
Enlarging the Sample at Low Cost è looking into past production cycles 0 UTC 15 UTC 18 UTC } 20 members 21 UTC } 20 members 00 UTC } 20 members COSMO GM – September 2011
Enlarging the Sample at Low Cost è looking into past production cycles 0 UTC 15 UTC 18 UTC } 20 members 21 UTC } 20 members 00 UTC } 20 members } 60 members COSMO GM – September 2011
Important Check è How good are forecasts from past production cycles? COSMO GM – September 2011
Important Check è How good are forecasts from past production cycles? Forecast Start: FBI June 2011 precipitation 1 h accumulations threshold: 1 mm/h ETS similar results for - different months - different starting times - different thresholds - 6 h accumulations 00 UTC 21 UTC 18 UTC 0 2 4 6 8 10 12 14 time of day [UTC] COSMO GM – September 2011 0 2 4 6 8 10 12 14 time of day [UTC]
Important Check è How good are forecasts from past production cycles? Forecast Start: FBI June 2011 precipitation 1 h accumulations threshold: 1 mm/h ETS similar results for - different months - different starting times - different thresholds - 6 h accumulations 00 UTC 21 UTC 18 UTC 0 2 4 6 8 10 12 14 time of day [UTC] The quality is similar. Exception: Most recent run with forecast lead time 0 -3 hours is the best. COSMO GM – September 2011
Probability of Precipitation > 10 mm/h Resulting Probabilities 20 members 20 + 20 members 2011 -05 -22 15 UTC Forecast Start: 09 UTC COSMO GM – September 2011 % Forecast Start: 09 UTC, 06 UTC, 03 UTC
Summary of Study è looking into past production cycles (20+20+20) è quality gain for precipitation (except first 2 forecast hours) è does not harm quality of T_2 M and VMAX_10 M è also applicable to „area probabilities“ Recommendation: Look into past production cycles COSMO GM – September 2011
Technical aspect: how to derive probabilities from 20+20+20 members 0 UTC 15 UTC 18 UTC 21 UTC 00 UTC 21 h forecast in (pre-)operational mode + 6 h additional lead time required COSMO GM – September 2011 21 UTC
COSMO GM – September 2011
Deutscher Wetterdienst Extra Slides
Deutscher Wetterdienst Example
Resulting Probabilities P(TOT_PREC) > 10 mm/h 2011 -05 -22 : 00 UTC 14 -15 h 20 members COSMO GM – September 2011 20 + 20 members
Resulting Probabilities P(TOT_PREC) > 10 mm/h 2011 -05 -22 : 03 UTC 11 -12 h 20 members COSMO GM – September 2011 20 + 20 members
Resulting Probabilities P(TOT_PREC) > 10 mm/h 2011 -05 -22 : 06 UTC 08 -09 h 20 members COSMO GM – September 2011 20 + 20 members
Resulting Probabilities P(TOT_PREC) > 10 mm/h 2011 -05 -22 : 09 UTC 05 -06 h 20 members COSMO GM – September 2011 20 + 20 members
Resulting Probabilities P(TOT_PREC) > 10 mm/h 2011 -05 -22 : 12 UTC 02 -03 h 20 members COSMO GM – September 2011 20 + 20 members
Resulting Probabilities P(TOT_PREC) > 10 mm/h 2011 -05 -22 : 12 UTC 02 -03 h 20 members 20 + 20 members very nice side effect: less “jumpiness” COSMO GM – September 2011
Enlarging the Sample at Low Cost (2) è looking into a spatial neighbourhood at each grid point Schwartz et al. , 2010 Wea. Forecasting only applicable to probabilites at a grid point not applicable to „area probabilities“ COSMO GM – September 2011
Probability of Precipitation > 5 mm/h 2011 -06 -16 20 UTC Example 20 members + neighbourhood (radius 10 Δx) % Forecast Start: 12 UTC COSMO GM – September 2011 Forecast Start: 12 UTC
Probability of Precipitation > 1 mm/h Quality Gain in Resulting Probabilities Resolution Gain Reliability Gain 20 + 20 members Sharpness Loss most important 20 members + neighbourh. reference: 20 members 0 5 10 time of day [UTC] 15 0 5 10 15 0 time of day [UTC] 5 10 15 June 2011 time of day [UTC] Definition of scores: Ben Bouallègue, Z. , 2011: Upscaled and fuzzy probabilistic foreasts: verification results. COSMO Newsletter 11, 124 -132. COSMO GM – September 2011
Probability of Precipitation > 1 mm/h Quality Gain in Resulting Probabilities Resolution Gain Reliability Gain 20 + 20 members Sharpness Loss most important 20 members + neighbourh. reference: 20 members 0 5 10 time of day [UTC] 15 0 5 10 15 0 time of day [UTC] 5 10 15 June 2011 time of day [UTC] Precipitation: Both methods achieve clear quality gain (except first 2 hours). COSMO GM – September 2011
Probability of Precipitation Quality Gain in Resulting Probabilities Resolution Gain Reliability Gain Sharpness Loss most important 20 + 20 members + neighbourh. reference: 20 members 0. 1 1. 2. 5. threshold [mm/h] Same conclusion for different precipitation thresholds COSMO GM – September 2011 June 2011
Probability of 2 m-Temperature (T_2 M) Resolution Gain Reliability Gain Sharpness Loss most important 20 + 20 members + neighbourh. reference: 20 members 10 20 30 threshold [°C] 10 20 threshold [°C] For 2 m-temperature: 20+20+20 does not harm quality COSMO GM – September 2011 30 June 2011
Probability of Wind Gusts (VMAX_10 M) Resolution Gain Reliability Gain Sharpness Loss most important 20 + 20 members + neighbourh. reference: 20 members 14 18 threshold [m/s] For wind gusts: 20+20+20 does not harm quality COSMO GM – September 2011 June 2011
Weights? Gain compared to L 1 F = (1 - W) L 1 + W L 2 COSMO GM – September 2011 0<W<1
Technical Setup of the Ensemble Chain 12 4 global models 06 00 12 18 IFS start BC-EPS at ECMWF arrival BC-EPS at DWD start COSMO-DE-EPS COSMO GM – September 2011 18 12 12 +06 12 18 15 +09 15 18 00 21 00 06 03 06 12 09 12 18 15 18 +06 +09 +06 18 00 06 12 18 21 03 09 15
Looking back how far? 00 UTC 09 UTC 2 runs 3 runs 4 runs BSS raw ensemble (1 run) as reference COSMO GM – September 2011 2 runs 3 runs 4 runs
- Slides: 66
