Application of the CRA Method William A Gallus

Application of the CRA Method William A. Gallus, Jr. Iowa State University Beth Ebert Center for Australian Weather and Climate Research Bureau of Meteorology

Idealized cases - geometric Observed (1) 50 pts to the right (2) 200 pts to the right (3) 125 pts to the right and big (4) 125 pts to the right and rotated (5) 125 pts to the right and huge Which forecast is best?

Traditional verification yields same statistics for cases 1 and 2 forecast

5 th case – traditional verification I E TH W R E N N forecast

1 st case – CRA verification

2 nd case – CRA verification CRA Technique yields similar results with cases 3 and 4

th 5 case – CRA verification

RESULTS ARE SENSITIVE TO SEARCH BOX FOR DISPLACEMENTS

Increase of size of rectangle (extra 90 instead of 30 pts) affects results

Further increase from 90 pts to 150 pts does not result in additional change

R E N IN vs. 5 th case 1 st case W E H T Area (# gridpoints) Observed Case 1 Case 5 7815 62789 Average rain rate (mm/h) . 36 10. 05 Maximum rain rate (mm/h) 25. 4 Rain volume (km 3) 1. 88 14. 45 Displacement east 2. 34° 6. 65° Displacement north 0° 0° RMS error after shift 0 11. 92 1. 00 0. 26 100% 12% Volume error (%) 0% 47% Pattern error (%) 0% 41% Correlation coefficient after shift Displacement error (%)

Perturbed cases (2) Shift 12 pts right, 20 pts down, intensity*1. 5 "Observed" Which forecast is better? 1000 km (1) Shift 24 pts right, 40 pts down

1 st case – traditional verification (1) Shift 24 pts right, 40 pts down

2 nd case – traditional verification (2) Shift 12 pts right, 20 pts down, intensity*1. 5 I E TH W R E N N

CRA verification Case 2 Case 1 Threshold=5 mm/h

CRA verification Case 2 Case 1 Threshold=5 mm/h

CRA verification Case 2 Case 1 Threshold=5 mm/h

Problem? System far from boundary in Case 1 – small shift – behaves as expected

System closer to boundary yields unexpected results – not all error is displacement

Problem is more serious for smaller system at edge of domain

Central system works well through medium displacements

But…. Larger displacement yields odd results

Summary • CRA requirement forecast and observed systems to be contiguous may limit some applications • Problems occur for systems near the domain boundaries – not yet clear what causes the problems

Results from separate study using object-oriented techniques to verify ensembles • Both CRA and MODE have been applied to 6 -hr forecasts from two 15 km 8 member WRF ensembles integrated for 60 h for 72 cases • This results in 10 x 16 x 72 = 11, 520 evaluations (plots, tables…. ) from each approach • Results were compared to Clark et al (2008) study

Clark et al. study • Clark et al. (2008) looked at two 8 member WRF ensembles, one using mixed IC/LBC, the other mixed physics/dynamic cores • Spread & skill initially may have been better in mixed physics ensemble vs. IC/LBC one, but spread grew much faster in the IC/LBC one, and it performed better than the mixed physics ensemble at later times (after 30 -36 h) in these 120 h integrations.

0. 5 mm 2. 5 mm Areas under ROC curves for both ensembles (Clark et al. 2007) Skill initially better in mixed ensemble but IC/LBC becomes better after hour 30 -36

Diurnal Cycle Variance continues to grow in IC/LBC ensemble but levels off after hour 30 in mixed ensemble. MSE always worse for mean of mixed ensemble – and performance worsens with time relative to IC/LBC ensemble.

0. 5 mm 2. 5 mm Spread Ratio also shows dramatically different behavior with increasing spread in IC/LBC ensemble but little or no growth in mixed ensemble after first 24 hours

Questions: • Do the object parameters from the CRA and MODE techniques show the different behaviors between the Mix and IC/LBC ensembles? • Do the object parameters from the CRA and MODE techniques show an influence from the diurnal trends in observed precipitation?

Diurnal signal not pronounced, only weak hint of IC/LBC tendency to have increasing spread with time – and only in MODE results Rain Rate Standard Deviation (in. ) – mean usually around. 5 inch Mix-MODE IC/LBC-MODE Mix-CRA IC/LBC-CRA 06 12 18 24 30 36 Forecast Hour 42 48 54 60 Wet times in blue

Standard Deviation of Rain Volume (km 3) – MODE values multiplied by 10 (mean ~ 1) No diurnal signal, hard to see different trends between 2 ensembles Mix-MODE IC/LBC-CRA IC/LBC-MODE Mix-CRA 06 12 18 24 30 36 42 48 54 60

CRA results show both ensembles with growing spread, and IC/LBC having faster growth Areal Coverage Standard Deviation (number of points above. 25 inch) – Mean ~ 800 pts IC/LBCCRA Mix-MODE IC/LBC-MODE 06 12 18 24 30 36 42 48 54 60

No clear diurnal signal, both CRA & MODE show max in 24 -48 h Mix-MODE IC/LBCMODE Mix-CRA IC/LBC-CRA 06 12 18 24 30 36 42 48 54 60

Mix-CRA IC/LBC-MODE Mix-MODE IC/LBC-CRA No diurnal signal, no obvious differences in behavior of Mix and IC/LBC 06 12 18 24 30 36 42 48 54 60

Other questions: • Is the mean of the ensemble’s distribution of object-based parameters a good forecast (better than ensemble mean put into CRA/MODE)? • Does an increase in spread imply less predictability? • How should a forecaster handle a case where only a subset of members show an object? These questions have been examined using CRA results

Mix Ensemble – in general, slight positive bias in rate, with Probability Matching forecast slightly less intense than mean of rates from members (PM usually better but not by much). Only during 06 -18 period does observed rate not fall within forecasted range. wet PM mean dry 06 12 18 24 30 36 42 48 54 60 wet IC/LBC – usually too dry with rain rate (at all hours except 0618), Probability Matching forecast exhibits much more variable behavior, again its performance is comparable to mean of rates of members mean PM 06 12 dry 18 24 30 36 42 48 54 60

Notice that at all times, the observed rain rate falls within the range of values from the full 16 member ensemble – indicating potential value forecasting

Mix Ensemble – clear diurnal signal, usually too much rain volume except at times of observed peak, when it is too small. Probability Matching equal in skill to mean of member volumes wet mean PM dry 06 12 18 24 30 36 42 48 54 60 wet IC/LBC Ensemble – also clear diurnal signal, less volume than Mix ensemble, Probability Matching usually a little wetter but generally comparable to mean of members dry 06 12 mean PM 18 24 30 36 42 48 54 60

NOTE: Even with all 16 members, there are still times when observed volume does NOT fall within range of predictions --not enough spread (indicated with red bar) Mix-PM Mix IC/LBC-PM IC/LBC Mix IC/LBC 06 12 18 24 30 36 42 48 54 60

Percentage of times the observed value fell within the min/max of the ensemble Rate - Mix Volume. IC/LBC Rate. IC/LBC Areal Coverage Mix

Skill (MAE) as a function of spread (> 1. 5*SD cases vs <. 5*SD cases) CRA applied to Mix Ensemble (IC/LBC similar) Area/1000 big SD Vol big SD Rate*10 low SD Rate*10 big SD Vol low SD 06 Area/1000 low SD 12 18 24 30 36 42 48 54 60

• It thus appears that total system rain volume and total system areal coverage of rainfall show a clear signal for better skill when spread is smaller • Rain rate does not show such a clear signal – (especially when 4 bins of SDs are examined). • Perhaps average rain rate for systems is not as big a problem in the forecasts as areal coverage (and thus volume)? Seems to be ~5 -10% error for rate, 1020% for volume, 10 -20% for area

Summary • Ensemble spread behavior for objectoriented parameters may not behave like traditional ensemble measures • Some similarities but some differences also in output from CRA vs MODE • Some suggestion that ensembles may give useful information on probability of systems having a particular size, intensity, volume

Acknowledgments • Thanks to Eric (and others? ) for organizing the workshop • Thanks to John Halley-Gotway and Randy Bullock for help with MODE runs for ensemble work, and Adam Clark for precip forecast output • Partial support for the work was provided by NSF Grant ATM-0537043

Gulf near-boundary system with increased rainfall