WAFC CAT verification Objective verification of GRIB CAT

WAFC CAT verification Objective verification of GRIB CAT forecasts Dr Philip G Gill, WAFS Workshop on the use and visualisation of gridded SIGWX forecasts, Paris, 14 September 2009 © Crown copyright Met Office

Contents This presentation covers the following areas • Introduction • SIGWX forecast comparison • Aircraft data • Verification methodology • Verification results • Summary • Further improvements © Crown copyright Met Office

Introduction • What – Objective verification of gridded binary (GRIB) and significant weather (SIGWX) Clear Air Turbulence (CAT) forecasts • Where – Global verification • When – November 2008 to May 2009 • Why – To demonstrate the quality of the new GRIB forecasts using objective verification. • How – Verification against aircraft observations from the Global Aircraft Data Set (GADS) © Crown copyright Met Office

Manual SIGWX BUFR and GRIB CAT forecasts • Manual SIGWX chart • Example field from new GRIB forecast © Crown copyright Met Office

Comparison of SIGWX BUFR charts UK © Crown copyright Met Office US UK&US

SIGWX BUFR CAT forecast comparison • One month UK-US comparison (January 2009) • Average coverage of globe UK ~6%, US ~3% • Percentage overlap of all forecasts between UK and US ~20% Areas forecast by both UK and US Areas forecast by US but not UK © Crown copyright Met Office Areas forecast by UK but not US

GRIB forecast comparison produced by HKO © Crown copyright Met Office

Global Aircraft Data Set • Archive of aircraft data set up by Joel Tenenbaum (State University of New York) • British Airways fleet of Boeing 747 -400 aircraft • Global coverage, but flights mainly over northern hemisphere • Automated aircraft observations every 4 seconds • Indicator of turbulence derived from vertical acceleration, aircraft mass, altitude and airspeed called the derived equivalent vertical gust (DEVG). © Crown copyright Met Office

GADS Data coverage 10 -day sample of GADS data © Crown copyright Met Office

Verification methodology © Crown copyright Met Office

Forecast assessment • Turbulent/non turbulent event defined on 10 min aircraft track ~120 km - approx grid size • Forecast turbulent event – CAT potential >= Threshold Turbulence observed No turbulence observed Turbulence forecast Hit False alarm No turbulence forecast Miss Correct rejection • Observed turbulent event – Slight or greater turb (DEVG>=2 m/s) Moderate or greater turb (DEVG>=4. 5 m/s) • Construct 2 x 2 contingency tables for each threshold • Sum entries in contingency tables over the verification period • Produce a Relative Operating Characteristic (ROC) curve by plotting the Hit rate against False alarm rate for each threshold. © Crown copyright Met Office 2 x 2 contingency table

Results UK GRIB and SIGWX moderate or greater turbulence ~500, 000 events ~300 turbulent events (devg>=4. 5 m/s) © Crown copyright Met Office

Variation with forecast range ~100 000 events Devg>=2 m/s © Crown copyright Met Office

Latitudinal variation UK GRIB moderate or greater turbulence 90 N to 50 N 20 N to 50 N ~200, 000 events ~50 turb events ~200 turb events 20 S to 20 N 50 S to 20 S ~60, 000 events ~12, 000 events ~100 turb events ~10 turb events © Crown copyright Met Office

Latitudinal variation US GRIB moderate or greater turbulence 90 N to 50 N 20 N to 50 N ~280, 000 events ~100, 000 events ~200 turb events ~100 turb events 20 S to 20 N 50 S to 20 S ~30, 000 events ~6, 000 events ~50 turb events ~3 turb events © Crown copyright Met Office

UK and US Nov 2008 and Jan 2009 ROC curve ~200 000 events Devg>=2

UK GRIB and SIGWX automated object UK automated SIGWX chart production system based on GRIB data ~300 000 events Devg>=2 m/s © Crown copyright Met Office

Summary of results • Both UK and US GRIB products show more skill than the manual SIGWX products. • Global UK and US GRIB CAT forecasts score similarly • Slight difference in scores as forecast range increases • Some differences in scores at individual latitude bands – best performance between 20 N and 90 N. • UK CAT coverage on manual BUFR SIGWX charts greater than the US © Crown copyright Met Office

Further improvements • Automate verification process • Improve consistency of forecasts by analysing verification data and altering production systems. • Use verification to test future model upgrades and retune algorithms © Crown copyright Met Office