Probabilistic turbulence forecasts from ensemble models and verification
Probabilistic turbulence forecasts from ensemble models and verification Philip Gill and Piers Buchanan NCAR Aviation Turbulence Workshop, Boulder, 28 August 2013 © Crown copyright Met Office
Contents This presentation covers the following areas • 1. Introduction • 2. Turbulence forecasting • 3. Ensemble turbulence trial • 4. Current status • 5. Summary and future work © Crown copyright Met Office
1. Introduction • Turbulence - major cause of aviation incidents & active area of research • Forecasts routinely produced by UK Met Office - World Area Forecast Centre (WAFC) service (along with WAFC Washington, USA) • Operational forecasts currently derived from deterministic models • There is always a degree of uncertainty in deterministic forecasts • Ensembles are a way of communicating that uncertainty Photos © P Gill © Crown copyright Met Office
MOGREPS-G Met Office Global and Regional Ensemble Prediction System Operational from Sep 2008 after 3 years of trials At the time of the trial (Nov 2010 – Oct 2011), MOGREPS-G: Ø Ø Ø 60 km, 70 Levels T+72 h Run at 00 Z, 12 Z with 24 members ETKF for initial condition perturbations Stochastic physics (SKEB 2) N. B MOGREPS-15 run to 15 days at 00 z and 12 Z (at ECMWF) MOGREPS-G Upgrade (January 2013) Ø Same as above but at a resolution of approx. 33 km in mid latitudes and keeping 70 levels Ø 12 member forecasts run every 6 hours i. e. 00 Z, 06 Z, 12 Z and 18 Z. © Crown copyright Met Office
MOGREPS-G “postage stamp” plots © Crown copyright Met Office
Ensemble Turbulence Forecasts Deterministic turbulence forecast shows the value of a given turbulence indicator Ensemble turbulence forecast shows the probability of a turbulence indicator exceeding certain chosen values © Crown copyright Met Office Deterministic Probabilistic
2. Turbulence forecasting © Crown copyright Met Office
Turbulence predictors Turbulence can come from different sources – wind shear, convection, (mountain-wave) • Windshear related: • Ellrod TI 1, Ellrod TI 2 • Brown • Dutton • Lunnon • Convection related: • Convective rainfall rate • Convective rainfall accumulation • Both wind shear and convection: Richardson number • Turbulence climatology • Gridded field of observed turbulence frequency produced from aircraft observations from previous year • Light or greater and moderate or greater turbulence climatology produced © Crown copyright Met Office
Combining predictors • Combining turbulence predictors has been shown to increase forecast skill (Sharman et al, 2006) • We use weights derived from verification using ROC area • Predictors combined using a weighted sum © Crown copyright Met Office Combined probabilistic predictors
3. Ensemble turbulence trial © Crown copyright Met Office
Ensemble turbulence trial • Objective verification of deterministic and probabilistic model forecasts • Global verification to assess T+24 h MOGREPS-G forecasts of turbulence • T+24 chosen because this is a typical product time range • Verification against automated aircraft observations from the Global Aircraft Data Set (GADS) • 12 -month trial from November 2010 -October 2011 • Eight numerical predictors and climatology verified • Five thresholds used on each predictor to generate probability forecasts • Thresholds designed to cover light to moderate to severe turbulence © Crown copyright Met Office
Turbulence indicator thresholds
Aircraft observations • Global coverage, but flights mainly over northern hemisphere • Automated aircraft observations available every 4 seconds 10 -19 January 2009 Good coverage of N Atlantic, US and Europe Poor coverage of E Asia/Pacific region • Derived Equivalent Vertical Gust (DEVG) – Measurement of observed turbulence derived from vertical acceleration, aircraft mass, altitude and airspeed © Crown copyright Met Office
Verification methodology Turbulent event Turbulence forecast field Aircraft track within +/- 1. 5 h of validity time © Crown copyright Met Office Ellrod TI 1
Forecast assessment • Turbulent/non turbulent event defined on 10 min aircraft track ~120 km approx grid size of WAFC grid • Deterministic: Forecast turbulent event – CAT potential >= Threshold • Ensemble: Probability of exceeding a certain threshold for given turbulence indicator • Observed (moderate or greater) turbulent event - DEVG>=4. 5 m/s • Construct 2 x 2 contingency tables for each threshold • Sum entries in contingency tables over the verification period © Crown copyright Met Office Turbulence observed No turbulence observed Turbulence forecast Hit False alarm No turbulence forecast Miss Correct rejection 2 x 2 contingency table
Verification measures • Relative Operating Characteristic (ROC) curve by plotting the hit rate against false alarm rate for each threshold. The area under the ROC curve is a measure of skill. Useful for both deterministic and probabilistic forecasts • Reliability Diagram by plotting the forecast probability against the frequency of occurrence • Relative economic value (Richardson, 2000) by calculating the value for a range of cost/loss ratios. Useful for both deterministic and probabilistic forecasts. © Crown copyright Met Office
Ensemble turbulence verification - ROC Perfect forecast No skill Greater skill combining probabilistic forecasts Gill PG, Buchanan P. 2013. “An ensemble based turbulence forecasting system”, Meteorological Applications © Crown copyright Met Office
ROC area for each predictor Latest combined AUC = 0. 77 Gill PG, Buchanan P. 2013. “An ensemble based turbulence forecasting system”, Meteorological Applications © Crown copyright Met Office
Ensemble turbulence verification Reliability Perfect reliability Gill PG, Buchanan P. 2013. “An ensemble based turbulence forecasting system”, Meteorological Applications © Crown copyright Met Office Low probabilities but significant compared to background frequency
Ensemble turbulence verification – Relative economic value Greater value combining probabilistic forecasts Gill PG, Buchanan P. 2013. “An ensemble based turbulence forecasting system”, Meteorological Applications © Crown copyright Met Office
4. Operational production Piers Buchanan and Lisa Murray © Crown copyright Met Office
Current status • Project underway to produce probabilistic forecasts within an operational setting ready for implementation – completion in Mar 2014 • Verification complete for November 2010 to Feb 2013 – analysis ongoing. • Studies into using logistic regression to combine predictors using regional and seasonal weightings • Work on sourcing additional observations continues. © Crown copyright Met Office
Logistic regression • Study to compare methods for training algorithms to produce the most skilful forecasts • Current method is not scalable to a large number of predictors • Logistic regression provides an alternative method and initial studies demonstrate it can produce a forecast with higher value. © Crown copyright Met Office
5. Summary and future work © Crown copyright Met Office
Summary Benefits of using Ensemble turbulence forecasts • Significant increase in skill • Increased economic value of forecast • Confidence can be communicated with every forecast • Use of verification can help users to maximise the value of the forecast © Crown copyright Met Office
Future plans and challenges • Create operational probabilistic aviation hazard forecasts for Turbulence, Cb and Icing • Include additional predictors (CAPE, Mountain wave) • Investigate using a multi-model ensemble for WAFC turbulence forecasts (in collaboration with WAFC Washington) • Educating users in interpretation of probabilistic forecasts and verification © Crown copyright Met Office Photo © P Gill
Acknowledgements • Thanks to the UK Civil Aviation Authority for funding this project • The content of this presentation is available as a paper Gill PG, Buchanan P. 2013. “An ensemble based turbulence forecasting system”, Meteorological Applications Other references from this talk: Richardson D. , 2000. Q. J. R. Meteorolog. Soc. 126 649 -667 Sharman R, Tebaldi C, Wiener G, Wolff J, 2006. Weather Forecast. 21 268 -287 © Crown copyright Met Office
Questions & answers © Crown copyright Met Office
- Slides: 28