Verification of Met Office Lightning Forecasts for Aviation

Verification of Met Office Lightning Forecasts for Aviation Rebecca Stretton, Piers Buchanan, Stephen Moseley, Clare Bysouth The forecasting of hazardous weather conditions such as lightning is of huge importance to the Civil Aviation Authority. The accuracy and skill of lightning forecasts is key to the safety of all flights in the UK. This project looks into the verification of two different models’ lightning products to assess how well they forecast this rare event during the 2012 summer. Forecasts Observations MOGREPS-GLOBAL UKV (post processed) Probability of lightning index exceeding 10 Risk of lightning within 50 km radius of a from a 12 member ensemble: location: 0 → lightning is unlikely LR 1 → lightning expected 10 → deep convectively unstable environment with lightning a possibility LR 5 → lightning not expected Model resolution: 60 km Model resolution: 1. 5 km The forecast products are verified using observations from the ATDnet (Arrival Time Difference) system; this is an automatic lightning location network that senses lightning flashes over a geographical area. By timing the arrival of unique very low frequency radio waves (‘Sferics’) that are generated by individual lightning strokes the location of every stroke can be calculated. A stroke count was processed to give the number of strokes within a 50 km radius of each airport over a 6 hour period. Forecast lead times: T+21 and T+33 Forecast lead times: T+24 and T+36 Model diagnostics: Lifted index, CAPE, lightning index, total precipitation rate. Model diagnostics: Precipitation, CAPE, convective cloud features (temperature, base height, depth), wet bulb temperature. Forecasts are all valid at 06 Z and 18 Z (± 3 hours) Heathrow Gatwick Figure 2: Stroke counts within 50 km radius of Heathrow and Gatwick airports. Figure 1: Hyperbola plot - the potential locations of the lightning source from every pair of stations, the intersection of all the hyperbolae is the lightning location. Verification has been undertaken for the 3 month Summer period (30/05/12 – 06/09/12) using forecasts and observations at all 205 UK civil and defence aerodromes. Various different verification measures have been calculated to compare the models and explore their strengths and weaknesses. CONTINGENCY TABLES AND RELATIVE OPERATING CHARACTERISTIC CURVE MOGREPS-Global Hits Misses 175 1359 False alarms Correct rejections 931 36076 Hit =Non zero probability UKV Hits Misses 1485 48 False alarms Correct rejections 11791 25216 Hit = LR 1 and LR 2 • UKV shows considerably higher hit rate (0. 97), however the global model results have a much smaller false alarm rate (0. 025). • A ROC curve displays the forecasts’ ability to distinguish between an event and a non event. • Both ROC curves have a very similar initial gradient showing similar skill where data is present. • The limited number of points on the MOGREPS-Global curve is due to the small number of members in the ensemble (lowest threshold = 1/12). • The rare nature of lightning focuses the points on the ROC curve closer to the origin. RELIABILITY DIAGRAM A reliability diagram displays how well the predicted probabilities correspond to their observed frequencies. COST-LOST ANALYSIS A value plot shows the relative improvement in economic value between climatology and a perfect forecast. The UKV curve fully encompasses the MOGREPS-Global curve, indicating UKV has greater value. Figure 3: MOGREPS-Global ROC curve. Assumption: zero probability of precipitable water, minimal cloud cover, minimal risk of lightning. Figure 4: UKV ROC curve. Case Study 28 th June 2012 There were almost 60, 000 recorded strikes across the UK within a 13 hour period (05 Z to 18 Z). Both models indicated lightning was expected with a focus on the East coast of England. The ATDnet observations for 1800 Z indicate two strong bands of lightning in that region. a) c) b) Figure 5: Reliability diagram of MOGREPS-Global and UKV. • MOGREPS-Global shows good reliability for available probabilities. • Due to the categorical nature of the UKV forecast a classic reliability diagram could not be produced. LR 3, 4, 5 indicate minimal risk of lightning which correlates with the near zero observed frequency and LR 2&1 indicate some risk of lightning correlating with observations. Figure 6: Value plot of MOGREPS-Global and UKV. Perfect score = 1. 0. Figure 7: Forecasts for 2012/06/28 18 Z: a) MOGREPS-Global T+36, b) UKV T+33, c) lightning stroke map for 15 Z to 21 Z on 28/06/12. Conclusions and Future Work UKV shows high skill and value, greater than MOGREPS-G, however the specific risk and observed frequencies of LR 1&2 could be explored further. MOGREPS-G does show some skill which shows it has potential given the resolution of the model and the rare nature of lightning. In the future MOGREPS-UK (resolution of 1. 5 km) will be verified to allow direct comparison of the two models for the same domain and resolution, to determine if an ensemble forecast would be better. The ensemble could also be expanded to use 24 members by combining two forecast runs to investigate the ROC curves of the two models further. Met Office Fitz. Roy Road, Exeter, Devon, EX 1 3 PB United Kingdom Tel: 01392 885680 Fax: 01392 885681 Email: rebecca. stretton@metoffice. gov. uk © Crown copyright Met Office and the Met Office logo are registered trademarks