Verification of WAFS Global Icing Products Jennifer Mahoney

Verification of WAFS Global Icing Products Jennifer Mahoney 1 and Sean Madine 1, 2 1 NOAA/Earth System Research Laboratory 2 Cooperative Institute for Research in the Atmosphere This work was completed on behalf of the U. S. Federal Aviation Administration Presented at: WAFS Workshop on the Use and Visualisation of Gridded SIGWX Forecasts Paris, September 14 -15 2009

Acknowledgements This work was completed on behalf of the U. S Federal Aviation Administration Project Team • • Project Lead: Sean Madine Project Analyst: Chungu Lu Programming Support: Mike Kay Program Chief: Jennifer Mahoney

Outline of Discussion • Motivation, Background, and Challenges • Approach and Assessment Strategy – Creation and validation of Cloud. Sat Icing Product (CLIP) • Forecast Analyses – Comparison of WAFC U. K. and WAFC U. S. Global Icing – Comparison of WAFC Global Icing to Significant Weather (Sig. Wx) Product • Summary

Motivation • Icing forecasts are used in strategic planning – One example: Planning for amount of fuel carried on board aircraft to manage possible icing encounters. • The WAFC Washington (U. S. ) and the WAFC London (U. K. ) have developed Global Icing Products, updated frequently, that may improve icing information that is provided to the aviation community.

Background • Project Goal: Evaluate the quality of the WAFC Global Icing Forecasts; with resource and time constraints: – Altitudes that typically contain icing (i. e. , mid-levels) – Scoped evaluation to Nov 2008 – Jan 2009; winter in northern hemisphere – Domains: Global, North Pacific, North Atlantic; with a North Atlantic focus • Evaluation criteria : 1) accuracy of the products should be at least as good as the human generated Significant Weather (Sig. Wx) forecasts produced today, and 2) WAFC U. S. and U. K icing products need to be compatible

Challenges • Obtaining a representative observation dataset – Voice pilot reports (PIREPS) for icing are sparse in time and space – PIREPs poorly sample at flight levels relevant to operational decisions (enroute flight levels are typically above the occurrence of icing) – PIREPs disproportionately sample icing conditions • Exploring new techniques for creating a proxy observation of icing in the atmosphere

Assessment Approach • Compare the quality of the WAFC U. K icing product to the WAFC U. S. icing products – Measure the quality – Qualitative examination of compatibility (i. e. , incompatible means ambiguity of the forecast in the planning process) • Compare the quality of the WAFC Global Icing Products to the Significant Weather Product – Measure the quality

Assessment Strategy • Build an icing verification proxy based on satellite products and use for evaluating the WAFC icing forecasts over global data-sparse domains • Development of Cloud. Sat Icing Product (CLIP) – Tuned and validated CLIP with respect to the Current Icing Potential (CIP), current U. S. operational icing analysis – Study period: November 2007 through January 2008 (Note: Same season, 1 year earlier) • Use CLIP to assess the quality of the WAFC global icing products and Sig. Wx products over global domains – Study period: November 2008 through January 2009

Cloud. Sat Icing Product (CLIP) Strengths • Cloud. Sat cloud profiling radar very effective at diagnosing existence of clouds, particularly at levels relevant to our study – CLIP highly accurate at detecting no cloud and therefore ‘no icing’ • Information for CLIP allows assessment of forecast risk (miss) and forecast efficiency (false alarms) – PIREPs don’t allow the false alarm measure because of the imbalance of yes/no reporting • CLIP’S radar input provides – An objective measure of cloud, at high resolution – Continuous reporting along orbiting track – An improvement upon the subjective nature and sparseness of using PIREPs

Comparison of the WAFC Global Icing Products

Analysis of WAFC Global Icing Products • Many stratifications examined – Geographic domain Global, N. Atlantic, N. Pacific – Flight level • All, FL 100, FL 140 – Cloud type (as diagnosed by Cloud. Sat) • All, Convective, Non-convective – Algorithm threshold • 0. 1, 0. 3, and 0. 5 – Lead time • All leads 6, 12, …, 36 -h individually

Analysis of WAFC Global Icing Products • Main thread of investigation – North Atlantic domain • Domain of interest during assessment winter time period and included significant amount of air traffic density – Altitudes 10, 000 - 20, 000 ft • Operationally significant altitudes and typically contain ice – Non-convective clouds • Products definition specifically includes non-convective icing – Max icing attribute • Most compatible WAFC forecasts

Domains Global, North Atlantic, North Pacific; 3 Cloud. Sat orbits, each orbit ~1. 5 h,

WAFS Maximum Icing 12 January 2009 UK/FL 100 US/FL 100 UK/FL 140 US/FL 140

Vertical Cross Sections Along Cloud. Sat Path Atlantic Domain WAFS-UK WAFS-US 12 January 2009 Cloud Classification CLIP

WAFS Maximum Icing 23 December 2008 UK/FL 100 US/FL 100 UK/FL 140 US/FL 140

Vertical Cross Sections Along Cloud. Sat Path Atlantic Domain 23 December 2008

Comparison of Performance: Flight Level (Statistics for 12 - and 18 -hr leads, Atlantic domain, Non-convective clouds, Max icing attribute) POD Product False Alarm Rate (1 -PODn) ROC Thresholds: 0. 1, 0. 3, and 0. 5

Comparison WAFC Icing Forecasts in Operational Context CLIP UK (fcst) Yes Icing Yes 12367 15356 No 6684 54837 US (fcst) Yes Icing Yes No Blue – Agreement Yellow – False alarms Red - Misses No Icing No Cloud 8987 11576 10063 58617 No Icing Warm Cloud 2940 6430 No Icing Warm Cloud 3661 5709 No Icing Cold Cloud 3640 3375 No Icing Cold Cloud 2251 4764 Atlantic domain, 0. 1 Threshold, FL 100/140, 12 - and 18 -hr leads, Max icing intensity, Non-convective clouds

Comparison Summary • The WAFC U. K and WAFC U. S. forecasts provide roughly the same forecast efficiency, but the WAFC U. K. provides lower operational risk WAFC U. K WAFC U. s FAR 64% 66% POD 65% 47% Bias 1. 8 1. 4

Comparison of the WAFC Global Icing Products to the Sig. Wx Product

WAFS Mid Level SIGWX Product • • • Thunderstorms/CB Tropical Cyclones Mod/Sev Turb (CAT or IC) Mod/Sev Icing Tropopause Heights Jetstreams (80 kt and above depicted) Jet Depth Volcanic Eruptions Widespread Sand/Dust storms Release of Radioactive Materials NOTE: Because of choice to use N. Atlantic domain, only U. S Sig. Wx Product was evaluated

Analysis - Main Thread • • Atlantic domain Altitudes 10, 000 - 20, 000 ft Non-convective clouds Max icing attribute

Comparison of Performance: Sig. Wx Chart (FL 100/140, Sig. Wx issuances/lead, Atlantic domain) POD Product False Alarm Rate (1 -PODn) NOTE: Because of choice to use N. Atlantic domain, only U. S Sig. Wx product was evaluated ROC Thresholds: 0. 1, 0. 3, and 0. 5

Comparison Summary • The WAFC U. K and WAFC U. S. forecasts provide roughly the same forecast efficiency, but the WAFC U. K. provides lower operational risk WAFC U. K WAFC U. S. Sig. Wx Product FAR 64% 66% 69% POD 65% 47% 31% Bias 1. 8 1. 4 1. 2

Summary • The U. K. icing product slightly outperforms the U. S. icing product in an operational; For similar FAR values, the U. K. has higher POD • Qualitatively, the WAFC U. K. and U. S products are incompatible in volume extent and intensity of potential • Both U. K. and U. S. products perform better than the Sig. Wx Product in the operational context; For similar FAR, both WAFC products have higher POD values • Based on this study and understanding of the algorithms, the automated WAFC product could be significantly improved with respect to volume coverage and calibration of potential

Appendix • • False alarm ratio = yn/yy+yn False alarm rate = 1 -PODn = yn/nn+yn PODn = nn/nn+yn POD = yy/yy+ny 2 x 2 Contingency Table Observation Forecast Y N Y YY YN N NY NN