HFIP Verification Team FY 10 Review HFIP Team

HFIP Verification Team: FY 10 Review HFIP Team Meeting Miami, FL 09 November 2010 • OAR / GFDL § Tim Marchok • OAR / AOML / HRD § Rob Rogers • OAR / ESRL § Mike Fiorino § Ed Tollerud • SUNY-Albany § Ryan Torn • NRL § Jim Goerss § Hao Jin • TCMT / DTC § § Barb Brown Louisa Nance Ligia Berndardet (DTC) Paul Kucera • NESDIS § Mark De. Maria • NCEP / NHC § James Franklin • NCEP / EMC § Vijay Tallapragada • NWS / OST § Tony Eckel

Verification Team 2010 Activities • Verification team milestone categories: Ø Plan and coordinate demo and retro tests Ø Perform model verification analyses Ø Develop, test and implement new tools for evaluation and verification of hurricane forecasts Ø Perform in-depth evaluations of collected hurricane forecast data sets.

Verification Team 2010 Milestones ü Distribute tracker to the community (GFDL, DTC): Ø Code finalization • Process required close collaboration between GFDL and DTC • Several bug fixes related to use of tracker with very hi-res, small domain grids • A few technical upgrades added to be in line with requirements that DTC had. • Added 10 -m wind speed minimum and 10 -m relative vorticity as tracking variables

Verification Team 2010 Milestones ü Distribute tracker to the community (GFDL, DTC): Ø Code distribution • Code released to community in March, 2010 • Background & use of tracker was taught in February, 2010 as part of the WRF for Hurricanes tutorial • Tracker is included in the HWRF Scientific Documentation and Users Guide.

Verification Team 2010 Milestones ü Complete upgrades to tracker for detecting genesis and tracking new storms (GFDL): Ø Most recent version sent to DTC in early October, will be released at some point prior to next HWRF Tutorial in April, 2011. Ø Upgrades for genesis tracking: • Code added to allow for more uniform, uninterrupted tracks. • 2 different methods are used to determine thermodynamic phase of model cyclones: – (1) Cyclone Phase Space – (2) Check of closed contour in the temperature anomaly field in the 300 -500 mb layer Ø Suggest unifying tracker code with NCEP for 2011 season

Verification Team 2010 Milestones ü Perform verification of TC genesis for the various global models (ESRL, GFDL, NESDIS, SUNYA): Ø Mike Fiorino will present results in his talk later today GFS http: //www. emc. ncep. noaa. gov/gmb/tpm/emchurr/gfs_gen_2010 ECMWF http: //www. emc. ncep. noaa. gov/gmb/tpm/emchurr/emx_gen_2010 Slight differences in this case between the GFS and ECMWF in terms of where and when they form Danielle, and how quickly thermal structure develops into that of a TC

TC Genesis: Seasonal Prediction Seasonal (Jul-Oct) predictions of Atlantic basin hurricanes (2000 -2010) • GFDL FV cubed-sphere model at 25 km • Inter-annual correlation = 0. 96 • RMSE=1. 84 without bias removal (left plot) • RMSE=0. 93 with bias removal (bottom plot) -- from S. J. Lin and Jan-Huey Chen, GFDL

Verification Team 2010 Milestones ü Create track, intensity and verification plots and post these online in real time (EMC, NRL): Operational HWRF Statistics Page • • New Plots Generated Every 6 hours when HWRF is run Individual and Season-to-date Stats Available EMC now uses James Franklin’s TC verification package, so that NHC, EMC and DTC are all using a common TC verification package.

COAMPS-TC Hao Jin (NRL) COAMPS-TC Web Site Real-time TC prediction and verification in support of HFIP project Storm names (http: //www. nrlmry. navy. mil/coamps-web/tc) Monthly Calendar • Unique and convenient TC sorting methods based on storm lists and dates • A monthly calendar with all available storm forecasts • Easy navigating features: animaiton, frame by frame, different date groups and variables, and at different scales (from regional, mesoscale, to vortex scales) 9

COAMPS-TC Real-Time Verification and Structure Diagnostics Forecasts for Hurricane Igor initialized at 1200 UTC 11 Sept 2010 Track Maximum Wind Speed 10 m Wind 200 Pa-850 Pa Wind Shear Minimum Sea Level Pressure 500 Pa-850 Pa Averaged RH and Wind BEST OFCL 10

Spatial Track & Intensity Verification GHMI HWFI LGEM DSHP -- Arielle Alpert, Johns Hopkins Univ.

Spatial Track & Intensity Verification http: //www. gfdl. noaa. gov/tc-spatial-verification

Verification of SST response in GFDL model 8) GFDL Ocean SST cooling in GFDL for Cat 1 -3 storms is similar to obs, however the cooling in the model is overdone for strong Cat-4 as well as Cat-5 storms. OBS -- from Ian Lloyd, Princeton Univ.

Summary of 2009 Demonstration Evaluation � Results presented at the AMS Conference on Hurricanes and Tropical Meteorology in Tucson, AZ � Project Website: http: //www. ral. ucar. edu/projects/hfip/d 2009/ � Evaluation was conducted for Global and Regional models: �Global: FIM, GFS, NOGAPS �Regional: GFDL, HWRF, COAMPS-TC, WRF-ARW, MMEM �Note: Evaluation of Tier 1 only because of limited availability of Tier 2 products Example: Absolute Intensity (left) and Track (Right) error distributions with respect to lead time for OCD 5 and MMEN; HFIP B = HFIP Baseline and HFIP NB = HFIP Normalized Baseline

2010 Retrospective Evaluation � TCMT worked with NHC to develop a Stream 1. 5 evaluation plan � The TCMT performed an evaluation of four Stream 1. 5 candidates � Retrospective results are posted on the TCMT Website http: //www. ral. ucar. edu/projects/hfip/h 2010/verify/

Comparison of Tier 1 and Tier 2 with Common Tracker �To eliminate uncertainty in the evaluation, a consistent tracker should be used, which requires Tier 2 products �TCMT applies the GFDL Community Tracker to all model forecasts. �Consistent tracker, available in community repository �Example: Comparison of AHW 1 Tier 1 and AHW 1 Tier 2 with different tracker codes � 5 -10 kt differences observed for median intensity error �Some 2 -5 nm differences observed for median track error

Perform verification of a comprehensive set of forecasts generated by HWRF-V 3 HWRF code management: atmospheric model HWRF releas e Tutoria l, HWRF Beta releas e 04/2011 WRF V 3. 3 02/2011 WRF V 3. 2. 1+ 09/2010 WRF V 3. 2. 1+ 08/2010 WRF V 3. 2. 1 03/2010 WRF V 3. 2 07/2009 05/2004 WRF V 3. 1. 1 (…) WRF V 2 HWRF for operations (2011) Contributions from EMC & NOAA Research to HWRF 2011 Baseline “R 2 final” 2010 baseline “R 1” (…) oper HWRF 2009 oper HWRF 2010 “R 2” 17

Perform verification of a comprehensive set of forecasts generated by HWRF-V 3 R 2 prime: R 2 WRF + Comm Vortex V 2. 0 + V 3. 2 + • 2010 operational HWRF in V 3. 2. 1+ framework is performing adequately. • After this test, bugs were found in V 3. 2. 1+ and a new test, R 2 -final, was run by EMC to create the 2011 baseline. • DTC will add R 2 -final capabilities to community codes and run R 2 final on HFIP jet computer to create a “Reference Configuration”, a benchmark of Community HWRF for users.

Investigate new techniques for the evaluation of ensemble forecasts of TC �New techniques being investigated: • Probability ellipses and cones (Hammill et al. 2010) • Two dimensional rank histograms (e. g. , Wilks 2004; Gneiting et al. 2008) • Dispersion-spread diagrams (e. g. , Tony Eckel 2010) • Using new spatial forecast verification techniques (e. g. , Features-based, Gallus 2010)

evaluation of ensemble forecasts of TC Probability Ellipses and Cones Hurricane Bill cyclone positions initialized at 0000 UTC 19 Aug 2009. Observed track in black, dots indicate ensemble members. Colors represent days (black is day zero). Ellipses generated from bivariate normal distribution fitted to forecast positions with contour enclosing 90% probability. Dispersion-spread diagrams Courtesy of Tony Eckel Is the ensemble spread consistent with the variability between individual members and the observations? Two dimensional rank histograms Multivariate Rank Histograms: How should the values be ordered? One way is to use minimum spanning trees (MST). Features-based application for ensembles

References for Ensemble Verification Techniques � Gallus, W. A. , Jr. , 2010: Application of object-based verification techniques to ensemble precipitation forecasts. Wea. Forecasting, 25, 144 - 158. � Gneiting, T. , L. I. Stanberry, E. P. Grimit, L. Held, N. A. Johnson, 2008: Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds. Test 17, (2), 211 - 235. � Hamill, T. M. , J. S. Whitaker, M. Fiorino and S. G. Benjamin, 2010: Global ensemble predictions of 2009's tropical cyclones initialized with an ensemble Kalman filter. Submitted to Mon. Wea. Rev. � Wilks, D. S. , 2004: The Minimum Spanning Tree Histogram as a Verification Tool for Multidimensional Ensemble Forecasts, Mon. Wea. Rev. , 1329 - 1340.