National Weather Service Verification Program Overview Brenton Mac

National Weather Service Verification Program Overview Brenton Mac. Aloney II National Weather Service Headquarters Silver Spring, MD July 19 th, 2006 Summer WAS*IS 2006

Verification Programs The National Weather Service (NWS) Performance Branch collects warning and forecast products, as well as observations to generate verification scores. These verification scores are displayed on the Performance Management website accessible only to NWS employees. Summer WAS*IS 2006

Verification Programs Verification data is available via the Stats on Demand program. Summer WAS*IS 2006

Verification Programs Verification statistics are calculated on two types of products. Forecasts Summer WAS*IS 2006 Warnings

Forecast Verification Aviation – Terminal Aerodrome Forecast (TAF) • Ceiling, Visibility, Significant Weather, Wind Speed, Direction, and Gust, and Flight Category Hydrology – River Stage Forecast – Quantitative Precipitation Forecasts (QPF) Summer WAS*IS 2006

Forecast Verification Marine – Coastal and Offshore Forecast • Wave Height, Wind Speed, Wind Direction Fire Weather – National Fire Danger Rating System Forecasts (NFDRS) • State of the Weather, Temperature, Humidity, Wind Speed Public – Point Forecast Matrices (PFM) Forecasts • Max/Min Temperature, Probability of Precipitation (Po. P) Summer WAS*IS 2006

Warning Verification Short Duration Warnings • • Tornado (TOR) Severe Thunderstorm (SVR) Flash Flood (FFW) Special Marine (SMW) Long Duration Warnings* • Winter Storm (WSW) • High Wind (NPW) • Coastal Flood (CFW) * Verification of long duration warnings is currently done by hand. Starting in October 2006, the verification of long duration warnings will be automated. Summer WAS*IS 2006

Warning Verification As far back as 1986, warnings have been verified using counties warned in the Universal Geographic Code (UGC) line of a warning product. Verifiable events are collected, entered into storm data at the forecast offices (WFO), and sent to NWS Headquarters monthly. Summer WAS*IS 2006

Warning Verification Any event occurring during the valid time of warned county, verifies the warning. The event is also considered “warned”. Summer WAS*IS 2006

Warning Verification The NWS is changing how they issue short duration warnings. In the next few years, warnings will be issued where the “polygon” is the warning area, not the county. This “polygon” information is currently available on all SVR, TOR, FFW, and SMW products. Summer WAS*IS 2006

Brent’s Dream World • Due to a limited budget, the majority of the money the Performance Branch receives is spent on the development of new verification programs. • Having verification information is great, but it does not tell the whole story. • I would love to see other areas of “Performance Evaluation” explored. Summer WAS*IS 2006

Questions and Issues • Performance evaluation questions I would like to find answers to: – What is an acceptable False Alarm Ratio to the public? to Emergency Managers? to businesses? to others who depend on our warnings? Summer WAS*IS 2006

Questions and Issues • Other questions: – Are there any new or existing metrics that we are not using and might be useful in showing a better picture of our performance? Summer WAS*IS 2006

Questions and Issues • Other questions: – How can we get the forecasters to better understand the current metrics so that they are able to improve their forecasts and warnings? Summer WAS*IS 2006

Questions and Issues • Other questions: – Are there any ways for the WFOs to easily do their own service assessments? Does anyone know of any software that might help? What types of relationships can be set up within the local community that can help channel feedback to the WFO on their performance? Summer WAS*IS 2006

Contact Information Brenton Mac. Aloney II Brent. Mac. Aloney@noaa. gov Phone: 301 -713 -0090 x 135 Summer WAS*IS 2006