COMPLETING THE FORECAST ASSESSING AND COMMUNICATING FORECAST UNCERTAINTY

COMPLETING THE FORECAST: ASSESSING AND COMMUNICATING FORECAST UNCERTAINTY Zoltan Toth Environmental Modeling Center NOAA/NWS/NCEP Acknowledgements: NRC Report; Louis Uccellini, Steve Lord & Ensemble Team http: //wwwt. emc. ncep. noaa. gov/gmb/ens/index. html 1

OUTLINE • REVIEW OF 3 RD ENSEMLBE USER WORKSHOP • MAJOR RECOMMENDATIONS REGARDING NRC REPORT • WHAT, WHY, AND HOW TO CHANGE? • SPECIFIC RECOMMENDATIONS 2

• Logistics 3 rd ENSEMBLE USER WORKSHOP – Oct. 30 – Nov. 1 2006, Laurel, MD – Close to 100 participants • NWS Regions (12), Headquarters (15), NCEP (37) • OAR (6), other government (4), private (4), academic (8) sectors & international (8) – For further info, see: http: //wwwt. emc. ncep. noaa. gov/gmb/ens/User. Wkshop_Oct 2006. html • Topics – Assessing & propagating uncertainty throughout entire forecast process • From observations to users • Working group discussions – – Ensemble configuration Statistical post-processing Data depository / Interrogation / Product generation/dissemination / Verification User support / Outreach / Training • Outcome – Enthusiastic discussions • Convergence on number of topics • Open questions needing further research identified – Strong support for sustained effort/engagement, annual meetings, etc • Recommendations – Course of action in response to NRC report • Presented here first – will solicit and incorporate feedback from participants

MAJOR RECOMMENDATIONS - 1 FOLLOW NRC REPORT RECOMMENDATIONS All major recommendations embraced • Develop roadmap for assessing & communicating forecast uncertainty – Based on science, technology, workforce considerations – Consistent with NOAA’s mission, NWS plans, embraced by “Enterprise” – Define end goal, 5 -10 years horizon • Adopt ensemble-based forecast process – Maximize forecast skill – Unify scientific, software, and technological infrastructure across NOAA » Weather, water, climate applications • Form high level planning and oversight team – March 2007 – Each NWS office to delegate one representative – Team reports to Corporate Board • Name programmatic and technical leads • Develop roadmap – Sept. 2007

MAJOR RECOMMENDATIONS - 2 • Develop roadmap for assessing & communicating forecast uncertainty • Revise operational requirements to make them probabilistic – Make probabilistic format the primary requirement • For each forecast application: – Replace single value/categorical format with probabilistic format as primary requirement – Revise/supplement corresponding performance measures (GPRA) – Essential for orderly transition from traditional to new forecast process • NWS is requirement-driven organization – Without clear new requirements, process is doomed – Phased implementation schedule in consultation with responsible offices • Allows orderly transition to new requirements • New requirements prepared by Planning & oversight team (or its designate) – Assisted by responsible NWS office – Presented to Corporate Board for approval – By March 2008

MAJOR RECOMMENDATIONS - 3 • Develop roadmap for assessing & communicating forecast uncertainty (. 5 yr) • Revise operational requirements to make them probabilistic (1 year) • Design, develop, and gradually implement new forecast process – Focus on missing scientific, technological, and human components • Identify self-contained components with – Clear requirements and interfaces between components – Define basic capabilities achievable in 2 -3 years • Limited but consistent with and leading to end goal – Full capabilities in 5 -10 years • Interface with NOAA THORPEX program – Research/development to improve skill & utility of probabilistic forecasts – Leverage related major NOAA, national, and international efforts • Integrate with NWS, NOAA, national activities – New NOAA CONOPS process – NOAA-wide regional service plans – NUOPC planning/development • Provide long-term funding support through PPBES – Overlap with W&W High Impact Event Theme Team • Form development teams for specific tasks/components following workshp – Identify potential contributors within & outside of NWS & NOAA

PROPOSED CHANGE • Major paradigm shift – Incorporate assessment and communication of uncertainty in forecast process • Is it a major change in course of “Weather Ship”? – Ie, abandon course of ever improving single forecast scenario (expected value)? • No – Expand, not abandon – Keep improving fidelity of forecasts, PLUS – Add new dimension • Capture other possible scenarios – ensemble forecasting – Use a flotilla, instead of one ship, in exploring nature – Existing activities are subset of expanded forecast process • Single value forecast is expected value of full probability distribution – Can keep serving forecasts in old format to users who prefer that

Single forecast (driven by GFS winds) example for drifting virtual ice floe 7 September 2006 Initial position Bob Grumbine, EMC

Ensemble forecast for drifting ice floe for same case Initial position Bob Grumbine, EMC

Most likely forecast for drifting ice floe for same case Initial position Bob Grumbine, EMC

WHY CHANGE IS NEEDED? • Why users (should) care about forecast uncertainty? – They admittedly want minimal or no uncertainty in forecasts • Distinction between no uncertainty in the forecast, vs. not talking about it – Forecast uncertainty cannot be arbitrarily reduced • Despite major ongoing & continuing efforts, they persist forever – Chaotic nature of atmosphere - land surface – ocean coupled system + initial/model errors – Level of uncertainty is determined by nature and level of sophistication in forecast system – Forecast uncertainty can be ignored though • Negative consequence on informed users – Not able to prepare for all possible outcomes » Assumes a certain scenario and remains vulnerable to others • Possibly serious loss in social/economic value of forecast information • Why forecasters (should) care about forecast uncertainty? – Imperfect forecasts are consistent w. observations (reliable) only if in prob format • If in other format, must be brought into probabilistic format through – Verification / bias correction

ADVANTAGES OF PROBABILISTIC FORMAT • More rationalized and enriched forecaster - user interactions Old paradigm – Convoluted forecaster-user decision process • User expects forecaster to make decision for them in presence of uncertainty – “Will it rain? ” – “ 80%” – “But tell me, will it rain? ” New paradigm – Forecaster and user decision processes enhanced and better linked • Allows forecasters to capture all knowledge about future conditions – Provision of information related to multiple decision levels in probabilistic format critical » Provider helps interpret probabilistic info & and modify user decision process if needed » Option to continue providing single value or other limited info until user ready • Allows users to decide about most beneficial course of action given all possibilities – Proper use of probability or other uncertainty information needed - Training » User requests critical weather forecast info depending on their sensitivity

TRADITIONAL FORECAST PROCESS • Focus on single forecast scenario – Reducing uncertainty in single forecast is main emphasis • Loss of accuracy in forecast estimate of expected value of distribution – Mean of ensemble cloud provides better estimate – Ignores or simplifies forecast uncertainty • Uncertainty assessed as statistically averaged error in single fcst (second thought) – Ensemble cloud provides better estimate of case dependent variations in uncertainty – Use of single value / categorical forecast format • Difficulty in formulating/communicating plausible alternate scenarios – Ensemble member forecasts can directly feed into Decision Support Systems • One-way flow of information from observations to users – Not adaptable to case dependent user requirements • Ensemble can propagate back user requirements to adaptive – Observing, assimilation, modeling/ensemble, post-processing and application components » Applications in planning and execution of new CONOPS in high impact events

Single value Distribution PROPAGATING FORECAST UNCERTAINTY z Ensemble Forecasting: Central role – bringing the pieces together

HOW CAN IT BE DONE? NEW PARADIGM • Adopt ensemble approach across all environmental prediction activities – Expand forecasting with new dimension of uncertainty • Multiple scenarios (in place of single scenario) – Provides best forecast estimate for both expected value (as before) and uncertainty (new) – Unified scientific, technological, human approach • Sharing resources across NWS & NOAA – Ensemble is centerpiece both symbolically and figuratively in forecast process • Ensembles act as a glue & two-way information channel – Observing system, data assimilation, numerical modeling » ENSEMBLES – Statistical post-processing, product generation, decision making • Design, develop, & implement missing components of new forecast process – Gradual, measured steps • Basic capability - Short-term, 2 -3 yrs, leading to • Full implementation - Long-term, 5 -10 yrs

ENSEMBLES AND THE RESEARCH COMMUNITY LINKED THROUGH THORPEX – MAJOR INTERNATIONAL RESEARCH PROGRAM SOCIOECON. SYSTEM TEST CENTER WEATHER-CLIMATE LINK USER CONTROLLABLE PROBABILISTIC FORECASTS MODEL ERRORS & HIGH IMPACT MODELING GLOBAL INTERACTIVE FORECAST SYSTEM (GIFS) TEST CENTER Days 15 -60 NWS OPERATIONS GLOBAL OPERATIONAL CLIMATE FORECASTING / CTB GOAL: Accelerate improvements of high impact weather forecasts INTEGRATED DATA ADAPTIVE COLLECTION & ASSIMILATION & USE OF OBSERVATIONS FORECASTING

SPECIFIC RECOMMENDATIONS - 1 • Continue development of expanded forecast process – Focus on adaptive methods applicable for high impact events • • • Collection & use of observations (targeted observations) Data assimilation (case dependent background error estimation) Numerical modeling (adaptive resolution & high impact modeling) Ensemble forecasting (case dependent variations in membership & composition) Decision support systems (flexible user actions depending on forecast probabilities) • Bias correction & downscaling methods for ensembles – Estimate/correct lead-time dependent bias in ensemble forecasts (on model grid) – Generate fine resolution (NDFD grid) uncertainty/ensemble data • Establish connection between (bias corrected) coarse model vs. fine user grids – Use reanalysis & hind-casts with operational systems as needed • Define summary ensemble information to be used to (ST) – Collapse vast amount of ensemble data for inclusion in expanded NDFD/NDGD • E. g. , 10, 50, and 90 percentile of forecast distribution (in place of single value) – Manually inspect/modify summary ensemble information ST – Short Term (2 -3 yrs); LT – Long Term (5 -10 yrs)

SPECIFIC RECOMMENDATIONS - 2 • Contribute to establishment of NOAA-wide environmental data depository – Expand NDFD/NDGD database to include forecast uncertainty (ST) – Develop capability to hold all ensemble trajectories (LT) • All members, variables, lead times • Develop ensemble interrogation, modification, & product generation tools to – – – Derive summary information from ensemble (ST) Manually modify summary ensemble info (ST) Derive additional statistics from summary info (product generation, ST) Automatically modify ensemble trajectories based on modified summary info (LT) Derive any info from full ensemble data (product generation, LT) • Develop telecommunication facilities to access data – Summary info & limited derived products (ST) – All ensemble forecasts & derived products (LT) ST – Short Term (2 -3 yrs); LT – Long Term (5 -10 yrs)

SPECIFIC RECOMMENDATIONS - 3 • Develop unified NWS/NOAA probabilistic verification package to (ST) – Assess statistical reliability and resolution for • Computing official performance measures • Evaluating value added along forecast chain • Assessing value in newly developed vs. operational techniques • Develop & implement comprehensive training to – Prepare all participants for their new roles in expanded forecast process, incl. • • Statistical background Ensemble methods Best forecast practices in assessing uncertainty Applications of probabilistic & other uncertainty information • Develop outreach program on use & communication of uncertainty – In partnership with entire Weather, Water & Climate Enterprise • Determine best ways of communicating uncertainty • Compile sample of Decision Support Systems using uncertainty information – Establish close partnership with public sector users (e. g. , emergency, water management) • User feedback on new activities – Explore how forecast process can be adapted according to user requirements ST – Short Term (2 -3 yrs); LT – Long Term (5 -10 yrs)

NFUSE PRIORITIES? Issues to be discussed/worked out in this order: • NRC Recommendations – Do we agree on major recommendations? • If not, let’s discuss any issues • If so, can we accept those as the basis/guidelines for our work • 3 rd Ensemble User Workshop Recommendations – Three major recommendations in response to NRC Report • High level roadmap guiding our collaboration (Sept 2007) • New operational requirements (March 2008) • Design, develop, and gradually implement new forecast process – Basic capabilities (2009 -2010) – Full capabilities (2012 -2017) – Nine specific recommendations on design, development, implementation • Work areas identified, links to be worked out – – – – – Continue development of expanded forecast process Bias correction & downscaling methods for ensembles Define summary ensemble information to be used (ST) Contribute to establishment of NOAA-wide environmental data depository Develop ensemble interrogation, modification, & product generation tools Develop telecommunication facilities to access data Develop unified NWS/NOAA probabilistic verification package (ST) Develop & implement comprehensive training Develop outreach program on use & communication of uncertainty Do we need to modify/augment etc these recommendations?