STATISTICAL POSTPROCESSING OF ENSEMBLE FORECASTS Zoltan Toth Yuejian
STATISTICAL POSTPROCESSING OF ENSEMBLE FORECASTS • Zoltan Toth, • Yuejian Zhu, Dingchen Hou, and Richard Wobus (4) • • Ackn. : S. Lord, H. -L. Pan, S. Saha, J. Schaake (1), P. Dallavalle (2), D. Unger(3) , W. Ebisuzaki 3) • (1) • • : Office of Hydrologic Developments, NWS (2) : Meteorological Development Laboratory, NWS (3) : Climate Prediction Center, NCEP/NWS 1
OUTLINE • DEFINITION • SOURCES OF STATISTICAL INCONSISTENCY • HOW TO IMPROVE STATISTICAL CONSISTENCY • CONDITIONS FOR USE OF STAT. METHODS TO IMPR. CONSISTENCY • MEASURES OF STATISTICAL CONSISTENCY • CHARACTERISTICS OF ENSEMBLE POSTPROCESSING • ENSEMBLE POSTPROCESSING METHODS 2
DEFINITION • STATISTICAL CONSISTENCY OF FORECASTS WITH OBSERVATIONS • Select a particular forecast • Consider that the same forecast is issued many times • Construct distribution of verifying observations (analysis), given selected fcst • If selected fcst is EQUIVALENT to distribution of obs. conditioned on fcst => Forecast is statistically consistent • If selected forecast is NOT EQUIVALENT to distribution of obs => Statistically post-process forecast to improve consistency EXAMPLES CONTROL FCST ENSEMBLE 3
SOURCES OF STATISTICAL INCONSISTENCY • TOO FEW FORECAST MEMBERS • Single forecast – inconsistent by definition, unless perfect • MOS fcst hedged toward climatology as fcst skill is lost • Small ensemble – sampling error due to limited ensemble size (Houtekamer 1994? ) • MODEL ERROR (BIAS) • Deficiencies due to various problems in NWP models • Effect is exacerbated with increasing lead time • SYSTEMATIC ERRORS (BIAS) IN ANALYSIS • Induced by observations • Effect dies out with increasing lead time • Model related • Bias manifests itself even in initial conditions • ENSEMBLE FORMATION (INPROPER SPREAD) • Not appropriate initial spread • Lack of representation of model related uncertainty in ensemble • I. E. , use of simplified model that is not able to account for model related uncertainty 4
HOW TO IMPROVE STATISTICAL CONSISTENCY? • MITIGATE SOURCES OF INCONSISTENCY • TOO FEW MEMBERS • Run large ensemble • MODEL ERRORS • Make models more realistic • INSUFFICIENT ENSEMBLE SPREAD • Enhance models so they can represent model related forecast uncertainty • OTHERWISE => • STATISTICALLY ADJUST FCST TO REDUCE INCONSISTENCY • Unpreferred way of doing it • What we learn can feed back into development to mitigate problem at sources • Can have LARGE impact on (inexperienced) users 5
WHAT WE NEED FOR POSTPROCESSING TO WORK? • LARGE SET OF FCST – OBS PAIRS • Consistency defined over large sample – need same for post-processing • Larger the sample, more detailed corrections can be made • BOTH FCST AND REAL SYSTEMS MUST BE STATIONARY IN TIME • Otherwise can make things worse • Subjective forecasts difficult to calibrate HOW WE MEASURE STATISTICAL INCONSISTENCY? • MEASURES OF STATIST. RELIABILITY • • Time mean error Analysis rank histogram (Talagrand diagram) Reliability component of Brier etc scores Reliability diagram 6
CHARACTERISTICS OF ENSEMBLE POSTPROCESSING • TRUTH • Observations (arbitrary location) • Analysis field (usually gridded field) * • Feedback to ensemble system development • Further post-processing (“Downscaling”) is needed as separate step • COVARIANCES • Point-wise correction (spatiotemporal covariances ignored) * • Spatial/temporal covariances considered • Added level of complexity (need more data? ) • Downscaling can be integral part of it • REALIZATIONS • Individual fcst trajectories corrected * • Spatio-temporal & cross-variable correlations may be useful • Possibly realistic ensemble solutions for users (hydro, energy power, etc) • Only probability distributions corrected, no individual realizations • If users need trajectories, how to create them? 7
ENSEMBLE POSTPROCESSING METHODS • GENERAL PROCEDURE: • COMPARE CERTAIN PROPERTIES OF OBS. & FCSTS • Various choices related to different measures of inconsistency • Need obs – fcst data pairs • MAKE ADJUSTMENTS • Individual ensemble forecasts • Ensemble-based pdfs • SYSTEMATIC ERROR DEPENDS ON: • • Model / Initial perturbations Lead time Geographic location Season / Flow regime • SEEK BALANCE BETWEEN • NEED FOR LARGE SAMPLE FOR STABLE STATISTICS • Pool together as much data as possible • DESIRE FOR DETAILED CORRECTIONS • Subdivide sample to make corrections more specific • Use only most recent part of sample (if fcst system changes rapidly) 8
EXISTING/PROPOSED APPROACHES • STATISTICAL MOMENTS • Estimate time mean error – adjust each fcst accordingly (1 st moment correction) • Estimate time mean bias in spread (2 nd moment correction) • Adjust each member separately for both 1 st & 2 nd moments • CUMULATIVE DISTRIBUTIONS • Compare cumulative distributions (not only 1 st moment) • Need more data • DRESSING • Fit predetermined pdf on each member • ADJUST FCST PROBABILITY VALUES • Based on Reliability diagram, fcst prob = obs. frequency • Does not correct model bias • Sub-optimal , does not improve statistical resolution • OTHERS • Not necessarily designed for use with ensembles • Neural net, traditional MOS 9
BACKGROUND 10
BIAS CORRECTION / DOWNSCALING TWO GOALS: Adjust sample – ensemble time trajectories, covariances, only then Construct bias-corrected pdf for individual variables APPROACH (A) – Bias corrected anomalies on model grid, then downscaling 1) ESTIMATE BIAS: Compare time mean fields FIRST MOMENT SECOND MOMENT B = DIFFERENCE BETWEEN Ensemble mean forecast and Verifying analysis R = RATIO BETWEEN Ensemble mean error and Ensemble spread STATISTICAL SAMPLING (Increase sample size): Use data from surrounding grid-points (with Gaussian weighting) Use climate means if available and forecast system is stable Use most recent past data with decaying averaging otherwise Ability to quickly learn bias of new NWP systems before upgrade Adjust temporal/spatial sampling domain to optimize performance 2) REMOVE BIAS: Compare time mean fields 1 st moment = Ensemble mean - B 2 nd moment = Ensemble spread * R 11
12
13
14
15
BIAS CORRECTION / DOWNSCALING, APPROACH (A) 3) FORM FORECAST ANOMALIES: FIRST MOMENT 4) SECOND MOMENT Ai = DIFFERENCE BETWEEN Each ensemble forecast and Reanalysis climate mean SAi = RATIO BETWEEN Anomaly and Reanalysis Standard Deviation BIAS-CORRECTED STANDARDIZED ANOMALY FORECAST ON MODEL GRID Temporal/spatial resolution can degrade with lead time / loss of predictability 4) COMBINE ENSEMBLES FROM DIFFERENT CENTERS: Follow steps 1 -3 for each ensemble separately Determine weights for each ensemble based on error statistics (D. Unger) Combine anomalous ensemble forecasts (with weights) 5) DOWNSCALE: 6) 7) Add coarse resolution forecast anomaly to NDFD (or other local) climate distribution FIRST MOMENT SECOND MOMENT Forecast anomaly Plus Local climate mean Multiply Standardized Anomaly and Local climate standard deviation BIAS CORRECTED LOCAL FORECAST 16 Only climatology is stored at high resolution, anomaly forecast is on coarse grid
BIAS CORRECTION / DOWNSCALING APPROACH (B) BIAS CORRECTION DIRECTLY ON NDFD GRID (or local sites, not on model grid) METHODS SUGGESTED BY MDL NEURAL NETWORK APPLICATIONS: Input: Raw ensemble forecasts and lat, lon, elevation, climatology, etc Output: Bias corrected ensemble forecasts Penalty: BSS (or RPSS), assuring whole distribution is corrected ENSEMBLE TRANSFORM KALMAN FILTER (ETKF) APPLICATIONS INCREASE ENSEMBLE SIZE: Transform lagged (older) ensembles to possess same mean & spread as current ensemble How old ensembles are still useful? MANUAL FORECAST MODIFICATION: Transform operational ensemble to describe limited manually prescribed information Propagate signal in time, space, and across variables using transformed ensemble 17
CHALLENGE: PROVIDE OBJECTIVE GUIDANCE FOR NDFD NATIONAL DIGITAL FORECAST DATABASE • Seamless suite of forecasts across different time ranges (hours – seasons) • 15+ variables NEED PROBABILISTIC INFORMATION – ADD 2 nd MOMENT • Requirements: • 2. 5 x 2. 5 spatial grid • High temporal resolution (hours) TEMPORAL/SPATIAL SCALES OF PREDICTABLE SIGNAL INCREASES WITH LEAD TIME – ADD FORECAST SIGNAL ON REDUCED SPACE/TIME GRID TO CLIMATE (INCLUDING DAILY CYCLE) ON FULL GRID OBJECTIVE GUIDANCE FOR NDFD • Based on best available NWP products • 1 st moment - Ensemble mean? • 2 nd moment - Ensemble spread IMPERFECT MODEL/ENSEMBLE – BIAS CORRECTION / POSTPROCESSING MANUAL MODIFICATION OF OBJECTIVE FIRST GUESS • Focus on 1 st moment initially PROPAGATE INFO ACROSS TIME/SPACE/VARIABLES – ENSEMBLE-BASED TOOLS? 18 • 2 nd moment can be handled similarly later – NEED TRAINING
OBJECTIVE GUIDANCE FOR NDFD BIAS-CORRECT AND DOWNSCALE NWP ENSEMBLE GUIDANCE COMPARE MODEL GRID FORECAST & HIRES OBS DATA A) Based on NWP analysis and hires observational climatology ON MODEL GRID: Bias-correct forecast wrt reanalysis Express forecast as anomaly from reanalysis ON NDFD GRID (or any local data): Add forecast anomaly to NDFD (or observed) climatology Problem: Must construct NDFD climatology first Advantage: Works with predictable signal - simple, more flexible approach? B) Based on hires observational data ON NDFD GRID (or any local data points) Bias-correct forecast wrt observations or NDFD “analysis” Problem: How to interpolate between obs data points? Easier done for clim. mean? Advantage: Can potentially correct flow dependent systematic error 19
20
- Slides: 20