Ensemble Prediction Systems and Probabilistic Forecasting Overview for
Ensemble Prediction Systems and Probabilistic Forecasting
Overview for the Week • • • Introduction Building an ensemble prediction system Probability distributions Basic concepts of ensemble prediction systems Calibration, averaging and verification Ensemble prediction system products North American Ensemble Forecast System Putting it into practice Summary and evaluation
1920’s • Norwegian meteorologists come up with the concept of fronts to explain the weather. • Concept developed from visual observation of the weather. (Courtesy NOAA)
1940’s • Advent of weather balloons and aviation age introduces vertical dimension to observations. • Rossby shows that the motions in the atmosphere can be predicted using physics. • However the usefulness of these computer models would have to wait until computing power increased.
1960’s • Advent of satellite imagery. • Meteorologists now have the ability to remotely view the evolution of weather systems.
1980’s • Computer models using supercomputers comes of age. • Increasing sophistication in terms of the calculations and the scale of the computer models. • Deterministic (single) forecasts.
00’s Ensemble Prediction Systems (EPS) • Ensemble Prediction Systems are able to convey information about the uncertainty that is inherent in weather forecasting.
Background • Ensemble prediction system (EPS) have been around for a long time: – An Ensemble forecast can be thought of as a collection of two or more Numerical Weather Prediction (NWP) model forecasts verifying at the same time • An ensemble prediction system can • provide possibilities of pattern evolution. • partially assess confidence in a particular model solution.
Background • Types of ensembles have included – – different issue times, same verification time different models statistical ensembles opinions of different forecasters • Goal as been to produce deterministic forecast – rather than to communicate uncertainty • Decision-makers may not be able to use probabilistic information
information The Meteorology Knowledge Gap w o kn e w at wh The Gap what we tell Time
Background • Ensemble forecasts have evolved significantly over the past years: – Systematic approach to model uncertainty. – Perturbations as simulation of uncertainty. – Better simulation of uncertainties in forecast processes. – Increasing number of members. – Increasing resolution of members. • With Ensemble forecasts, it is possible to evaluate, forecast and express uncertainty.
How are probabilistic forecasts made using computers? Deterministic Forecast Possible range of error Starting point in time Several ‘members’ X Climatology time Forecast Uncertainty Laurence Wilson, 2005. On the Use of Ensembles in the Forecasting Process; 39 th CMOS Congress Vancouver, May 31 – June 3, 2005.
Background • Common uses of Ensemble forecasts: – ensemble mean as a deterministic forecast – clustering to produce a small set of forecast states characterized with the cluster mean – estimation of forecast skill – ensemble probability distributions – measures of uncertainty
The parable of the three meteorologists. Three meteorologists come to a river, they want to know if they can cross safely. (They cannot swim. ) © Lenny Smith
Three meteorologists wish to cross a river. They cannot swim! Each has a forecast of depth which indicates they will drown. Forecast 1 Forecast 2 Forecast 3 © Lenny Smith So they have an ensemble. Each of the 3 forecasts is a member of the ensemble.
They take the average of their 3 forecasts and call it the ensemble mean. Based on the ensemble mean they decide to cross the river. Ensemble mean Is this a good idea? © Lenny Smith
Background • Scope of this training on EPS: – Introduce basic principles of ensemble prediction systems – Provide training on the use and limitations of ensemble prediction systems – Move away from the deterministic paradigm toward a probabilistic paradigm – estimating and expressing uncertainty. Uncertainty is a part of every weather forecast and no forecast is complete without a description of its uncertainty
Eventual Outcome • Shift from a determinism to system where uncertainty is part of forecasts: – EPS can provide probability distributions for future weather quantities or events – Probabilistic forecasts allow one to quantify weatherrelated risks – Potential to show greater value than deterministic forecasts in decision-making • basic principle that weather forecasts have no intrinsic value
Cost / loss model « Weather forecasts possess no intrinsic value in an economic sense. They acquire value by influencing the behaviour of individuals or organizations whose activities are sensitive to weather » A. Murphy, 1994 • Forecasts (deterministic or probabilistic) acquire value when used.
Disclaimer • This is our first remote training workshop on EPS – more will be required – forecasters need training – decision-makers need to understand what is possible • Format is relatively new – this is a pilot project • Instructors are not professional trainers and have no expertise in tropical meteorology • many questions will remain unanswered • not clear how to fully incorporate EPS in the forecast process – it will take time
Conclusion • Weather is deterministic – forecasting is not • This uncertainty comes from data assimilation and modelling approximations and errors and from the chaotic nature of the atmosphere • If this uncertainty can be assessed, it should be part of the weather forecasts Probabilistic Forecasts
Conclusion Uncertainty is the only thing that is certain.
Part 2 – Building an Ensemble Prediction System
Building an Ensemble Forecast System one member …. single continuous variable A B two possibilities, actual result either higher or lower than forecast
Building an Ensemble Forecast System two members …. A C one of three possible outcomes within EPS envelope B
Building an Ensemble Forecast System three members …. A C D two of four possible outcomes within EPS envelope B
Building an Ensemble Forecast System five members …. A E F C 4 of 6 possible outcomes within EPS envelope D B
Building an Ensemble Forecast System n members …. A D, E, F, G, …. . P, Q, R, S, …. . n-1 of n+1 possible outcomes within EPS envelope B
Building an Ensemble Forecast System • For an ideal EPS, probability of verifying observation within ensemble envelope
Summary • Every weather forecast is probabilistic. • Probabilistic forecasts are a decision support tool. • Probabilistic forecasts describe the level of uncertainty in weather forecasts. • A good probabilistic forecast can indicate likely alternatives and thus assist in decision support. If one forecast is good, then 30 forecasts will be better! (but not 30 times better)
Summary • There is uncertainty in every forecast • No forecast is complete without an assessment of uncertainty • There are several interpretation of probabilities – the frequentist approach is widely used in meteorology • Probability is a measure and an expression of uncertainty Questions ?
Part 3 – Probability or frequency distributions
Probabilities • Probabilities are estimated from observed and/or forecast frequencies of occurrences of events: – Frequency distributions. • Probabilities can be estimated from: – Climatology distributions. – Ensemble prediction systems – Statistical adaptation. • Frequentist approach to the problem.
Frequency distribution P(16 ≤ T ≤ 18) = ? P(T > 19) = ? Total number of cases = 50 P(T < 15) = ? R. Verret, CMC
Frequency distribution P(16 ≤ T ≤ 18) = 22/50 = 44% P(T > 19) = 7/50 = 14% Total number of cases = 50 P(T < 15) = 10/50 = 20% R. Verret, CMC
Normal PDF (Probability Density Function) 50 % Probability % Both PDFs would produce the same deterministic forecast 20 % Temperature o. C 0 5 10
Gamma Distribution • Most likely value is near zero • What would a deterministic forecast predict? Probability • Consider a hypothetical distribution for convective precipitation amount 40 % 0 mm 50 mm Convective precipitation
Multimodal PDF Probability • Two possible solutions Weather Parameter
Part 4 – Basic Concepts
EPS basic concepts • Sources of error in NWP – initial conditions, model errors, boundary conditions – dependency on initial conditions – chaos • EPS principles • EPS and probabilistic forecasts – Spread-skill relationship – Probabilities • Member count, calibration, clustering, PDF – value of EPS. • Conclusions Uncertainty is part of forecasting…
Sources of error – uncertainties Sampling of current state of the atmosphere OBSERVATIONS Processing data to initialize models INITIALIZATION Projection forward in time NUMERICAL MODEL Trial field Uncertainty
Sources of error – uncertainties • Initial conditions related uncertainty: – Measurement errors inherent to the instruments – Improperly calibrated instruments • Systematic errors – bias • Random errors – Incorrect registration of observations – Data coding or transmission errors – Lack of coverage – incomplete information
Sources of error – uncertainties • Initial conditions related uncertainty: – Data assimilation errors: • Imperfect data quality control • Deficiencies in trial fields – the trial fields are usually 6 -h model forecasts • Unrepresentative observations and model error statistics • Deficiencies in the data assimilation scheme • Assimilation of accurate but unrepresentative data – observing scales which cannot be resolved in the model • upper air observation in a convective cloud • local, terrain-induced wind phenomena • observations in banded or convective precipitation
Sources of Error - Downscaling • in many applications, model grid converted to high resolution topography – example is US National Weather Service forecast production tool • For lower resolution EPS, this could create even larger errors • See also http: //www. spc. noaa. gov/exper/sref http: //www. comet. ucar. edu/nwp/RTMA (slides 19 to 22) http: //www. meted. ucar. edu/nwp/pcu 3/cases/etaanl/ (page 8) http: //www. meted. ucar. edu/nwp/pcu 1/ic 6/6_5 d. htm
Basic concepts • EPS simulates uncertainties inherent to NWP: – – Initial conditions Model simulations Boundary conditions (including surface forcings) Forecast errors that depend on circulation • The predictive aspect of the atmosphere is chaotic: – Predictability is flow dependent and depends on the initial conditions – Several model runs (with different initial conditions and/or model error characteristics and/or boundary conditions) – Explore different realistic realization of the atmosphere – Generate quantitative estimates of probabilities
Basic concepts • Expected qualities of an EPS: – Spread/skill relationship. – EPS realizations should include the truth. – Enough members to define pdf. • Costs for an EPS: – Real time constraints – outputs should be available as close as possible to deterministic models. – Members usually have lower resolution than deterministic models: • Hence, skill of each EPS member individually is expected to be lower than that of the higher resolution deterministic model (at least at the shorter lead times).
Basic concepts Final states Initial states Uncertainty on initial state Deterministic forecast Analysis Ensemble mean True initial state True final state R. Verret, N. Gagnon, CMC Climatology
EPS and probability forecasts • If there is a spread-skill relationship, it is possible to generate probabilistic forecasts from the EPS members: Low spread high probability of occurrence High spread low probability of occurrence
EPS and probability forecasts • The EPS must have enough members: – Members are sampling the underlying pdf. – In principle, the larger the ensemble, the better it is to establish pdf and probabilistic forecasts: • Large ensemble may bring in un-necessary probability resolution but definitely helps to define the pdf. • Depends on the meteorological variables. • Large ensemble required to explore the distribution tails – low probability events but with large impacts. • The larger the ensemble is, the larger the verification sample will have to be. • The ensemble is a compromise between resolution and number of members given the available computer resources. • Each member should have similar skill – member equiprobability. – In practice a few tens of member is considered enough generally.
EPS and probability forecasts • Clustering: – Allows to group members according to their similarity. – Creates sub-sets of possible scenarios. – Probability of occurrence is proportional to the number of members in each scenario. • PDF: – – Empirical PDF fitting. Measures of central tendencies and spread. Bayesian methods.
EPS and probability forecasts • Value of EPS is in probabilistic forecasts – probabilities and pdf. • Each member individually has little value per se compared to the deterministic model: – – – Lower quality and skill. Lower resolution. Initialized off perturbed initial conditions. Perturbed boundary conditions and physics. Each member may be subject to model jumpiness. • It is pointless to try to choose the member of the day: – Very difficult task particularly when the number of members is large. – Preferable in this case to go with the higher resolution deterministic model.
EPS and probability forecasts • The ensemble mean – It is likely to verify better than any individual member – And even better than the deterministic model – Provides wrong information in case of clustering or multi-modal distributions
EPS and probability forecasts Suppose a three member ensemble… All members agree on a 50 mm precipitation event that will last 24 hours. However, they are 12 -h out of phase on the timing of the event. The ensemble mean also predicts a 50 mm precipitation event, but… It spreads the event over a 48 -h period rather than a 24 -h one. It also fails to capture the maximum precipitation rate. R. Verret, CMC
EPS and probability forecasts Suppose a 20 -member ensemble with the following frequency histogram for temperature: • Bi-modal distribution. • The ensemble mean is 11. 35 – amongst the least probable events R. Verret, CMC
The Ensemble Mean • The ensemble mean is only a “good” forecast for verification • Use the ensemble mean if your goal is to be “least wrong” • Use a deterministic forecast if you want to be “right” • Use the entire distribution if you want to maximize information for decision-making
Summary of Part 4 • errors in EPS are similar to those for deterministic NWP models • decision-makers (such as forecasters) need to consider the entire distribution • errors are likely greater in the tropics • beware of the ensemble mean • more “working in the tails” of the probability distribution
- Slides: 57