Probabilistic Weather Forecasting Using Bayesian Model Averaging J

Probabilistic Weather Forecasting Using Bayesian Model Averaging J. Mc. Lean Sloughter Adviser: Tilmann Gneiting GSR: Susan Joslyn Committee members: Adrian Raftery & Cliff Mass 8 May, 2009 This work was supported by MURI, JEFS, & NSF grants

n Background Motivation n Ensemble forecasting n Bayesian model averaging Dissertation outline BMA for vector wind n Data n Decomposing the problem n Bias-correction n Error distributions n Model n Results Future directions References Acknowledgements n n n 2

Why probabilistic forecasting? n Situations where certain ranges or thresholds are of interest n Situations where knowing not just the most likely outcome, but possible extremes are important n Situations involving a cost / loss analysis, where probabilities of different outcomes need to be known n Examples: n n n Wind energy Military Sailing Airports Winter road maintenance 3

Ensemble Forecasting 48 -hour forecasts for maximum wind speeds on 7 August 2003 4

Ensemble Forecasting n Single forecast model is run multiple times with different initial conditions n Forecasts created on a 12 -km grid, and bilinearly interpolated to locations of interest n Ensemble mean tends to outperform individual members n Spread-skill relationship: spread of forecasts tends to be correlated with magnitude of error 5

Ensemble Forecasting n Would like the ensemble to look like draws from the same distribution as the observed values n Ensemble only captures one source of variability – uncertainty in initial conditions n Ensemble distribution is underdispersed relative to observed values n Ensemble members agree with one another more than they agree with observations 6

Bayesian model averaging (BMA) n Weighted average of multiple component models n One component per ensemble member n Each component a distribution of observed value conditioned on an ensemble member forecast n Model fit based on training data – past sets of forecasts / observations n Use a sliding window of training data n Weights determined by how well each member fits the training data 7

Bayesian model averaging where is the deterministic forecast from member k, is the weight associated with member k, and is the estimated density function for y given member k 8

n Background Motivation n Ensemble forecasting n Bayesian model averaging Dissertation outline BMA for vector wind n Data n Decomposing the problem n Bias-correction n Error distributions n Model n Results Future directions References Acknowledgements n n n 9

Dissertation Outline n Precipitation forecasting Sloughter et al. , 2007, MWR n Extends BMA to a specific case of skewed and noncontinuous distributions n Wind speed forecasting n Sloughter et al. , 2009, JASA n Extends methods of Sloughter et al. (2007) to other forms of skewed and non-continuous distributions n Examines robustness of BMA to details of model selection n Vector wind forecasting n This talk n Extends BMA to multivariate distributions n 10

n Background Motivation n Ensemble forecasting n Bayesian model averaging Dissertation outline BMA for vector wind n Data n Decomposing the problem n Bias-correction n Error distributions n Model n Results Future directions References Acknowledgements n n n 11

BMA for vector wind n Methods exist for using Bayesian Model Averaging to create probabilistic forecasts for weather quantities that can be expressed as a mixture of normals (Raftery et al. , 2005), such as temperature and pressure. n Expanded to be applied to non-continuous and skewed quantities such as precipitation and wind speed in Sloughter et al. 2007, Sloughter et al. 2009. n A method is needed for modeling multivariate quantites such as wind vectors. 12

Knot n A knot is a measure of speed used in n n nautical, meteorological, and aviation settings 1. 852 kilometers per hour 1. 151 miles per hour 0. 514 meters per second Sailors would throw out the chip log (a board designed to stay stationary in water) tied to a rope with knots spaced 7 fathoms (42 feet) apart They would then count how many knots were fed out in 30 seconds 13

Knot n 4 -6 knots is a light breeze – leaves move, breeze can be felt on one’s face n 11 -16 knots is a moderate breeze – dust and paper will be blown about, whitecaps will form on the water n 20 -21 knots is generally the threshold for issuing a small craft advisory n 34 -40 knots is a gale – small branches break from trees, walking becomes difficult 14

Data n This work uses wind data from the Pacific Northwest n n n for the full year 2003, plus November and December 2002 (results for 2003 data, 2002 used only for training) “Instantaneous” vector wind measurements Measured in knots Each forecast consists of 8 ensemble members Data were available for 343 days, missing for 83 days A total of 38091 observations, averaging 111 observations per day All work that follows is based on 30 -day training periods, with 2 -day-ahead forecasting 15

Data n Data from Surface Airway Observation stations n Airports in BC, Washington, Oregon, Idaho, and California 16

Decomposing the problem n Wind has two dimensions, east/west direction and north/south direction n BMA uses a mixture distribution with one component per ensemble member n Consider each mixture component a bivariate distribution parameterized in terms of a mean vector and a covariance matrix n Assume that the mean of the distribution is some function of the forecast vector, and that the covariance matrix does not depend upon the forecast (exploratory plots support these assumptions) 17

Decomposing the problem n h(fk) is the mean (a bias-corrected forecast) n BV(0, Q) is the distribution of the forecast error n Model the distribution of the errors rather than the observed values n Has the advantage of having constant parameters across forecast values n Can then be decomposed into two separate problems: n n bias-correcting the forecast modeling the error distribution 18

Bias-correction n For simplicity, consider affine bias corrections n Two potential forms of bivariate bias correction n Additive bias-correction n Full affine bias-correction n Where Y is the observed wind vector, fk is the kth vector forecast, ak is an additive bias vector, and Bk is a transformation matrix 19

Bias-correction Bivariate root mean squared error (in knots) for one ensemble member n Out-of-sample results using 30 -day training period n Similar results hold for other ensemble members n Affine bias-correction shows a marked improvement 20

Error distributions n Now deal with the error field (observations minus bias-corrected forecasts) n Exploratory work suggests that the distributions are ellipsoidal, but have heavier tails than normal distributions n Transform the error vector (rkcosqk, rksinqk)T by raising the magnitude of the vector to the 4/5 power while preserving the angle n Model this as a bivariate normal distribution 21

Model n Thus, our final model is: n Where the gk are the distributions on y implied by the distributions of the transformed error vectors n Model parameters are estimated globally using all observation locations 22

Model n Bias-correction fit via linear regression (separate bias correction for each mixture component) n Weights and covariance matrix estimated via maximum likelihood using the EM algorithm n Use latent variables zkst which are indicators that forecast k was the best forecast at station s at time t 23

Model n E step: n M step: 24

Model n M step (continued) 25

Results n We simulate a large number of forecasts from our distribution n Can evaluate the forecast of either the wind vector or derived quantities (marginal speed or direction) from the empirical distribution of our forecasts n Essentially creating a new, larger ensemble of forecasts that should be better-calibrated than the original ensemble 26

Example n To illustrate what the BMA distribution is doing, consider the case of forecasting at Omak, Washington on February 4 th, 2003 27

28

Results n Our goal is to maximize sharpness subject to calibration (the Gneiting principle) n By calibration, we mean that we want our probability distribution function to be correct – if we forecast an event as happening with probability. 9, we want it to happen 90% of the time n By sharpness, we mean that we want predictive intervals to be as narrow as possible 29

Results n For univariate quantities, the verification rank histogram is a tool that can be used to assess the calibration of an ensemble forecast n Find the rank of each forecast relative to the ensemble members n If the ensemble is properly calibrated, the observation and forecasts should be interchangeable n If so, each potential rank of the forecast should have equal probability n Thus, a histogram of the ranks should look flat 30

Results n For multivariate quantities, there is an analogous multivariate rank histogram (MVRH), again based on the assumption of exchangeability n Define if and only if in every dimension n For each member of the combined set of the observation and the forecasts, find the pre-rank n The multivariate rank is the rank of the observation pre-rank, with any ties resolved at random n If we have a set of 8 forecasts and 1 observation, there are 9 possible rankings of the observation relative to the forecasts 31

Results n MVRH for the raw ensemble (left) and BMA forecast distribution (right) n Raw ensemble is under-dispersed n BMA forecast distribution is much better-calibrated 32

Results n The energy score (ES) is a scoring rule for multivariate probabilistic forecasts that takes into account both calibration and sharpness n In the univariate case, it reduces to the continuous ranked probability score (CRPS) n P is the predictive distribution, x the observed wind vector, X and X’ independent random variables with distribution P 33

Results n There may still be interest in a point forecast as well n We can use the spatial median as a point forecast n We can assess the quality of a multivariate point forecast using the multivariate mean absolute error (MMAE) 34

Results n BMA outperforms climatology and the raw ensemble both in terms of the probabilistic forecast and the deterministic forecast 35

Results – marginal speed and direction n Again consider verification rank histograms to assess calibration n Both speed (top) and direction (bottom) are much improved by BMA 36

Results – marginal speed and direction n CRPS is the scalar equivalent of the energy score n DCRPS is the angular equivalent n Scalar point forecasts can be assessed by the MAE, and angular point forecasts by the mean directional error (MDE) n Can also look at coverage and width of 77. 8% prediction intervals for scalar forecasts – coverage assesses calibration, width assesses sharpness 37

Results – marginal speed and direction n Wind speed: n Wind direction: 38

Results – marginal speed and direction n We can see that for both speed and direction, BMA improves the quality of both the probabilistic and deterministic forecasts n BMA produces marginal distributions that are better-calibrated than the raw ensemble and sharper than climatology 39

n Background Motivation n Ensemble forecasting n Bayesian model averaging Dissertation outline BMA for vector wind n Data n Decomposing the problem n Bias-correction n Error distributions n Model n Results Future directions References Acknowledgements n n n 40

Future Directions n Develop a BMA method to explicitly model marginal instantaneous wind speed and compare to the performance of the forecasts from this model (current BMA for marginal wind speed is for maximum wind speeds, not instantaneous) n Incorporate spatial information, either through explicitly modeling some spatial structure to our parameters or by estimating parameters locally rather than globally n Investigate using an exponential forgetting for training data rather than a sliding window, which could allow for faster computation through the use of updating formulae for parameter estimates n Extend multivariate methods to jointly forecast multiple weather quantities simultaneously 41

References n Raftery, A. E. , Gneiting, T. , Balabdaoui, F. and Polakowski, M. (2005). Using Bayesian Model Averaging to calibrate forecast ensembles. Monthly Weather Review, 133, 1155 -1174. n Sloughter, J. M. , Raftery, A. E. , Gneiting, T. and Fraley, C. (2007). Probabilistic quantitative precipitation forecasting using Bayesian model averaging. Monthly Weather Review, 135, 3209 -3220. n Sloughter, J. M. , Gneiting, T. , and Raftery, A. E. (2009). Probabilistic wind speed forecasting using ensembles and Bayesian model averaging. Journal of the American Statistical Association, accepted. n Mass, C. , Joslyn, S. , Pyle, P. , Tewson, P. , Gneiting, T. , Raftery, A. , Baars, J. , Sloughter, J. M. , Jones, D. , and Fraley, C. (2009). PROBCAST: A web-based portal to mesoscale probabilistic forecasts. Bulletin of the American Meteorological Society, in press. n http: //probcast. com 42

Acknowledgements n Committee: n Tilmann Gneiting - adviser n Adrian Raftery, Cliff Mass - committee members n Susan Joslyn - GSR n Statistics folks: n Veronica Berrocal, Chris Fraley, Thordis Thorarinsdottir, Will Kleiber, Larissa Stanberry, Matt Johnson, Robert Yuen, Michael Polakowski, Nicholas Johnson n Atmospheric Sciences folks: n Jeff Baars, Eric Grimit, Jeff Thomason, Tony Eckel n APL folks: n Patrick Tewson, John Pyle, David Jones, Janet Olsonbaker, Scott Sandgathe n Psychology folks: n Limor Nadav-Greenberg, Buzz Hunt, Queena Chen, Jared Le Clerc, Rebecca Nichols, Sonia Savelli 43