Testing understanding of forecast uncertainty in the Experimental

Testing understanding of forecast uncertainty in the Experimental Economics Laboratory Todd R. Kaplan and Mark S. Roulston Introduction Discipline Meteorologists are well aware that weather forecasts are uncertain, with some of that uncertainty deriving from the atmosphere’s chaotic nature. Motivated by this awareness, the meteorological community has developed “ensemble forecasting, ” in which multiple simulations of the atmosphere are made to determine the sensitivity of the forecast to uncertainty in the initial condition and model structure. Ensemble forecasting is now an established tool at medium-range horizons of 3 to 14 days, and is also becoming increasingly useful for short-range forecasts up to 2 days ahead. Meteorologists also understand that ensemble forecasting yields the type of information about forecast uncertainty that can, in theory, enhance the economic value of forecasts by improving the decision making of forecast users (AMS, 2002; NRC, 2006). It is not clear, however, how much of this enhanced value will be realised because we do not know how well forecast users are able to understand forecast uncertainty information or forecasts in a probabilistic format. This is particularly true for users of free-at-the-point-of-use channels for disseminating weather forecasts, such as TV and the Internet. The value of a weather forecast derives from its ability to influence decisions that are made in the face of uncertainty. The question of how people make decisions under uncertainty is one of the principal research themes in the field of experimental economics (Kagel and Roth, 1995). The methods developed by experimental economists to study individual choice are useful tools for objectively determining how well users understand weather forecasts (Roulston et al. 2006). In this study experimental economic methods have been used to objectively evaluate whether forecast users can understand a specific format for presenting uncertainty in 5 -day temperature forecasts for a specific location. This type of forecast would be suitable for use on websites, local newspapers and possibly forecasts delivered on local television. Method Experiments were conducted at the Finance and Economics Experimental Laboratory at Exeter University (FEELE). Undergraduates studying a range of subjects participated in a sequence of 20 lotteries in which they could choose whether they would prefer to receive £ 0. 50 (about $1. 00) if one of two criteria concerning midday temperatures was satisfied. Examples of two of these lotteries are shown below. Students in group A were shown a deterministic (or “point” forecast) with no uncertainty information to help them make their decisions, whereas students in group B were presented with a forecast that included information about forecast uncertainty. After each lottery students were informed of the actual temperatures and whether either of the criteria had been satisfied. At the end of the experiment students were paid their lottery winnings in addition to a £ 5. 00 payment for participating. Format A Format B Round 12: Would you prefer to receive £ 0. 50 if… 1: The temperature at Midday on Monday is below 7ºC or if 2: The temperature at Midday on Wednesday is below 6ºC 1: The temperature at Midday on Sunday is above 15ºC or if 2: The temperature at Midday on Wednesday is above 14ºC In 14 of the 20 lotteries a hypothetical participant who assumed that the uncertainty associated with the deterministic forecast (format A) was the same at all forecast times would identify the same option as being most probable as a participant with access to the uncertainty information (format B). In the remaining 6 lotteries the assumption of constant uncertainty would lead to the hypothetical group A participant making a different choice to a fully informed group B participant. We will refer to these 6 questions as “swing” questions. The round 15 lottery above is an example where both participants would make the same choice, while round 12 is a swing question. School of Business and Economics University of Exeter Streatham Court, Rennes Drive, Exeter, EX 4 4 PU, UK Email: t. r. kaplan@ex. ac. uk Male Business/Econ Female Business/Econ Order of options* normal Format A Format B 13 15 normal 7 5 Male reversed 14 12 Business/Econ Female reversed 3 4 Science/Engr Male normal 16 15 Science/Engr Female normal 4 5 Humanities Male normal 9 11 Humanities Female normal 11 9 A total of 153 undergraduates participated in the experiments. A summary of the numbers in the different treatment groups is shown in the table on the left. * normal ordering of the two options was with the option referring to earlier time given first (as in the examples opposite). Results Group Business/Econ Science/Engr Humanities Male Female Swing questions Non-Swing questions Overall % correct (Format A) 69. 6 68. 5 66. 5 69. 3 66. 8 24. 5 87. 4 68. 5 % correct (Format B) 85. 7 85. 8 83. 8 85. 8 83. 7 76. 3 89. 0 85. 2 A summary of the results of the experiments. A “correct” response was one in which the participant chose the most probable outcome. Note that irrespective of how the participants are segmented, those with uncertainty information outperformed those without. Probit regression: Estimating chance of getting a question correct. The following probit model was fitted to the data. It predicts the probability that a participant will respond correctly as a function of a collection of predictor variables. Where Φ is the cumulative distribution function of the standard normal distribution. The list of predictors with the corresponding coefficients found by fitting the model are listed in the table below. All the predictors, except “Question Number”, are dummy variables that are either equal to 0 or 1. § d. F/dx is the change in probability. For instance, British Predictor Coeff d. F/dx P-value participants were 12% more Question Number 0. 0060 0. 0017 0. 236 likely to get a question correct. Reverse order -0. 0871 -0. 0247 0. 295 § Participants unable to answer a simple test question about Early Correct -0. 2446 -0. 0699 0. 003 probability were 3. 6% less likely Early Correct & Format B -0. 1560 -0. 0451 0. 180 to get the lottery questions correct. Format B 0. 0244 0. 0068 0. 785 § Swing questions hurt chances Swing Question -0. 5287 -0. 1587 0. 000 by 15. 9% overall. For subjects Swing Question & Format A -1. 3672 -0. 4761 0. 000 using format A this increased to 63. 5%! Gender is Male 0. 0344 0. 0096 0. 587 Humanities -0. 1478 -0. 0422 0. 086 § From Early Correct, there was a bias to choose the option Science/Engineering -0. 0906 -0. 0256 0. 254 referring to the later day. Nationality is British 0. 3842 0. 1207 0. 001 § Being a humanities student hurt chances by 4. 2% on average, Prob Question Mistake -0. 1269 -0. 0361 0. 082 but humanities students with Sample Question Mistake -0. 0169 -0. 0047 0. 793 uncertainty information Constant 0. 9932 0. 000 outperformed those without. § Gender is not significant: it is captured in field of study. This Significant at the 1% level Significant at the 10% level can happen since females are roughly 1/3 of all subjects but 1/2 of all humanities subjects. Summary Round 15: Would you prefer to receive £ 0. 50 if… Todd R. Kaplan Business/Econ Gender On average participants who were given uncertainty information made significantly better decisions than those without. This improvement occurred irrespective of gender or field of study. References AMS Council, 2002. Enhancing weather information with probability forecasts. Bulletin of the American Meteorological Society, 83, 450– 452 Kagel, J. H. , and A. E. Roth (eds. ), 1995. Handbook of Experimental Economics. Princeton, NJ : Princeton University Press, 721 pp. National Research Council, 2006. Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts, Washington, DC : National Academies Press, 124 pp. Roulston, M. S. , G. E. Bolton, A. N. Kleit, and A. L. Sears-Collins, 2006. A laboratory study of the benefits of including uncertainty information in weather forecasts. Weather and Forecasting, 21, 116– 122 Mark S. Roulston Met Office Fitz. Roy Road Exeter, EX 1 3 PB, UK Email: mark. roulston@metoffice. gov. uk © Crown copyright 2007 07/0134 Met Office and the Met Office logo are registered trademarks