Alternative Forecasting Methods Bootstrapping Bryce Bucknell Jim Burke

Alternative Forecasting Methods: Bootstrapping Bryce Bucknell Jim Burke Ken Flores Tim Metts

Agenda Scenario Obstacles Regression Model Bootstrapping Applications and Uses Results

Scenario You have been recently hired as the statistician for the University of Notre Dame football team. You are tasked with performing a statistical analysis for the first year of the Charlie Weis era. Specifically, you have been asked to develop a regression model that explains the relationship between key statistical categories and the number of points scored by the offense. You have a limited number of data points, so you must also find a way to ensure that the regression results generated by the model are reliable and significant. Problems/Obstacles: n n Central Limit Theorem Replication of data Sampling Variance of error terms

Constrained by the Central Limit Theorem In selecting simple random samples of size_n from a population, the sampling distribution of the sample mean x can be approximated by a normal probability distribution as the sample size becomes large. It is generally accepted that the sample size must be 30 or greater to satisfy the large-sample condition of theorem. Sample N = 1 Sample N = 2 Sample N = 3 Sample N = 4 1. http: //www. statisticalengineering. com/central_limit_theorem_(summary). htm

Central Limit Theorem Central Limit theorem is the foundation for many statistical procedures, because the distribution of the phenomenon under study does NOT have to be Normal because its average WILL tend to be normal. Why is the assumption of a normal distribution important? n n n A normal distribution allows for the application of the empirical rule – 68%, 95% and 99. 7% Chebyshev’s Theorem no more than 1/4 of the values are more than 2 standard deviations away from the mean, no more than 1/9 are more than 3 standard deviations away, no more than 1/25 are more than 5 standard deviations away, and so on. The assumption of a normally distributed data allows descriptive statistics to be used to explain the nature of the population

Not enough data available? Monte Carlo simulation, a type of spreadsheet simulation, is used to randomly generate values for uncertain variables over and over to simulate a model. n n n Monte Carlo methods randomly select values to create scenarios The random selection process is repeated many times to create multiple scenarios Through the random selection process, the scenarios give a range of possible solutions, some of which are more probable and some less probable As the process is repeated multiple times, 10, 000 or more, the average solution will give an approximate answer to the problem The accuracy can be improved by increasing the number of scenarios selected

Sampling without Replacement Simple Random Sampling n n A simple random sample from a population is a sample chosen randomly, so that each possible sample has the same probability of being chosen. In small populations such sampling is typically done "without replacement“ Sampling without replacement results in deliberate avoidance of choosing any member of the population more than once This process should be used when outcomes are mutually exclusive, i. e. poker hands

Sampling with Replacement n n n Initial data set is not sufficiently large enough to use simple random sampling without replacement Through Monte Carlo simulation we have been able to replicate the original population Units are sampled from the population one at a time, with each unit being replaced before the next is sampled. One outcome does not affect the other outcomes Allows a greater number of potential outcomes than sampling without replacement If observations were not replaced there would not be enough independent observations to create a sample size of n ≥ 30

X Homoscedasticity – constant variance n n n Residuals Hetroscedasticity vs. Homoscedasticity X Hetroscedasticity – nonconstant variance All random variables have the same finite variance n Simplifies mathematical and computational treatment n Leads to good estimation results in data mining and regression n n Random variables may have different variances Standard errors of regression coefficients may be understated T-ratios may be larger than actual More common with cross sectional data

Regression Model For ND Points Scored ND Points = 38. 54 + 0. 079*b 1 - 0. 170*b 2 - 0. 662*b 3 - 3. 16*b 4 b 1 = Total Yards Gained b 3 = Total Plays b 2 = Penalty Yards b 4 = Turnovers

4 Checks of a Regression Model 1. Do the coefficients have the correct sign? 2. Are the slope terms statistically significant? 3. How well does the model fit the data? 4. Is there any serial correlation?

4 Checks of a Regression Model 1. Do the coefficients have the correct sign? Could this represent a big play factor?

4 Checks of a Regression Model 2. Are the slope terms statistically significant? 3. How well does the model fit the data? 4. Is there any serial correlation?

4 Checks of a Regression Model 3. How well does the model fit the data? Adjusted R 2 = 74. 22%

4 Checks of a Regression Model 4. Is there any serial correlation? Data is cross sectional With limited data points, how useful is this regression in describing how well the model fits the actual data? Is there a way to tests its reliability?

How to test the significance of the analysis What happens when the sample size is not large enough (n ≥ 30)? Bootstrapping is a method for estimating the sampling distribution of an estimator by resampling with replacement from the original sample. n n Commonly used statistical significance tests are used to determine the likelihood of a result given a random sample and a sample size of n. If the population is not random and does not allow a large enough sample to be drawn, the central limit theorem would not hold true Thus, the statistical significance of the data would not hold Bootstrapping uses replication of the original data to simulate a larger population, thus allowing many samples to be drawn and statistical tests to be calculated

How It Works Bootstrapping is a method for estimating the sampling distribution of an estimator by resampling with replacement from the original sample. n n n The bootstrap procedure is a means of estimating the statistical accuracy. . . from the data in a single sample. Bootstrapping is used to mimic the process of selecting many samples when the population is too small to do otherwise The samples are generated from the data in the original sample by copying it many number of times (Monte Carlo Simulation) Samples can then selected at random and descriptive statistics calculated or regressions run for each sample The results generated from the bootstrap samples can be treated as if it they were the result of actual sampling from the original population

Characteristics of Bootstrapping Sampling with Replacement Full Sample

Bootstrapping Example Original Data Set Limited number of observations Random sampling with replacement can be employed to create multiple independent samples for analysis 1 st Random Sample Pittsburgh Navy Michigan Ohio State Michigan State USC Washington Purdue Ohio State USC BYU 109 Copies of each observation USC BYU Tennessee Stanford Navy Pittsburgh Syracuse Stanford Ohio State Creating a much larger sample with which to work Ohio State Stanford Michigan

When it should be used Bootstrapping is especially useful in situations when no analytic formula for the sampling distribution is available. n n n Traditional forecasting methods, like exponential smoothing, work well when demand is constant – patterns easily recognized by software In contrast, when demand is irregular, patterns may be difficult to recognize. Therefore, when faced with irregular demand, bootstrapping may be used to provide more accurate forecasts, making some important assumptions…

Assumptions and Methodology n Bootstrapping makes no assumption regarding the population n No normality of error terms n No equal variance n Allows for accurate forecasts of intermittent demand n n If the sample is a good approximation of the population, the sampling distribution may be estimated by generating a large number of new samples For small data sets, taking a small representative sample of the data and replicating it will yield superior results

Applications and Uses Criminology n n n Statistical significance testing is important in criminology and criminal justice Six of the most popular journals in criminology and criminal justice are dominated by quantitative methods that rely on statistical significance testing However, it poses two potential problems: tautology and violations of assumptions

Applications and Uses Criminology n n n Tautology: the null hypothesis is always false because virtually all null hypothesis may be rejected at some sample size Violation of assumptions of regression: errors are homogeneous and errors of independent variables are normally distributed Bootstrapping provides a user-friendly alternative to cross-validation and jackknife to augment statistical significance testing

Applications and Uses Actuarial Practice Process of developing an actuarial model begins with the creation of probability distributions of input variables n Input variables are generally asset-side generated cash flows (financial) or cash flows generated from the liabilities side (underwriting) n Traditional actuarial methodologies are rooted in parametric approaches, which fit prescribed distribution of losses to the data n

Applications and Uses Actuarial Practice n n However, experience from the last two decades has shown greater interdependence of loss variables with asset variables Increased complexity has been accompanied by increased competitive pressures and more frequent insolvencies There is a need to use nonparametric methods in modeling loss distributions Bootstrap standard errors and confidence intervals are used to derive the distribution

Applications and Uses Classifications Used by Ecologists n n Ecologists often use cluster analysis as a tool in the classification and mapping of entities such as communities or landscapes However, the researcher has to choose an adequate group partition level and in addition, cluster analysis techniques will always reveal groups Use bootstrap to test statistically for fuzziness of the partitions in cluster analysis Partitions found in bootstrap samples are compared to the observed partition by the similarity of the sampling units that form the groups.

Applications and Uses Human Nutrition Inverse regression used to estimate vitamin B 6 requirement of young women n Standard statistical methods were used to estimate the mean vitamin B-6 requirement n Used bootstrap procedure as a further check for the mean vitamin B-6 requirement by looking at the standard error estimates and confidence intervals n

Application and Uses Outsourcing n n Agilent Technologies determined it was time to transfer manufacturing of its 3070 in-circuit test systems from Colorado to Singapore Major concern was the change in environmental test conditions (dry vs humid) Because Agilent tests to tighter factory limits (“guard banding”), they needed to adjust the guard band for Singapore Bootstrap was used to determine the appropriate guard band for Singapore facility

An Alternative to the bootstrap Jackknife n n A statistical method for estimating and removing bias* and for deriving robust estimates of standard errors and confidence intervals Created by systematically dropping out subsets of data one at a time and assessing the resulting variation Bias: A statistical sampling or testing error caused by systematically favoring some outcomes over others

A comparison of the Bootstrap & Jackknife n Bootstrap n n Yields slightly different results when repeated on the same data (when estimating the standard error) Not bound to theoretical distributions n Jackknife n n Less general technique Explores sample variation differently Yields the same result each time Similar data requirements

Another alternative method Cross-Validation n The practice of partitioning data into a sample of data into sub-samples such that the initial analysis is conducted on a single sub-sample (training data), while further sub-samples (test or validation data) are retained “blind” in order for subsequent use in confirming and validating the initial analysis

Bootstrap vs. Cross-Validation n Bootstrap n n Requires a small of data More complex technique – time consuming n Cross-Validation n Not a resampling technique Requires large amounts of data Extremely useful in data mining and artificial intelligence

Methodology for ND Points Model n n Use bootstrapping on ND points scored regression model Goal: determine the reliability of the model Replication, random sampling, and numerous independent regression Calculation of a confidence interval for adjusted R 2

Bootstrapping Results R 2 Data Sample # Adjusted R^2 1 0. 7351 13 0. 7482 2 0. 7545 14 0. 8719 3 0. 7438 15 0. 7391 4 0. 7968 16 0. 9025 5 0. 5164 17 0. 8634 6 0. 6449 18 0. 7927 7 0. 9951 19 0. 6797 8 0. 9253 20 0. 6765 9 0. 8144 21 0. 8226 10 0. 7631 22 0. 9902 11 0. 8257 23 0. 8812 12 0. 9099 24 0. 9169 The Mean, Standard Dev. , 95% and 99% confidence intervals are then calculated in excel from the 24 observations

Bootstrapping Results R 2 Data Mean: STDEV: 0. 8046 0. 1131 Conf 95% Conf 99% 0. 0453 or 75. 93 - 84. 98% 0. 0595 or 74. 51 - 86. 41% So what does this mean for the results of the regression? Can we rely on this model to help predict the number of points per game that will be scored by the 2006 team?

Questions?