Bootstrap Test for Pairs of Means of a

Bootstrap Test for Pairs of Means of a Non-Normal Population – small samples • Suppose X 1, …, Xn are iid from some distribution independent of Y 1, …, Ym are iid from another distribution. Further suppose that both n and m are small and we are interested in testing whether the two populations have the same means. • Can use the t-test (pooled or unpooled) since it is robust as long as there are no extreme 1 outliers and skewness. • Alternatively, we can use bootstrap hypothesis testing. STA 248 week 12 1

Bootstrap Hypothesis Testing - Introduction • Suppose X 1, …, Xn is a random sample of size n, independent from another random sample Y 1, …, Ym of size m. and we wish to test vs . • As a test statistics we will use . • The P-values of this test is . • We want the bootstrap estimate of this P-value. STA 248 week 12 2

Bootstrap Test Procedure • To obtain the bootstrap estimate of the P-value we need to generate samples with H 0 true. • One way of doing this (assuming X and Y have same distribution) is to combine 2 samples into 1 of size n+m. • Then re-sample with replacement from this combined sample such that each re-sampling has two groups … • For each bootstrap sample calculate the bootstrap estimate of the test statistics , j = 1, …, B. • The bootstrap estimate of the P-value is …. STA 248 week 12 3

Example STA 248 week 12 4

Data Collection • There are three main methods for collecting data. Ø Observational studies Ø Sample survey Ø Planned / designed experiments • These methods differ in the strength of conclusion that can be drawn. STA 248 week 12 5

Observational Studies • In some cases, a study may be undertaken retrospectively. • In observational studies we simply collect information about variables of interest without applying any intervention or controlling for any factors. • When factors are not controlled we are not able to infer a causeeffect relationship. • Other problems with observation studies are: Ø Confounding – can’t separate effect of one variable from another. Ø Lack of generalization. STA 248 week 12 6

Sample Surveys • • Sample surveys are observational in nature. Surveys require existence of physically real population. Data is collected on a random sample from the target population. Survey design includes selection of sample so it is representative of the population as a whole. Use statistics to make inference about entire population. Confounding is still a problem. However, the results can be generalized to the population. Cause of any observed differences cannot be determined. To allow generalization and to avoid bias – sample must be chosen randomly e. g. , SRS. STA 248 week 12 7

Planned / Designed Experiments • There are few key features of designed experiments that distinguish it from any other type of study. • Independent variables of interest are carefully controlled by the experimenter in order to determine their effect on a response (dependent) variable. • Researcher randomly assign a treatment to the subjects or experimental units. • Control of independent variables and randomization make it possible to infer cause and effect relationship. • Use of replication – multiple observation per treatment. • Replication allows measurement of variability. STA 248 week 12 8

• Treatments are sometimes called predictor variables and sometimes called “factors”. • The values of a factor are its “levels”. • A design is balanced if each treatment has the same number of experimental units. • Problem: can’t always carry out an experiment. STA 248 week 10 9

Randomization • The use of randomization to allocate treatments to experimental units (or vice versa) is the key element of well-designed experiment. • Random allocation tends to produce subgroups which are comparable with respect to the variables known to influence the response. • Randomization ensures that no bias is introduced in allocation of treatments to experimental units. • Randomization reduces the possibility that factors not included in the design will be confounded with treatment. STA 248 week 12 10

Cautions Regarding Experiments • “Effective sample size” – all statistical techniques we have learned assume observations are independent. If they are not but treated as if they were, get more power and smaller CI than you should. • “Fishing expedition” – if doing 100 tests at α = 0. 05 significant level, expect 5 of 100 tests to show significant differences from H 0 even when H 0 is always true (type I errors). STA 248 week 10 11

Controlling for Type I error • One widely use method for controlling for type I error uses Bonferoni Inequality…. • If Ai is the event that the ith test has a type I error, and typically P(Ai ) = α, then by Bonferoni Inequality we that: . . That is the probability of committing at least one type I error in k tests is at most kα. • Therefore, if use significant level of α/k for each individual test, then the “overall significant level” (P(at least 1 type I error)) is at most α. • The Bonferoni method is very conserevative. STA 248 week 10 12

Analysis of Variance – Introduction • Generalization of the two sample t-procedures (with equal variances). • The objective in analysis of variance is to determine whethere are differences in means of more than 2 groups. • The statistical methodology for comparing several means is called analysis of variance, or simply ANOVA. • When studying the effect of one factor only on the response we use one-way ANOVA to analyze the data. • When studying the effect of two factors on the response we use twoway ANOVA. STA 248 week 10 13

One-Way ANOVA model • The response variable Y is measured on each experimental unit in each treatment group. Measure Yij for the jth subject in the ith group. • The one-way ANOVA model is: Yij = μi + εij for i = 1, 2, …, k and j = 1, 2, …, ni. • μi is the unknown mean response for the ith group. • The εij are called “random errors” and are assumed to be i. i. d N(0, σ2). • The parameters of the model are the population means μ 1, μ 2, …, μk and the common standard deviation σ. • The objective of one-way ANOVA is to test whether the mean response in each treatment group is the same. • The null and alternative hypotheses are…. STA 248 week 10 14

Derivation of Test Statistics STA 248 week 10 15