Experimental Design SPH 247 Statistical Analysis of Laboratory
Experimental Design SPH 247 Statistical Analysis of Laboratory Data April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 1
Basic Principles of Experimental Investigation �Sequential Experimentation �Comparison �Manipulation �Randomization �Blocking �Simultaneous variation of factors �Main effects and interactions �Sources of variability April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 2
Sequential Experimentation �No single experiment is definitive �Each experimental result suggests other experiments �Scientific investigation is iterative. �“No experiment can do everything; every experiment should do something, ” George Box. April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 3
Analyze Data from Experiment Plan Experiment Perform Experiment April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 4
Comparison �Usually absolute data are meaningless, only comparative data are meaningful �The level of m. RNA in a sample of liver cells is not meaningful �The comparison of the m. RNA levels in samples from normal and diseased liver cells is meaningful April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 5
Internal vs. External Comparison �Comparison of an experimental results with historical results is likely to mislead �Many factors that can influence results other than the intended treatment �Best to include controls or other comparisons in each experiment April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 6
Manipulation �Different experimental conditions need to be imposed by the experimenters, not just observed, if at all possible �The rate of complications in cardiac artery bypass graft surgery may depend on many factors which are not controlled (for example, characteristics of the patient), and may be hard to measure April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 7
April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 8
Randomization �Randomization limits the difference between groups that are due to irrelevant factors �Such differences will still exist, but can be quantified by analyzing the randomization �This is a method of controlling for unknown confounding factors April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 9
�Suppose that 50% of a patient population is female �A sample of 100 patients will not generally have exactly 50% females �Numbers of females between 40 and 60 would not be surprising �In two groups of 100, the disparity between the number of females in the two groups can be as big as 20% simply by chance, but not much larger �This also holds for factors we don’t know about April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 10
�Randomization does not exactly balance against any specific factor �To do that one should employ blocking �Instead it provides a way of quantifying possible imbalance even of unknown factors �Randomization even provides an automatic method of analysis that depends on the design and randomization technique. April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 11
The Farmer from Whidbey Island �Visited the University of Washington with a Whalebone water douser � 10 Dixie cups, 5 with water, 5 empty, each covered with plywood �Placed in a random order defined by generating 10 random numbers and sorting the cups by the random number �If he gets all 10 right, is chance a reasonable explanation? April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 12
�The randomness is produced by the process of randomly choosing which 5 of the 10 are to contain water �There are no other assumptions April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 13
�If the randomization had been to flip a coin for each of the 10 cups, then the probability of getting all 10 right by chance is different �There are 210 = 1024 ways for the randomization to come out, only one of which is corresponds to the choices, so the chance is 1/1024 =. 001 �The method of randomization matters �If the farmer could observe condensation on the cups, then this is still evidence of non-randomness, but not of the effectiveness of dousing! April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 14
Randomization Inference � 20 tomato plants are divided 10 groups of 2 placed next to each other in the greenhouse (to control for temperature and insolation) �In each group of 2, one is chosen using a random number table to receive fertilizer A; the other receives fertilizer B �The yield of each plant in pounds of tomatoesis measured �The null hypothesis is that the fertilizers are equal in promoting tomato growth April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 15
1 2 3 4 5 6 7 8 9 10 A 132 82 109 143 107 66 95 108 88 133 B 140 88 112 142 118 64 98 113 93 136 diff 8 6 3 -1 11 -2 3 5 5 3 Pounds of yield of tomatoes for 20 plants April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 16
�The average yield for fertilizer A is 106. 3 pounds �The average yield for fertilizer B is 110. 4 pounds �The average difference is 4. 1 �Could this have happened by chance? �Is it statistically significant? �If A and B do not differ in their effects (null hypothesis is true), then the plants’ yields would have been the same either whether A or B is applied �The difference would be the negative of what it was if the coin flip had come out the other way April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 17
Actual Fert A Hypothetical Fert B ∆=8 132 lb April 2, 2013 140 lb Fert B Fert A ∆ = − 8 132 lb 140 lb SPH 247 Statistical Analysis of Laboratory Data 18
�In pair 1, the yields were 132 and 140. �The difference was 8, but it could have been − 8 �With 10 coin flips, there are 210 = 1024 possible outcomes of + or − on the difference �These outcomes are possible outcomes from our action of randomization, and carry no assumptions �The measurements don’t have to be normally distributed or have the same variance April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 19
�Of the 1024 possible outcomes that are all equally likely under the null hypothesis, only 3 had greater values of the average difference, and only four (including the one observed) had the same value of the average difference �The likelihood of this happening by chance is [3+4/2]/1024 =. 005 �This does not depend on any assumptions other than that the randomization was correctly done April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 20
1 2 3 4 5 6 7 8 9 10 A 132 82 109 143 107 66 95 108 88 133 B 140 88 112 142 118 64 98 113 93 136 diff April 2, 2013 8 6 3 -1 11 -2 SPH 247 Statistical Analysis of Laboratory Data 3 5 5 3 21
Paired t-test April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 22
Randomization in practice �Whenever there is a choice, it should be made using a formal randomization procedure, such as Excel’s rand() function. �This protects against unexpected sources of variability such as day, time of day, operator, reagent, etc. April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 23
Pair Number 1 2 3 4 5 6 7 8 9 10 April 2, 2013 First Sample Treatment A or B? A or B? A or B? SPH 247 Statistical Analysis of Laboratory Data 24
Pair Num 1 2 3 4 5 6 7 8 9 10 April 2, 2013 First Sample random Treatment number A or B? 0. 871413 A or B? 0. 786036 A or B? 0. 889785 A or B? 0. 081120 A or B? 0. 297614 A or B? 0. 540483 A or B? 0. 824491 A or B? 0. 624133 A or B? 0. 913187 A or B? 0. 001599 SPH 247 Statistical Analysis of Laboratory Data 25
�=rand() in first cell �Copy down the column �Highlight entire column �^c (Edit/Copy) �Edit/Paste Special/Values �This fixes the random numbers so they do not recompute each time �=IF(C 3<0. 5, "A", "B") goes in cell C 2, then copy down the column April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 26
Plant Pair 1 2 3 4 5 6 7 8 9 10 April 2, 2013 First Plant Treatment B B B A A B B A random number 0. 871413 0. 786036 0. 889785 0. 081120 0. 297614 0. 540483 0. 824491 0. 624133 0. 913187 0. 001599 SPH 247 Statistical Analysis of Laboratory Data 27
�To randomize run order, insert a column of random numbers, then sort on that column �More complex randomizations require more care, but this is quite important and worth the trouble �Randomization can be done in Excel, R, or anything that can generate random numbers April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 28
Randomization in R rand 1 <- runif(10) treat <- rep("", 10) rand 2 <- order(rand 1) < 5. 5 treat[rand 2] <- "Treatment A" treat[!rand 2] <- "Treatment B" > rand 1 [1] 0. 23459799 0. 18243579 0. 07706528 0. 68511653 0. 70065774 0. 59058980 [7] 0. 84561795 0. 96164966 0. 20475362 0. 49222996 > order(rand 1) [1] 3 2 9 1 10 > rand 2 [1] TRUE 6 TRUE FALSE 4 5 7 8 TRUE FALSE > treat [1] "Treatment A" "Treatment B" [6] "Treatment B" "Treatment A" "Treatment B" April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 29
Blocking �If some factor may interfere with the experimental results by introducing unwanted variability, one can block on that factor �In agricultural field trials, soil and other location effects can be important, so plots of land are subdivided to test the different treatments. This is the origin of the idea April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 30
�If we are comparing treatments, the more alike the units are to which we apply the treatment, the more sensitive the comparison. �Within blocks, treatments should be randomized �Paired comparisons are a simple example of randomized blocks as in the tomato plant example April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 31
Simultaneous Variation of Factors �The simplistic idea of “science” is to hold all things constant except for one experimental factor, and then vary that one thing �This misses interactions and can be statistically inefficient �Multi-factor designs are often preferable April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 32
Interactions �Sometimes (often) the effect of one variable depends on the levels of another one �This cannot be detected by one-factor-at-a-time experiments �These interactions are often scientifically the most important April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 33
�Experiment 1. I compare the room before and after I drop a liter of gasoline on the desk. Result: we all leave because of the odor. April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 34
�Experiment 1. I compare the room before and after I drop a liter of gasoline on the desk. Result: we all leave because of the odor. �Experiment 2. I compare the room before and after I drop a lighted match on the desk. Result: no effect other than a small scorch mark. April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 35
�Experiment 1. I compare the room before and after I drop a liter of gasoline on the desk. Result: we all leave because of the odor. �Experiment 2. I compare the room before and after I drop a lighted match on the desk. Result: no effect other than a small scorch mark. �Experiment 3. I compare all four of ±gasoline and ±match. Result: we are all killed. �Large Interaction effect April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 36
Statistical Efficiency �Suppose I compare the expression of a gene in a cell culture of either keratinocytes or fibroblasts, confluent and nonconfluent, with or without a possibly stimulating hormone, with 2 cultures in each condition, requiring 16 cultures April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 37
�I can compare the cell types as an average of 8 cultures vs. 8 cultures �I can do the same with the other two factors �This is more efficient than 3 separate experiments with the same controls, using 48 cultures �Can also see if cell types react differently to hormone application (interaction) April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 38
Fractional Factorial Designs �When it is not known which of many factors may be important, fractional factorial designs can be helpful �With 7 factors each at 2 levels, ordinarily this would require 27 = 128 experiments �This can be done in 8 experiments instead! �Each two factors form a replicated two-by-two �Some sets of three factors form an unrelicated two-by-two April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 39
1 2 3 4 5 6 7 8 April 2, 2013 F 1 F 2 F 3 F 4 F 5 F 6 F 7 H H H H H L L L H L H L L L L H H L L H L L H H L L L H H H L SPH 247 Statistical Analysis of Laboratory Data 40
1 2 3 4 5 6 7 8 April 2, 2013 F 1 F 2 F 3 F 4 F 5 F 6 F 7 H H H H H L L L H L H L L L L H H L L H L L H H L L L H H H L SPH 247 Statistical Analysis of Laboratory Data 41
1 2 3 4 5 6 7 8 April 2, 2013 F 1 F 2 F 3 F 4 F 5 F 6 F 7 H H H H H L L L H L H L L L L H H L L H L L H H L L L H H H L SPH 247 Statistical Analysis of Laboratory Data 42
1 2 3 4 5 6 7 8 April 2, 2013 F 1 F 2 F 3 F 4 F 5 F 6 F 7 H H H H H L L L H L H L L L L H H L L H L L H H L L L H H H L SPH 247 Statistical Analysis of Laboratory Data 43
Main Effects and Interactions �Factors Cell Type (C), State (S), Hormone (H) �Response is expression of a gene �The main effect C of cell type is the difference in average gene expression level between cell types April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 44
�For the interaction between cell type and state, compute the difference in average gene expression between cell types separately for confluent and nonconfluent cultures. The difference of these differences is the interaction. �The three-way interaction CSH is the difference in the two way interactions with and without the hormone stimulant. April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 45
Sources of Variability in Laboratory Analysis �Intentional sources of variability are treatments and blocks �There are many other sources of variability �Biological variability between organisms or within an organism �Technical variability of procedures like RNA extraction, labeling, hybridization, chips, etc. April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 46
Replication �Almost always, biological variability is larger than technical variability, so most replicates should be biologically different, not just replicate analyses of the samples (technical replicates) �However, this can depend on the cost of the experiment vs. the cost of the sample � 2 D gels are so variable that replication is required �Expression arrays, PCR, RNA-Seq, Mass Spect and others do not usually require replication April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 47
Quality Control �It is usually a good idea to identify factors that contribute to unwanted variability �A study can be done in a given lab that examines the effects of day, time of day, operator, reagents, etc. �This is almost always useful in starting with a new technology or in a new lab April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 48
Possible QC Design �Possible factors: day, time of day, operator, reagent batch �At two levels each, this is 16 experiments to be done over two days, with 4 each in morning and afternoon, with two operators and two reagent batches �Analysis determines contributions to overall variability from each factor April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 49
References �Statistics for Experimenters, Box, Hunter, and Hunter, John Wiley April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 50
Exercise 1 �You have a clinical study in which 10 patients will either get the standard treatment or a new treatment �Randomize which 5 of the 10 get the new treatment so that all possible combinations can result. Use Excel or R or another formal randomization method. �Instead, randomize so that in each pair of patients entered by date, one has the standard and one the new treatment (blocked randomization). �What are the advantages of each method? �Why is randomization important? April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 51
Course Website �http: //dmrocke. ucdavis. edu April 2, 2013 SPH 247 Statistical Analysis of Laboratory Data 52
- Slides: 52