Probability Sampling uses random selection N number of

Probability Sampling uses random selection N = number of cases in sampling frame n = number of cases in the sample NCn = number of combinations of n from N f = n/N = sampling fraction

Variations Simple random sampling n based on random number generation Stratified random sampling n divide pop into homogenous subgroups, then simple random sample w/in Systematic random sampling n select every kth individual (k = N/n) Cluster (area) random sampling n randomly select clusters, sample all units w/in cluster Multistage sampling n combination of methods

Non-probability sampling accidental, haphazard, convenience sampling. . . may or may not represent the population well

Measurement. . . topics in measurement that we don’t have time to cover. . .

Research Design Elements: n n n Samples/Groups Measures Treatments/Programs Methods of Assignment Time

Internal validity the approximate truth about inferences regarding cause-effect (causal) relationships can observed changes be attributed to the program or intervention and NOT to other possible causes (alternative explanations)?

Establishing a Cause-Effect Relationship Temporal precedence Covariation of cause and effect n n if x then y; if not x then not y if more x then more y; if less x then less y No plausible alternative explanations

Single Group Example Single group designs: n Administer treatment -> measure outcome X -> O w assumes baseline of “ 0” n Measure baseline -> treat -> measure outcome 0 X -> O w measures change over baseline

Single Group Threats History threat n a historical event occurs to cause the outcome Maturation threat n maturation of individual causes the outcome Testing threat n act of taking the pretest affects the outcome Instrumentation threat n difference in test from pretest to posttest affects the outcome Mortality threat n do “drop-outs” occur differentially or randomly across the sample? Regression threat n statistical phenomenon, nonrandom sample from population and two imperfectly correlated measures

Addressing these threats control group + treatment group n both control and treatment groups would experience same history and maturation threats, have same testing and instrumentation issues, similar rates of mortality and regression to the mean

Multiple-group design at least two groups typically: n n n before-after measurement treatment group + control group treatment A group + treatment B group

Multiple-Group Threats internal validity issue: n n degree to which groups are comparable before the study “selection bias” or “selection threat”

Multiple-Group Threats Selection-History Threat n an event occurs between pretest and posttest that groups experience differently Selection-Maturation Threat n results from differential rates of normal growth between pretest and posttest for the groups Selection-Testing Threat n effect of taking pretest differentially affects posttest outcome of groups Selection-Instrumentation Threat n test changes differently for the two groups Selection-Mortality Threat n differential nonrandom dropout between pretest and posttest Selection-Regression Threat n different rates of regression to the mean in the two groups (if one is more extreme on the pretest than the other)

Social Interaction Threats Problem: n social pressures in research context can lead to posttest differences that are not directly caused by the treatment Solution: n n isolate the groups Problem: in many research contexts, hard to randomly assign and then isolate

Types of Social Interaction Threats Diffusion or Imitation of Treatment n control group learns about/imitates experience of treatment group, decreasing difference in measured effect Compensatory Rivalry n control group tries to compete w/treatment group, works harder, decreasing difference in measured effect Resentful Demoralization n control group discouraged or angry, exaggerates measured effect Compensatory Equalization of Treatment n control group compensated in other ways, decreasing measured effect

Intro to Design/ Design Notation Observations or Measures Treatments or Programs Groups Assignment to Group Time

Observations/Measure Notation: ‘O’ n Examples: w Body weight w Time to complete w Number of correct response Multiple measures: O 1, O 2, …

Treatments or Programs Notation: ‘X’ n n Use of medication Use of visualization Use of audio feedback Etc. Sometimes see X+, X-

Groups Each group is assigned a line in the design notation

Assignment to Group R = random N = non-equivalent groups C = assignment by cutoff

Time Moves from left to right in diagram

Types of experiments True experiment – random assignment to groups Quasi experiment – no random assignment, but has a control group or multiple measures Non-experiment – no random assignment, no control, no multiple measures

Design Notation Example R O 1 X O 1, 2 Pretest-posttest treatment versus comparison group randomized experimental design

Design Notation Example N O X Pretest-posttest Non-Equivalent Groups Quasi-experiment O O

Design Notation Example X Posttest Only Non-experiment O

Goals of design. . Goal: to be able to show causality First step: internal validity: If x, then y AND n If not X, then not Y n

Two-group Designs Two-group, posttest only, randomized experiment R R X O O Compare by testing for differences between means of groups, using t-test or one-way Analysis of Variance(ANOVA) Note: 2 groups, post-only measure, two distributions each with mean and variance, statistical (non-chance) difference between groups

To analyze … What do we mean by a difference?

Possible Outcomes:

Measuring Differences …

Three ways to estimate effect Independent t-test One-way Analysis of Variance (ANOVA) Regression Analysis (most general) equivalent

Computing the t-value

Computing the variance

Regression Analysis Solve overdetermined system of equations for β 0 and β 1, while minimizing sum of e-terms

Regression Analysis

ANOVA Compares differences within group to differences between groups For 2 populations, 1 treatment, same as t-test Statistic used is F value, same as square of t-value from t-test

Other Experimental Designs Signal enhancers n Factorial designs Noise reducers n n Covariance designs Blocking designs

Factorial Designs

Factorial Design Factor – major independent variable n Setting, time_on_task Level – subdivision of a factor n n Setting= in_class, pull-out Time_on_task = 1 hour, 4 hours

Factorial Design notation as shown 2 x 2 factorial design (2 levels of one factor X 2 levels of second factor)

Outcomes of Factorial Design Experiments Null case Main effect Interaction Effect

The Null Case

The Null Case

Main Effect - Time

Main Effect - Setting

Main Effect - Both

Interaction effects

Interaction Effects

Statistical Methods for Factorial Design Regression Analysis ANOVA

ANOVA Analysis of variance – tests hypotheses about differences between two or more means Could do pairwise comparison using ttests, but can lead to true hypothesis being rejected (Type I error) (higher probability than with ANOVA)

Between-subjects design Example: n n Effect of intensity of background noise on reading comprehension Group 1: 30 minutes reading, no background noise Group 2: 30 minutes reading, moderate level of noise Group 3: 30 minutes reading, loud background noise

Experimental Design One factor (noise), three levels(a=3) Null hypothesis: 1 = 2 = 3 Noise None Moderate High R O O O

Notation If all sample sizes same, use n, and total N = a * n Else N = n 1 + n 2 + n 3

Assumptions Normal distributions Homogeneity of variance n Variance is equal in each of the populations Random, independent sampling Still works well when assumptions not quite true(“robust” to violations)

ANOVA Compares two estimates of variance n n MSE – Mean Square Error, variances within samples MSB – Mean Square Between, variance of the sample means If null hypothesis n n is true, then MSE approx = MSB, since both are estimates of same quantity Is false, the MSB sufficiently > MSE

MSE

MSB Use sample means to calculate sampling distribution of the mean, =1

MSB Sampling distribution of the mean * n In example, MSB = (n)(sampling dist) = (4) (1) = 4

Is it significant? Depends on ratio of MSB to MSE F = MSB/MSE Probability value computed based on F value, F value has sampling distribution based on degrees of freedom numerator (a-1) and degrees of freedom denominator (N-a) Lookup up F-value in table, find p value For one degree of freedom, F == t^2

Factorial Between-Subjects ANOVA, Two factors Three significance tests n n n Main factor 1 Main factor 2 interaction

Example Experiment Two factors (dosage, task) 3 levels of dosage (0, 100, 200 mg) 2 levels of task (simple, complex) 2 x 3 factorial design, 8 subjects/group

Summary table SOURCE Task Dosage TD ERROR TOTAL df Sum of Squares 1 47125. 3333 2 42. 6667 2 1418. 6667 42 5152. 0000 47 53738. 6667 Sources of variation: n n Task Dosage Interaction Error Mean Square F p 47125. 3333 384. 174 0. 000 21. 3333 0. 174 0. 841 709. 3333 5. 783 0. 006 122. 6667

Results Sum of squares (as before) Mean Squares = (sum of squares) / degrees of freedom F ratios = mean square effect / mean square error P value : Given F value and degrees of freedom, look up p value

Results - example Mean time to complete task was higher for complex task than for simple Effect of dosage not significant Interaction exists between dosage and task: increase in dosage decreases performance on complex while increasing performance on simple

Results