Sampling Experimental Control Psych 231 Research Methods in

Sampling & Experimental Control Psych 231: Research Methods in Psychology

n Errors in measurement n Sampling error Population n Why do we use sampling methods? n Typically don’t have the resources to test everybody, so we test a subset Sample Sampling Everybody that the research is targeted to be about The subset of the population that actually participates in the research

Population Sampling to make data collection manageable Inferential statistics used to generalize back n Sample Sampling Allows us to quantify the Sampling error

n Goals of “good” sampling: – Maximize Representativeness: – To what extent do the characteristics of those in the sample reflect those in the population – Reduce Bias: – A systematic difference between those in the sample and those in the population n Key tool: Random selection Sampling

n Probability sampling n n Simple random sampling Systematic sampling Stratified sampling Have some element of random selection Non-probability sampling n n Convenience sampling Quota sampling Sampling Methods Susceptible to biased selection

n Every individual has a equal and independent chance of being selected from the population Simple random sampling

n Selecting every nth person Systematic sampling

n n Step 1: Identify groups (clusters) Step 2: randomly select from each group Cluster sampling

n Use the participants who are easy to get Convenience sampling

n n Step 1: identify the specific subgroups Step 2: take from each group until desired number of individuals Quota sampling

n Control variables n n Holding things constant - Controls for excessive random variability Random variables – may freely vary, to spread variability equally across all experimental conditions n Randomization • A procedure that assures that each level of an extraneous variable has an equal chance of occurring in all conditions of observation. n Confound variables n n Variables that haven’t been accounted for (manipulated, measured, randomized, controlled) that can impact changes in the dependent variable(s) Co-varys with both the dependent AND an independent variable Extraneous Variables

n Divide into two groups: n n men women n Instructions: Read aloud the COLOR that the words are presented in. When done raise your hand. n Women first. Men please close your eyes. Okay ready? n Colors and words

Blue Green Red Purple Yellow Green Purple Blue Red Yellow Blue Red Green List 1

n n n Okay, now it is the men’s turn. Remember the instructions: Read aloud the COLOR that the words are presented in. When done raise your hand. Okay ready?

Blue Green Red Purple Yellow Green Purple Blue Red Yellow Blue Red Green List 2

n So why the difference between the results for men versus women? n Is this support for a theory that proposes: n n “Women are good color identifiers, men are not” Why or why not? Let’s look at the two lists. Our results

Matched List 1 List 2 Women Men Blue Green Red Purple Yellow Green Purple Blue Red Yellow Blue Red Green Mis-Matched

n Blue Green Red Purple Yellow Green Purple Blue Red Yellow Blue Red Green n n What resulted in the perfomance difference? n Our manipulated independent variable (men vs. women) n The other variable match/mis-match? Because the two variables are perfectly correlated we can’t tell This is the problem with confounds IV Co-vary together Confound DV Blue Green Red Purple Yellow Green Purple Blue Red Yellow Blue Red Green

n Our goal: n To test the possibility of a relationship between the variability in our IV and how that affects the variability of our DV. • Control is used to minimize excessive variability. • To reduce the potential of confounds. Experimental Control

n Sources of Total (T) Variability: T = NRexp + NRother + R Nonrandom (NR) Variability - systematic variation A. (NRexp) manipulated independent variables (IV) i. our hypothesis is that changes in the IV will result in changes in the DV B. (NRother) extraneous variables (EV) which covary with IV i. Condfounds Sources of variability (noise)

n Sources of Total (T) Variability: T = NRexp + NRother + R Non-systematic variation C. Random (R) Variability • Imprecision in manipulation (IV) and/or measurement (DV) • Randomly varying extraneous variables (EV) Sources of variability (noise)

n Sources of Total (T) Variability: T = NRexp + NRother + R Goal: to reduce R and NRother so that we can detect NRexp. That is, so we can see the changes in the DV that are due to the changes in the independent variable(s). Sources of variability (noise)

n Imagine the different sources of variability as weights The effect of the treatment NR other NR exp R Treatment group Weight analogy NR other control group R

n If NRother and R are large relative to NRexp then detecting a difference may be difficult NR other NR NR R other exp Difference Detector Weight analogy R

n But if we reduce the size of NRother and R relative to NRexp then detecting gets easier NR other NR exp R Difference Detector Weight analogy R

n Potential Problems n n n Confounding Excessive random variability Dissimulation Difference Detector Things making detection difficult

n Confounding n If an EV co-varies with IV, then NRother component of data will be "significantly" large, and may lead to misattribution of effect to IV IV DV Co-vary together EV Potential Problems

Hard to detect the effect of NRexp because the effect looks like it could be from NRexp but is really (mostly) due to the NRother R NRother NR exp R Difference Detector Confounding

n Excessive random variability n If experimental control procedures are not applied • Then R component of data will be excessively large, and may make NRexp undetectable n So try to minimize this by using good measures of DV, good manipulations of IV, etc. Potential Problems

Hard to detect the effect of NRexp R NR NR NR other exp Difference Detector Excessive random variability

n Potential problem caused by experimental control n Dissimulation • If EV which interacts with IV is held constant, then effect of IV is known only for that level of EV, and may lead to overgeneralization of IV effect n This is a potential problem that affects the external validity Potential Problems

n Methods of Experimental Control n n n Comparison Production Constancy/Randomization Controlling Variability

n Comparison n An experiment always makes a comparison, so it must have at least two groups • Sometimes there are control groups • This is typically the absence of the treatment • Without control groups if is harder to see what is really happening in the experiment • It is easier to be swayed by plausibility or inappropriate comparisons • Sometimes there are just a range of values of the IV Methods of Controlling Variability

n Production n The experimenter selects the specific values of the Independent Variables • Need to do this carefully • Suppose that you don’t find a difference in the DV across your different groups • Is this because the IV and DV aren’t related? • Or is it because your levels of IV weren’t different enough Methods of Controlling Variability

n Constancy/Randomization n If there is a variable that may be related to the DV that you can’t (or don’t want to) manipulate • Control variable: hold it constant • Random variable: let it vary randomly across all of the experimental conditions n But beware confounds, variables that are related to both the IV and DV but aren’t controlled Methods of Controlling Variability

n n So far we’ve covered a lot of the about details experiments generally Now let’s consider some specific experimental designs. n n Some bad designs Some good designs • • 1 Factor, two levels 1 Factor, multi-levels Factorial (more than 1 factor) Between & within factors Experimental designs

n Example: Does standing close to somebody cause them to move? n So you stand closely to people and see how long before they move n Problem: no control group to establish the comparison group (this design is sometimes called “one-shot case study design”) Poorly designed experiments

n Does a relaxation program decrease the urge to smoke? n n One group pretest-posttest design Pretest desire level – give relaxation program – posttest desire to smoke Poorly designed experiments

n One group pretest-posttest design Dependent Variable participants n Pre-test Independent Variable Training group Dependent Variable Post-test Measure Problems include: history, maturation, testing, instrument decay, statistical regression, and more Poorly designed experiments

n n Example: Smoking example again, but with two groups. The subjects get to choose which group (relaxation or no program) to be in Non-equivalent control groups n Problem: selection bias for the two groups, need to do random assignment to groups Poorly designed experiments

n Non-equivalent control groups Self Assignment Independent Variable Dependent Variable Training group Measure No training (Control) group Measure participants Poorly designed experiments

n Post-test only designs Random Assignment Independent Variable Dependent Variable Experimental group Measure Control group Measure participants “Well designed” experiments

n Pretest-posttest design Random Dependent Independent Assignment Variable Dependent Variable Measure Experimental group Measure Control group Measure participants “Well designed” experiments