Experimental Designs HOW DO WE FIND THE ANSWERS

Experimental Designs HOW DO WE FIND THE ANSWERS ? 1

Characteristics of Experimental Designs Ø Manipulation of one or more factors Ø (Independent Ø Measurement of the effects of manipulation Ø (Dependent Ø we in control? Reliability Ø Can Ø Variables) Validity Ø Are Ø Variables) the results be replicated? Sensitivity Ø Are we measuring what we want to measure? 2

Validity (Internal) Ø Ø Ø amount of control over experimental conditions allows conclusion that the IV causes an effect on the DV allows exclusion of other variables causing an effect on the DV 3

Internal Validity ä Challenges to Internal Validity ä Using intact groups ä (such as classes of students) ä Not balancing extraneous variables ä (individual differences) ä ä * hypnosis volunteers early or late in term Subject Loss ä mechanical subject loss (equipment failure) ä selective subject loss (related to paradigm? ) 4

Validity (External) ä Can findings be generalized ä to other species ä to other individuals ä to other settings or situations ä to other conditions 5

Validity (External) ä For some lab experiments we do not establish external validity ä Is external validity needed? ä Mook (1983) argues that external validity is irrelevant if we are testing a specific hypothesis in a laboratory setting ä Lab experiments typically try to test a specific hypothesis instead of imitating a typical situation 6

Validity (External) ä External validity needed when results are to generalized to a population ä External validity requires a representative sample ä Partial replication (repeating some but not all of the experimental conditions) can provide evidence for external validity 7

Sensitivity ä Is our measure appropriate for the effect we are looking for? ä Are we measuring enough of the effect? ä Are we measuring too much of the effect (even if we get an effect, is it meaningful? ) 8

Issues of Control ä Methods of Control ä Manipulation ä Holding conditions constant ä Balancing 9

Control ä Manipulation ä Systematic varying of an Independent Variable 10

Control ä Hold Conditions Constant ä Make the IV the only variable the differentiates between the groups ä Example: Use only males to hold the gender effect constant 11

Control ä Balancing ä Technique used to control for individual differences of participants ä Used in independent groups designs ä Insures that all groups are equivalent in areas such as age, motivation, sex, intelligence, etc. 12

Independent Groups Design ä Each group represents a different condition ä Conditions are defined by the level of the IV ä Groups are formed by participants being assigned to conditions ä Nature of group formation makes balancing a major consideration of control 13

Random Groups Design ä Groups are formed prior to introducing the IV ä Subjects are sampled in such a way that the selection of one subject in no way influences the selection of another subject ä All subjects have an equal chance of being in any given group ä May be accomplished by random selection or random assignment 14

Random Selection ä Requires a well defined population ä Requires randomization processes for selection of subjects ä Subjects are randomly selected for each group 15

Random Assignment ä Used when random selection is not possible ä Most samples are accidental and not from well defined populations (Intro Psyc students) ä Random assignment is then used to randomize subjects into different groups instead of random selection 16

Random Assignment ä Block Randomization ä Most often used for random assignment ä Have number of blocks = number of subjects in each condition ä Randomize conditions in each block ä Assign subjects to each condition in each block until all blocks are filled 17

Matched Groups Design ä Used when comparable groups is required ä Instead of random assignment ä the researcher makes the groups equivalent by matching the subjects in each group ä Most useful if a good matching task is used 18

Matched Group Design ä Example ä 1) pretest for dependent variable (BP) ä 2) match subjects by BP level and group by the number of conditions ä 3) randomly assign to conditions ä 4) compare BP of subjects by condition at posttest 19

Natural Groups Design ä Subjects are selected based on levels of IV ä Used when impossible to manipulate IV ä age, gender, personality traits, etc. ä Used when not ethical to manipulate IV ä married, divorced, widowed, etc. 20

Repeated Measures Design ä One group of subjects ä Subjects receive all levels of the IV ä Eliminates problem of Individual Differences ä Reduces the number of subjects required ä Counterbalancing necessary for control 21

Counterbalancing ä Counterbalancing necessary to control practice effect. ä ABBA design optimal (IVs A & B) ä ABBA (complete counterbalance) 22

Counterbalancing ä Problems with ABBA design ä Each subject has to complete all presentations of IV ä ä ABBA As IV levels increase design becomes unmanageable ä IVs A, B, & C ä ABCACBBACBCACABCBA (Complete Counterbalance) 23

Counterbalancing ä Alternate to complete ABBA design ä Use Incomplete design ½ of group receives conditions AB ä ½ of group receives conditions BA ä ä IVs ABC (complete design) ä ABCACBBACBCACABCBA (Complete Counterbalance – each subject receives 18 conditions) ä ABC ACB BAC BCA CAB CBA ä (Incomplete Counterbalance – each subject receives 3 conditions) 24

Design Problems and Solutions ä Independent Groups Design ä Individual Differences ä (Differences between subjects in each group) ä Use Repeated Measures Design to eliminate individual differences (using same subjects) ä Repeated Measures Design ä Differential Transfer ä (Carryover effects) ä Use Independent Groups Design to eliminate differential transfer 25

Complex (Factorial) Designs ä Main Effects ä Effects of the Main IVs ä Two possible Main Effects in your experiment ä ä Difference in RT between Caffeine and No Caffeine ä (IV # 1 or “A”) Difference in RT of PH and NPH ä (IV # 2 or “B”) 26

Complex (Factorial) Designs ä Interaction Effects ä How one IV (A) may impact another IV (B) ä Will Caffeine influence RT in one hand but not the other hand? ä Four possible Interaction Effects in your experiment ä PH Caffeine No Caffeine RT (Caff&PH) RT (No. Caff&PH) NPH RT (Caff&NPH) ä RT (No. Caff&NPH) This is a 2 X 2 Factorial Design ä 2 IVs (Caffeine & Handedness) ä Each has 2 levels (Caffeine or No Caffeine & PH or NPH) 27

Analysis of Factorial Designs ä Analyze with a Factorial ANOVA (F test) ä F test analyses reflects Systematic variance due to manipulation ä Error variance due to confounds ä ä äF= ä Including Individual differences of subjects variation between groups variation within groups error variation + systematic variation error variation 28

Analysis of Factorial Designs ä F test analyses reflects ä F= variation between groups variation within groups F= error variation + systematic variation error variation ä ä F test may indicate significant differences in Main Effects and Interaction Effects ä Requires a Post Hoc text to determine differences 29

Analysis of Factorial Designs ä Post Hoc test for Main Effects ä One-Way or Repeated Measures ANOVAS if needed ä Post Hoc test for Interaction Effects 500 ä 400 300 200 100 0 Graph data Parallel lines indicate no interaction ä Converging interactions 600 or Intersecting lines indicate 600 PH f. . Ca o PH 200 NPH 0 Caffeine No Caffeine NPH No Caffeine 400 PH 200 NPH 0 Caffeine No Caffeine N in e . NPH ffe PH ä 400 Ca 400 300 200 100 0 30

Experimental Designs EXPERIMENTAL DESIGNS THAT ARE CORRECTLY EXECUTED RESULT IN SUCESSFUL OUTCOMES 31

Experimental Designs äQUESTIONS? 32