- Slides: 22
Linking Data Collection to Causality
Collecting for a Causality When we collect data, we have varying purposes. Sometimes we just want to describe a population. Other times we want to determine whether variables are causally related. When measuring phenomena, the timing and whom we contact should match our objectives: Describe or Explain. How does timing and whom we contact (design) affect ability to make causal statements?
Collecting for a Causality Y X z Z One thing causes another when there is: a) Association—when X and Y change in tandem X Y Variable b) Time Order—for X to cause Y, value of X must occur prior to Independent value Dependent of Y c) Nonspuriousness—relationship between X and Y is not coincidental or caused by changes in a third variable (z)
Collecting for a Causality Cross-Sectional Design Collecting data at one point in time, using same instruments for everyone—observing or asking questions only during a single limited time-frame. n Great for descriptive work. n Effect on Causality: 1. Can establish association, 2. Time-order is hard to establish n Answers on variables such as sex and race can be assumed to have pre-dated answers on other variables Answers to many variables, however, do not clearly precede answers to others We often rely on respondents’ memories to establish time order and this can be erroneous (why? )
Collecting for a Causality Longitudinal Designs n n n Collecting data, using same instrument for everyone, at more than one point in time— observing or asking questions across time, typically at discrete points. Works for description over time. Effect on Causality: Depends on design, trend or fixed-sample design?
Collecting for a Causality Longitudinal Designs n Repeated Cross-Sectional Designs or Trend Studies New sample used to collect data at each new time point. n Political Polls, General Social Survey. Descriptive: Can see change over time. Effect on Causality: 1. Can establish association at distinct times. Cannot establish association at the individual level over time points. May establish macro level association over time points. 2. Like cross-sectional design at each time point. Cannot establish time order at the individual level over time points. May establish macro level time order over time points.
Collecting for a Causality Longitudinal Designs n Fixed-Sample Panel Design or Panel Study Same sample used to collect data at each new time point. Descriptive: Can see change over time. Effect on Causality: 1. Can establish association at distinct times, at the individual level over time points, and at the macro level over time points. 2. Can establish time order at the individual level over time points and at the macro level over time points. An independent variable’s value at a previous time can be linked to a dependent variable’s value at a subsequent time. Very time-consuming, expensive.
Collecting for a Causality Non-Spuriousness Cross-sectional and longitudinal designs cannot establish that associations are not spurious. Breadth of data collection—having collected enough of the right variables—allows one to take into account other extraneous variables. Can you establish nonspuriousness with your papers’ analyses?
Collecting for a Causality Experiments n n Treating groups differently, but collecting the same information from them. True experiments have: At least two comparison groups (experimental and control) Random assignment of subjects to comparison groups. Variation (or manipulation) in an independent variable before assessment of outcome on the dependent variable Independent Variable Dependent Variable Sample Random Assignment Experimental Group Vary a condition, X Control Group Do nothing, X Measure Y Compare scores Measure Y
Collecting for a Causality Experiments Devised to assess causality by controlling everything possible while allowing for a change in just one variable to see how it would affect variables of interest in subjects. Control is created by randomly placing persons in two or more groups and treating them the same except… Time-order is established by manipulating an independent variable between groups—changing just one thing for one group but not the other. Association is determined by observing change in the dependent variable after allowing only the independent variable to vary. Non-spuriousness is determined by not allowing anything else to vary between groups. If nothing else is changing, there is no extraneous variable influencing those of interest. n Random assignment (NOT RANDOM SAMPLING) of persons to comparison groups eliminates possibility of systematic variation between groups.
Collecting for a Causality Experiments n n n Sometimes, pretests are used prior to manipulation of the independent variable. This does not establish causality as much as it provides a baseline allowing one to determine just how much the dependent variable changes and can demonstrate similarity of comparison groups prior to manipulation. Pre-Measure Y Independent Variable Dependent Variable Sample Random Assignment Experimental Group Vary a condition, X Control Group Do nothing, X Measure Y Compare scores Measure Y
Collecting for a Causality Experiments n n n Sometimes, matching of subjects influences assignment. This is so that one can guarantee similarity along certain dimensions across comparison groups. If using matching alone, the design is “quasi – experimental, ” ‘quasi’ meaning “something that appears to be something it is not” Matching can be used with random assignment Independent Variable Dependent Variable Matching Sample & Random Assignment Experimental Group Vary a condition, X Control Group Do nothing, X Measure Y Compare scores Measure Y
Collecting for a Causality Experiments are good when one can control and manipulate. n n Experiments are much more common in the natural sciences Sociologists rarely use experiments, generalizeability for complex social phenomena is limited: Ethical concerns lead us to observe rather than control and manipulate (we just can’t control the way we’d have to) Control is artificial, setting up nonrepresentative contexts Observation changes the observed, especially among humans Variables of interest are more complex than can be represented in a controlled setting Subjects forming the sample are typically recruited, leading to nonrepresentative samples
Collecting for a Causality Quasi-Experiments n n n Quasi-experiments attempt to adapt good things about experiments to situations where controlled experiments are impossible. Helpful if it is impossible to randomly assign people to groups that determine their experiences—like when studying real-world situations or interventions They are common in evaluation research—determining whether an intervention is effective. Missing typically is Random Assignment to groups. Technically, groups should be determined prior to manipulation of the independent variable or “intervention. ” Independent Variable Dependent Variable Sample Random Assignment Experimental Group Vary a condition, X Control Group Do nothing, X Measure Y Compare scores Measure Y
Collecting for a Causality Quasi-Experiments n Nonequivalent control group designs: A ; T-O ; N-S 1. Individual Matching Persons are assigned to different groups in “pairs” so that experimental and control groups will be similar. 2. Aggregate Matching Another group of persons that resembles the experimental group is selected to act as the control group. matching Sample Random Assignment Independent Variable Dependent Variable Experimental Group Vary a condition, X Control Group Do nothing, X Measure Y Compare scores Measure Y
Collecting for a Causality Quasi-Experiments n Before-and-After designs: A ; T-O ; N-S 1. A group acts as it’s own control. A pretest measure (the control) is compared with a posttest measure. The control group becomes the experimental group and is then compared with itself. Helpful when a control group is almost impossible to create or find, such as when an entire organization changes procedures. Sample Random Assignment Independent Variable Dependent Variable continue Experimental Vary a Measure Y Group condition, X Compare scores Start here Control Measure Y Do nothing, X Group
Collecting for a Causality Quasi-Experiments n Before-and-After designs 2. Comparing multiple groups that experience the same independent variable manipulation improves confidence in conclusions about causality. Repeated measurement prior to and after change in the independent variable provides even more evidence for causality and permits analysis of how long effects last. Sample Random Assignment Independent Variable Dependent Variable continue Experimental Vary a Measure Y Group condition, X Compare scores Start here Control Measure Y Do nothing, X Group
Collecting for a Causality Nonexperiments These lack some key element of experiments such as lacking random assignment to groups, lacking matching prior to manipulation of the independent variable or lacking comparison groups. n Ex Post Facto Control Group Design —A ; T-O ; N-S Experimental Group The groups cannot be determined in advance, so there is the possibility of extraneous factors determining group membership. This is often necessary when studying events that have occurred or practices that are already in place. Vary a Measure Y condition, X Compare Find another scores on Y similar group. Control Measure Y Do nothing, X Group
Collecting for a Causality Factorial Surveys —A ; T-O ; N-S n n Random Sample A research “bright spot” where researchers attempt to combine generalizability of a random sample with random assignment to groups. Randomly selected participants randomly get treatment or no treatment in the survey, typically vignettes, and then dependent variable is measured later. Often survey methods are tested this way, with randomly selected sample being randomly surveyed with different techniques such as with interview, paper/pencil, or web-based. The biggest issue is typically that only attitudes can be measured, not particular behaviors. Independent Variable Dependent Variable Random Assignment Experimental Group Vary a condition, X Control Group Do nothing, X Measure Y
Collecting for a Causality A Note: n Regardless of the research method you employ, you should be thinking in terms of: Association Time-order Nonspuriousness
Collecting for a Causality Some other things to consider, threats to determining causality and validity. Make sure you study these. n Selection bias Differential attrition n Endogenous Change n Testing Maturation Regression Effect n External Events n Contamination n Treatment Misidentification Researcher demand Self-fulfilling prophesies Placebo effect Hawthorne effect
Collecting for a Causality In-class Group Assignment (worth 2 Q & A) n n n The Fantasy Island Preservation Society has offered you a lot of money to do research. They believe that watching Fantasy Island increases willingness to pursue dreams. Your job is to devise an experiment that is reasonably feasible that will determine whether watching Fantasy Island affects pursuit of dreams. Due at the end of class!