Psychology 242 Introduction to Research Exam 2 topics

Psychology 242 Introduction to Research Exam 2 topics 1 This exam is worth 10% of your grade. Items will be very similar to exam #1 3/17/16 q Basics of Experimental design (Notes Week 5) q Sampling (Notes Week 6) q Quasi-experimental designs (Quasi-experimental notes) q Descriptive Research (Descriptive Research notes) q Surveys (Surveys notes) q Some basic statistics Dr. David J. Mc. Kirnan, University of Illinois at Chicago, Psychology; mckirnanuic@gmail. com

Psychology 242 Introduction to Research 2 Experimental Design What needs explaining? Why is it important? What assumptions are you making; what does not need to be explained? Phenomenon Larger question the research addresses How / why does it work? What Hypothetical Constructs Theory or basic processes are Explanatory processes & how they involved? are related How are they related? NOT just a statement of the hypothesis. Hypothesis Concrete variables Specific prediction? Must clearly relate to / express theory. Concrete evidence or data? You all know this, but just to remind you… Methods / Data Operational definitions Study procedures What was the outcome? Was the hypothesis supported? How do you know? Operational definitions Procedures Sample (population you will generalize to). Results Hypothesis-wise analysis of outcomes Discussion & Conclusion Relate results back to theory Study limitations & Future studies What do the results mean for theory or question? What is still unanswered?

Psychology 242 Introduction to Research Basics of Design: Internal Validity 3 Internal Validity core concept: Can we validly determine what is causing the results of the experiment? Default Hypothesis: the experimental outcome (value of the Dependent Variable) is caused only by the Independent Variable. Confound: a “ 3 rd variable” (unmeasured variable other than the Independent Variable) actually led to the results. Design requirements: Appropriate control group Equivalent experimental & control groups Be able to easily articulate and use the concept of internal validity. It should be obvious to you by now. Key distinction: Simple random error (such as a biased sample, poorly thought through measurement, incompetent procedures…) may threaten external validity (the data may not generalize), but not internal validity, because it affects all participants / groups equallly. Non-Random Error that affects one group only (or one more than the other) is a confound since it may lead to group differences rather than the independent Variable.

Psychology 242 Introduction to Research I may ask you to define or two of these (slide 12 of the lecture). Lack of a control group Group 4 Key threats to internal validity Observe 1 Intervention or event Observe 2 Observe 1 Confound Observe 2 Maturation: Participants may be older / wiser by the post-test History; Cultural or historical events may occur between pre- and post-test that change the participants Mortality: Participants may non-randomly drop out of the study Regression to baseline: Participants who are more extreme at baseline look less extreme over time as a statistical confound. Reactive Measurement: Participants may change their scores due to being measured twice, not the experimental manipulation.

Psychology 242 Introduction to Research 5 Generalizability External Validity: Can we validly generalize from this experiment to the larger world? How well does your research sample represent the larger population. Does your measure of the Dep. Var. reflect how the process works outside of the lab? If you are studying, e. g. , happiness, are you really (validly) assessing it? The larger population Other outcomes (dependent variables) What might be an example of each “arm” of external validity? External Validity How well can we generalize to: Other settings Other conditions (independent variables) How well does your experimental manipulation represent how that variable works in nature? Is it artificial? To extreme? Have you administered the correct “dose”. . . How representative (or realistic) is the physical and cultural setting of the research study?

Psychology 242 Introduction to Research 6 Generalizability External Validity: Can we validly generalize from this experiment to the larger world? How well does your research sample represent the larger population. Understand issues in Probability v. nonprobability sampling. Does your measure of the Dep. Var. reflect how the process works outside of the lab? If you are studying, e. g. , happiness, are you really (validly) assessing it? The larger population Other outcomes (dependent variables) Be able to give an example of each, External Validity How well can we generalize to: Be able to define: Face Validity Content Validity Predictive Validity Construct Validity

Psychology 242 Introduction to Research 7 Some key terms: § Replicate Repeat the study exactly You might have to define some of these…. § “Converging” study § Address the same hypothesis with a different form of study § Blind § Participant does not know what group s/he is in § Double blind § Neither participant nor Researcher knows what group the Participant is in § Social desirability responding § § Responding in a way that makes you look good Particularly common with face valid survey items § Reactive measures § People change their behavior if they know they are being assessed § Operational Definition § The specific, “hands on” operations you use to measure or manipulate a variable. § Hypothetical Construct § Abstract psychological / social process that: § Cannot be observed directly § Underlies behaviors than can be observed § Key element of a theory

8 Psychology 242 Introduction to Research Sampling Who do you want to generalize to? Mammals Humans All Western people Breadth of population to sample from (i. e. , size of sampling frame). Represents increasing external validity. All Americans Young Americans College students UIC Students This class Specificity (and ease) of sampling frame. Generally increases internal validity.

9 Psychology 242 Introduction to Research Sampling Probability sampling § simple § multi-stage § stratified Most externally valid Involves some systematic and random form of selection. Assumes: Clear sampling frame Population is available Less externally valid for hidden groups. Be able to talk about basic sampling questions in YOUR course research project. Understand this basic distinction (especially in terms of external validity). Non-probability sampling § targeted / multi-frame § snowball § quota, etc. Less externally valid High “convenience” Best when: No clear sampling frame Hidden / avoidant population.

10 Psychology 242 Introduction to Research Sampling Probability sampling § simple § multi-stage § Cluster / stratified Non-probability sampling Haphazard Modal instance Venue – time / space Multi-frame Snowball / Respondent driven Heterogeneity I may ask you to choose one from each category and describe them…!

11 Psychology 242 Introduction to Research Sampling What is the criterion for ‘membership’ in your sample? What are your inclusion criteria? Exclusion criteria? Be able to use these terms in a sentence. Demographic Gender, zip code, ethnicity… Behavioral Voting history, spending patterns… Self-identification Identification as, e. g. , ‘conservative’ vs. ‘liberal’ The criteria used to define the group – key part of your sampling frame - will determine who specifically gets sampled.

Psychology 242 Introduction to Research Quasi-experiments 12 1. Study naturally occurring events that could not be brought into a lab or a true experiment. • Measurement studies of, e. g. , disasters, historical events • Retrospective designs; using archival data 2. Evaluate existing groups or program(s), or issues where groups cannot be equivalent at baseline In Quasi-experiments the researcher has less control over all aspects of the research, since it is occurring in the field / in nature. This lessens internal validity (less control) …and increases external validity (generalizes better to the “real world”. . . ) • Understand these two basic forms of quasiexperiments. • Be able to provide an example of each. • Be able to discuss the internal v. external validity trade-off for each.

Psychology 242 Introduction to Research Four general types of quasi -experiments Group Naturally occurring event or social change 13 Observe Study the aftermath or consequences of an unanticipated, naturally occurring event. . No control or comparison group is possible; e. g. , natural disaster. Group Observe 1 Intervention or event Observe 2 Study the effect of an event where measurement is already taking place. . . No comparison group is possible; e. g. , introduction of a new school-wide curriculum. Group 1 Group 2 (No baseline) Intervention or event Observe 1 Contrast group Observe 1 Study a naturally occurring, unanticipated event. A comparison group is possible, but the groups cannot be equivalent (…compare two cities after a major cultural event. ) Group Observe 1 Intervention or event Observe 2 Group Observe 1 Contrast group Observe 2 Equivalent of a true experiment or randomized controlled trial, except that the groups cannot be made equivalent (…e. g. , intervention group participants cannot be blind. . . ).

Psychology 242 Introduction to Research Group 14 Naturally occurring events Naturally occurring event or social change No control over who is exposed to the event Possible control over who is selected for the research sample May compromise both internal and external validity. Event not controlled / manipulated. Not a true Independent Variable Often no control group Significant threat to internal validity Observe May or may not have control over measures. Archival measures – medical records, climate data, crime reports – may not assess exactly what the study needs. Survey or other posthoc measures can address hypotheses. Heuristic value of field studies: generating hypotheses for later experimental study, confirm controlled / experimental data in a “real world” (field) setting.

Psychology 242 Introduction to Research 15 One group pre- post-test Group Observe 1 Selected or convenience sample. Intervention or event Baseline Assessment May or may not have control over measures (e. g. , surveys v. archival measures). Observe 2 Outcome Assessment Typically controllable, but may be archival. Event or intervention May or may not be controllable by researcher, e. g. , policy change. Uses: Educational & social environments Political or health policy change Not feasible to have a control group System-wide intervention / social change (school, public health campaign. . )

Psychology 242 Introduction to Research Group One group pre- post-test Observe 1 Observe Key design 1 Intervention or event 16 Observe 2 feature: no control group. Confound Threats to internal validity (confounds): History Historical / cultural events occur between baseline & follow-up. Maturation Individual maturation or growth occurs between baseline & follow-up. Reactive measures People respond to being measured or being a measured a second time. Statistical regression Extreme scores at baseline “regress” to a more moderate level over time. Mortality / drop-out People leave the experiment nonrandomly (i. e. , for reasons that may affect the results…).

Psychology 242 Introduction to Research Group One group pre- post-test 17 Observe 1 Intervention or event Observe 2 Observe 1 Confound Observe 2 Threats to internal validity • I will ask you about these; be (confounds): able to define them. History Historical / cultural events occur between baseline & follow-up. Maturation Individual maturation or growth occurs between baseline & follow-up. Reactive measures People respond to being measured or being a measured a second time. Statistical regression Extreme scores at baseline “regress” to a more moderate level over time. Mortality / drop-out People leave the experiment nonrandomly (i. e. , for reasons that may affect the results…).

Psychology 242 Introduction to Research Quasi-experiments 18 Study interventions or programs where groups cannot be equivalent. Group Observe 1 Intervention or event Observe 2 Group Observe 1 Contrast group Observe 2 Groups are not equivalent at baseline, due to. . Self-selection Non-random assignment Use of existing groups Participants not blind Baseline observation: Measure change, Compare groups prior to the intervention Intervention & Assessments are typically controlled or designed by the researcher; Research in psychotherapy or behavioral interventions. Health-related research… The same as a true experimental design, except for non-equivalent groups • I will ask you about these forms of non-equivalence.

Psychology 242 Introduction to Research True v. quasi-experimental designs True experiments: Emphasize internal validity n Assess cause & effect (in relatively artificial environment) n Test clear, a priori hypotheses Participants assigned to experimental v. control groups n Random or matching n Participants & experimenter Blind to assignment High Control over study procedures § Create / manipulate independent variable § Control procedures & measures 19 • Take a good look at this slide Quasi-experiments: Emphasize external validity n Describe “real” / naturally occurring events n May have clear or only exploratory hypotheses Existing or non-equivalent groups Self-selection Non-random assignment Use of existing groups Participants not blind Control often not possible n May not be able to manipulate the independent variable n Partial control of procedures & measures n Control group not possible?

Psychology 242 Introduction to Research Understand time series data, e. g. : 20 Be able to identify when / where we may use this design. Understand these key elements.

Psychology 242 Introduction to Research Also understand … 21 This design uses a Blocking Variable to examine > 1 group before & after some event. Understand clearly what a blocking variable is.

Psychology 242 Introduction to Research Descriptive research Basic overview of a behavior § 22 Simple but important: “Descriptive” or survey research does not just count stuff. They can test hypotheses. “Who, what, where & when” § … In depth (qualitative) portrayal of the behavior Generate hypotheses § Use qualitative or quantitative descriptions to begin asking “why? ” or “how? ” a behavior occurs. § Develop hypotheses about how to change a behavior…

Psychology 242 Introduction to Research 23 Forms of Descriptive research Qualitative or Observational Quantitative Describe an issue via valid & reliable numerical measures Simple: frequency counts Study behavior “in nature” (high ecological validity). Existing data Use existing data for new quantitative (or qualitative) analyses Qualitative Accretion § In-depth interviews “Blocking” by other variables § Focus (or other) groups § Study “remnants” of behavior § Textual analysis § Wholly non-reactive Correlational research: § Qualitative quantitative Archival of key behavior “what relates to what” Complex modeling • What are these approaches? • What would be an example? Observational § Direct § Use existing data to test new hypothesis § Unobtrusive § Typically non-reactive • When might you use them?

Psychology 242 Introduction to Research n n n Descriptive research: Correlational designs Testing hypotheses with simple correlations: Procedures: n Careful selection of sample to reflect target population n Systematic development of measurements: Reliability Validity Core virtues: n “Natural” look at how variables relate n Less control = less reactivity than experimental designs n Can model very complex phenomena 24

Psychology 242 Introduction to Research 25 Descriptive research: Correlational designs 1. Causality; a simple correlation may confuse cause & effect: which causes which? . ? Depression Alcohol consumption 2. Confounds!; unmeasured 3 rd variable problem: is something else causing both our measured variables? • Understand both these problems in inferring causality from a simple correlation. I will ask you about them!! Depression Genetic disposition ? Alcohol use

Psychology 242 Introduction to Research n Descriptive research: Observational studies Assess behavior directly rather than by participants’ self-reports or recall n Typical data collection is highly reactive: participants know they are being studied, and react to that. n n n 26 Often by making socially desirable responses: faking to look good. Observational methods are often less (or non-) reactive. Directly observe the social & physical settings or environments of behavior. § Direct observation; visual observation & note taking or recording e. g. , sitting in on classroom discussion, therapy session, etc. § Unobtrusive observation; participants unaware of data collection e. g. , 1 -way mirror & therapy research, “stake out” drug scene § Major advantage: Eliminate reactive effects of data collection § Participant observation; become part of social phenomenon to describe it e. g. , joining political organization or cult, posing as prostitute (c. f. ; Hunter S. Thomson Hells Angels; NY Times Down Low article here).

Psychology 242 Introduction to Research Descriptive research 27 Reactive measurement § Participants react to the knowledge that they are being measured. § Threatens External Validity if all participants are giving skewed answers due to the measures. § Threatens Internal Validity – is a confound - if participants in one group have more reactive measures than the other group. § Reactive bias increases with. . § Clarity (face validity) of measures § Face-to-face interview methods § Often lessened with computer interviews Understand how something like reactive measurement • Can just add error (junk variance) to the data, or can… • Represent a confound. Also get: • Error variance only = threat to external validity • Confound = threat to internal validity

Psychology 242 Introduction to Research Descriptive research; existing data 28 Accretion; Study remnants of behavior § Data wholly unobtrusive / non-reactive § Indirect; may only partially map onto phenomenon. Archival; data collected for other purposes § Often in highly reliable, large & rich data sets § Provide unbiased correlations, but most be adapted to new purpose or hypothesis (may not “map on” fully. . ). • Generally understand what these methods are and why we use them. • Look at the lecture notes for examples.

Psychology 242 Introduction to Research 29 Surveys • All this info. is in the Survey focus module. • These are the key concepts in surveys; be able to give an example of them or say why they are important. Core topic areas knowledge Attitudes & Beliefs Behavior Uses of surveys Descriptive data Hypothesis testing Pragmatic; planning & evaluation Closed v. open-ended Face-to-face (interview, telephone) Questionnaire (pencil & paper, internet). Access to the target population Survey formats Social desirability Issues in survey design Time frame of question Question order Bias, political uses of surveys Participant sophistication

Psychology 242 Introduction to Research What do surveys measure? 30 § Knowledge § Information re: current events, political or consumer choices § Awareness of Public health resources, health practices, etc. § Attitudes and Beliefs § Preferences or evaluations: e. g. , attitudes toward gays, ethnic groups, etc. , consumer preferences. § Beliefs about political or social events: “which party provides the strongest security for the U. S…. ? ” § Feelings or moods: quality of life, depression / anxiety, marital satisfaction, etc. § Behavioral intentions; Intent to vote, financial plans, etc. § Self-reports of previous or on-going behavior; topics range from voting to alcohol and drug use. • What might be some examples of these?

Psychology 242 Introduction to Research n n 31 Closed-ended items Chief virtue: clear operationalization n Very specific & concrete; know exactly what participant is responding to n Easy to quantify & use statistically n Can be tested for reliability Understand: • Closed v. open-ended items, • Examples of each • Advantages & disadvantages of each Chief liability: potential insensitivity n Often brief, simply worded; potentially superficial n “Top down”; issues or constructs are imposed on participant

Psychology 242 Introduction to Research Open-ended items 32 General textual / qualitative response; More sensitive to the respondent How have you enjoyed your methods class so far? • Please list the three things that first come to mind when you think of Psychology 242. • Virtues and drawbacks of qualitative or “open ended” items? More difficult to interpret • What might be an example? • Can be analyzed as qualitative data (see discussion in Descriptive data. ) • Can be quantified; • frequency counts of citations or statements • “linkages” analysis of co-occurring statements • Often presented as textual portrayal plus minor quantitative analysis.

Psychology 242 Introduction to Research 33 Example of mixed survey formats Closed-ended attitude scale Example of mixed question format from survey of women’s sexual practices. Open-ended description Simple behavioral index. Personal Safer Sex Guidelines ✓ How strict are your personal guidelines or rules for safer sex (e. g. , condom use, “safe relationships, ” etc. )? 1 ✓ ✓ 2 3 4 Not at all Somewhat strict 5 6 7 Very Extremely What are your rules for safer sex? strict Be able to describe the main advantages & disadvantages of each item format. Have you ever refused to have sex with someone in order to stay safe? 0 1 2 3 Never twice once or times a few times many

Psychology 242 Introduction to Research n n 34 Social Desirability Responding Clear face-valid items addressing embarrassing topics yield less valid responses n How often are you dishonest with your friends? n Have you ever cheated on an exam. . ? High social desirability wording elicits inaccurate responses… n Do you support protecting our Nation’s forests for future generations? (Does “yes” make you an “environmentalist”? ). n Do you feel there are ways your husband could be closer. . . ? (Does “yes” make you are unhappy in your marriage”? ). n Populations differ in social desirability responding; confound in analyses of population effects n Women report more suicidal thoughts, but may be more willing to disclose thought, creating a possible confound (are women more suicidal, or more open…) Desirability can be minimized by: n Anonymous surveys n Assurances of confidentiality n Computer administration (no personal interaction) n Careful wording / pilot testing of items Understand the basic issues here

Psychology 242 Introduction to Research 35 Surveys Survey administration Internet increasingly important as self-report method Face-to-face interviews more common in clinical research Time frames & question order can influence responses Summary Population access & sophistication Some groups are difficult to reach Creates threat to External validity Assumption that participants understand survey materials often questionable. Social desirability responding Inhibited responding threatens Internal Validity May represent a confound if groups differ in desirability set. Understand the highlighted terms

Psychology 242 Introduction to Research Evaluating our measures: Reliability and Validity. Reliability § § If we are assessing a stable characteristic (IQ, personality, temperament, core values…) a good measure will give about the same result each time we administer it and for different sections of the measure. Validity § Our survey or scale must actually measure what we designed it to. § There are several ways we think about validity, each getting at a different element… Be familiar enough with the different variants on reliability and validity that you can define or give an example of them 36

Foundations of Research n Test - retest; n n Reliability similar responses over time? Assume stable attribute; e. g. , “personality” disposition If measure is reliable, should show similar scores across time, e. g. , at baseline and after a year. Split-half; similar responses across item sets? n Assume redundant / converging items or scales. n If scale is reliable, each half should yield similar scores. Chronbach’s alpha; overall internal reliability n 37 Converging items should inter-correlate.

Foundations of Research n Scale appears to measure what it is designed to n E. g. , interview item; “How dependent are you on heroin? ” n Simple skill index; assess computer skills by writing program n Intuitively valid; clearly addresses topic n May yield socially desirable responses. Know / understand this. Content validity n n Validity Face validity n n Descriptive research: Assesses all key components of a topic or construct: n e. g. , the various components of complex political attitudes… n Mid-term; test all core skills for research design… Predictive validity n Validly predicts a hypothesized outcome: n n e. g. , I. Q. is a moderately good predictor of college success, criminality, etc. A measure may be predictive valid without being face or content valid: the MMPI. 38

Foundations of Research n Descriptive research: Validity (2) 39 Construct validity n Test whether the hypothetical construct itself is valid (differs from other constructs, corresponds to measures or outcomes it should. . ). n n Test if the Measure addresses the construct it was designed for n n E. g. ; “anxiety” and “depression” and “anger” may not be separate constructs, but may all be part of “negative affectivity”. e. g. , measures of social support (“do you have people who care for you”) often strongly influenced by depression, a separate construct… “Ecological” validity n n Measure corresponds to how the construct “works” in the real world External validity of assessment device.