Foundations of Research 1 12 Research Sampling What

Foundations of Research 1 12. Research Sampling. What is the target population for your study? Probability & Non-Probability sampling methods. Run this as a Power. Point Show ü “Slide show” “run show”. ü Click through by pressing any key. ü Focus & think about each point; do not just passively click. © Dr. David J. Mc. Kirnan, 2014 The University of Illinois Chicago Mc. Kirnan. UIC@gmail. com Do not use or reproduce without permission

Foundations of Research 12. Research Sampling. 2 q Define your target population q What group do you want to generalize to? q What is your sampling frame? q Who is / is not a member of the group? Shutterstock

Foundations of Research n n Sampling 3 Any study assesses only a sample of the population. We always must generalize from our sample to the larger population. n Even the census does not enroll 100% of Americans. n Research often addresses a specific population or sub-population. n n Our definition of the population to target is an important step. There are many different ways we may collect a research sample. The size and breadth of the population we are generalizing to can affect the Internal or External validity of the study. We will cover these topics here.

Foundations of Research The overall flow of sampling decisions. We begin with a decision about who we are interested in. General population or targeted sub-population We then make decisions about who is in the target population and how to recruit them. From there we collect our sample, typically a very small % of the larger population. We get results of our experiment or study within our sample. Sampling frame The sample Inference Study results …and attempt to infer what the entire target population must be like. 4

Foundations of Research 5 Sampling; the population We may design our study to inform us about a very general population. So, a cognitive-neuroscience study may test hypotheses about the brain generally. More typically we study a subgroup, e. g. women, a class of medical patients, the homeless. . . Or we may compare segments of the general population, e. g. , African -Americans & Caucasians. This more narrow focus makes generalization more clear. General population or targeted sub-population Sampling frame The sample Inference Study results Many Psychology studies purport to generalize broadly, but by enrolling college sophomores or survey lists actually generalize to a very small portion of the population.

Foundations of Research 6 The Sampling Frame Sampling frame refers to several elements of our study; General population or targeted sub-population • What do we already know about our population? § Do we have census or other Sampling frame data? • What criteria do we use to determine who is or is not a member of a target population? § If we are studying homelessness, who “counts” as homeless? The sample Inference Study results • How do we contact and recruit participants? § Where may a representative sample of our target population be reached? § How do we actually approach and enroll them?

Foundations of Research 7 Sampling; assignment There are many ways for us to sample our target population(s). The main distinction is Probability (or random) vs. Non-Probability sampling. We will address that later. General population or targeted sub-population Sampling frame Within our sample we may use Blocking Variables to compare different segments, e. g. , age or ethnic groups. The sample For experiments we assign participants to groups, typically using Random Assignment. Study results Inference Randomized Block Assignment may, for example, randomize within ethnic blocks, to ensure that the same proportion of African-Americans, Caucasians and Latinos are in each group.

Foundations of Research 8 Sampling: population inferences In Inferential Research we are not interested in simply describing or analyzing our sample. We use our results to Infer the characteristics of the larger population we sampled from. General population or targeted sub-population The quality of our inference is shaped by the representativeness of our sample. ü Did our sample accurately reflect the population we are generalizing to? The Reliability and Validity of our measures also affects how much we can say about our target population from our study. Sampling frame The sample Inference Study results

Foundations of Research 9 Sampling: population inferences In Inferential Research we are not interested in simply describing or analyzing our sample. We use our results to Infer the characteristics of the larger population we sampled from. Descriptive Research: General population or ü …Simply describes characteristics of targeted sub-population a sample or population. • Epidemiology: what is the distribution of a certain viral infection inframe the U. S. population? Sampling Inferential Research: The quality of our inference is ü … attempts to use a sample to infer shaped by the representativeness the characteristics of a larger The sample population. of our sample. Inference a ü Did our sample accurately reflect • I test the hypothesis that specific educational innovation the population we are enhances learning. Study results generalizing to? • I use statistical procedures* The Reliability and Validity of our to see how well that finding may measures also affects how much we generalized beyond my sample, to can say about our target population students at large. from our study. *Coming up soon!

Foundations of Research Sampling: population inferences In Inferential Research we are not interested in simply describing or analyzing our sample. We use our results to Infer the characteristics of the larger population we sampled from. General population or targeted sub-population Reliability: Sampling frame ü How stable are scores on the assessment over time (or across The quality of our inference is different measures)? shaped by the representativeness Theassume samplescores on a high ü We of our sample. quality measure do not change Inference ü Did our sample accurately reflect substantially over modest the population we are periods of time; Study results personality generalizing to? ü E. g. , a Reliable index should not yield different The Reliability and Validity of our scores when taken a 2 nd time. measures also affects how much we can say about our target population from our study. 10

Foundations of Research Sampling: population inferences 11 Validity: In Inferential Research we ü Predictive (or criterion) valid measures predict a specific behavior or attribute: are not interested in simply General population or describing or analyzing our targeted sub-population • An aptitude measure has predictive validity if it tells us who will do better sample. /worse in school or on a job. We use our results to Infer the ü A Content Valid measure assesses every characteristics of the larger Sampling frame element of a variable. population we sampled from. • e. g. , an exam that assesses every The quality of our inference is aspect of the course material. shaped by the representativeness ü Most important is Construct Validity: are The sample of our sample. we measuring what we think we are? Inference the basic ü Did our sample accurately reflect • Are we validly assessing psychological or behavioral process the population we are (Hypothetical Study results. Construct) underlying generalizing to? our theory. The Reliability and Validity of our • e. g. , is our measure of a specific measures also affects how much we emotional state (…alienation) actually can say about our target population assessing that, or just some general from our study. attribute such as depression?

Foundations of Research 12 Sampling: population inferences General population or targeted sub-population Of course to develop a sample we must deciding who, exactly, we want to generalize to. An important element of this is whether our sample is broad – encompassing many diverse people – or narrow, focusing on one sub-set of the population. Sampling frame The sample Inference Study results

Foundations of Research Who do you want to generalize to? 13 Mammals Humans All Western people Sampling a broader population (i. e. , larger sampling frame) increases external validity. All Americans Young Americans College students This College This class Sampling a more specific or smaller frame generally increases internal validity.

Foundations of Research 14 Who do you want to generalize to? Samples typically represent targeted sub-populations n n Demographic or ‘status’ groups; n Ethnicity, income or educational groups… n Geography; e. g. , urban dwellers… n n n Medical / clinical groups; people with a specific diagnosis or condition Behavioral groups n Registered voters n Home owners n Ever used marijuana… Targeting specific Click for test. groups increases Internal validity by decreasing the complexity of the sample. …but may lessen External validity by Shutterstock. com narrowing the focus. Shutterstock. com Groups defined by self-identification or subjective state n Views oneself as “highly likely to vote…”. n Above a ‘cut point’ on a stress, depression, or alcohol use scale n “Conservatives” vs. “Liberals”

Foundations of Research samples & validity Clinical drug trials illustrate the conflict between internal v. external validity in sampling. EXAMPLE n n n People with diverse symptoms and backgrounds see physicians for depression. To enhance internal validity drug researchers use exclusion criteria to select only participants who fit a specific definition of depression Zimmerman et al. suggest that too many exclusion criteria compromises the validity of this research area. (click image for article) Zimmerman, M. l, Mattia, J. I. , & Posternak, M. A. (2002). Are Subjects in Pharm-acological Treatment Trials of Depression Representative of Patients in Routine Clinical Practice? Am J Psychiatry, 159, 469– 473. 15

Foundations of Research. Exclusion criteria & validity EXAMPLE The study begins with a large N of people self-referred for depression They exclude those with serious mental illness, drug abuse or personality disorder… …whose symptoms are not severe enough, are suicidal, or who have other affective disorders. . …whose symptoms are too recent OR too long-standing… …and end up with a small, carefully selected sub-set of patients (8. 4% of general depression patients). 16

Foundations of Research. External EXAMPLE n n vs. internal validity in sampling Applying rigorous study selection criteria for drug trials excludes the great majority of routine depression patients. Rigorous participant selection for internal validity seriously compromises external validity in these studies. This can be a real problem: the drug is likely to be prescribed for anyone with routine depression, not just those like the 8% in the study. This leaves the actual usefulness of anti-depressant (and other) medications for the general population in doubt. 17

Foundations of Research. External EXAMPLE n n vs. internal validity in sampling Applying rigorous study selection criteria for drug trials excludes the great majority of routine depression patients. To be useful research must Rigorous participant for balanceselection needs for: internal validity seriously § external Careful validity subjectinselection compromises these studies. (internal validity) n n This can be §a real problem: the drug (external Representativeness is likely to be prescribed validity) for anyone with routine depression, not just those like the 8% in the study. This leaves the actual usefulness of anti-depressant (and other) medications for the general population in doubt. 18

Foundations of Research 19 Who is a group member? Are you between 14 and 30 and have a computer or smart phone available? A = Yes B = No imgur. com, public domain

Foundations of Research Who is a group member? Do you use Facebook or other social media 5 times a week or more? A = Yes B = No C = Not sure – lost count. Is Facebook making us lonely? Click. 20

Foundations of Research Who is a group member? Are you a “Facebook user”? A = Yes B = No C = Not sure – let me Facebook that. 21

Foundations of Research 22 Who is a group member? A = Yes B = No C = Maybe – I’m not sure Click for a Washington Post story; Who is Latino? Shutterstock. com Do you live in a neighborhood or town that is mostly Latino?

Foundations of Research Who is a group member? Do you speak Spanish? A = Yes B = No C =¿cuál era la pregunta? 23

Foundations of Research Who is a group member? Are you Latino? A = Yes B = No C = Maybe – I’m not sure 24

Foundations of Research Who do you want to generalize to: who is in the group? n Once we choose our sampling group, we must decide on criteria for membership… This is an important part of our sampling frame: 25

Foundations of Research n 26 Define the target population Once we choose our sampling group, we must decide on criteria for membership… n n To sample social media users do I use a … n Rough demographic criterion? n Behavioral criterion (which behavior? ) n Self-identification? To sample “Latinos”… n n n Clearer and narrower group criteria increases Internal validity by making the sample more homogeneous. Is geographic status specific enough? Is Spanish language the defining characteristic? Can / must one call oneself “Latino” (even if you do not speak Spanish…)? Shutterstock

Foundations of Research 27 Define the target population Who do you want to generalize to: who is in the group? n Once we choose our sampling group, we must decide on criteria for membership… n To sample social media users do I use a … n n Rough demographic criterion? Some of these criteria are reliably measure Behavioral criterion (which behavior? ) easier to than others; Self-identification? To sample “Latinos”… n n n ü Demographic variables are often available in census data Is geographic status specific enough? ü Behavioral or subjective Is Spanish language the defining characteristic? criteria require direct assessment, and can be less reliable. Can / must one call oneself “Latino” (even if you do not speak Spanish…)?

Foundations of Research Define the target population 28 Who do you want to generalize to: who is in the group? n Once we choose our sampling group, we must decide on criteria for membership… n To sample social media users do I use a … n n ü Of course different criteria may yield very Behavioral criterion (which behavior? ) different samples. Self-identification? ü Our choice of sampling criteria must be based on To sample “Latinos”… our theory, hypothesis, or research question. n Is geographic status specific enough? n n Rough demographic criterion? n n Is Spanish language the defining characteristic? Can / must one call oneself “Latino” (even if you do not speak Spanish…)?

Foundations of Research 29 Sampling criteria Demographic or ‘status’ marker Behavioral Subjective / selfidentification Who is a “Latino”? Neighborhood residence? Spanish speaking? Cultural practices? Self-description? # Hours registered Describes occupation as ‘student’ Who is a “Student”? Lives on a campus Who is “gay” or “lesbian”? Lives same-sex 2 person household? Sexual or other patterns? Self-identification as gay / lesbian? Pattern of behaviors and feelings? Describes self as “depressed”? Who is “depressed”? Received a diagnosis from MH professional ? Presents at Doctor’s office for general malaise?

Foundations of Research 30 Sampling criteria Demographic or ‘status’ marker Behavioral Subjective / selfidentification Who is “Latino”? Neighborhood residence? Spanish speaking? Cultural practices? Self-description? # Hours registered Describes occupation as ‘student’ Who is a “Student”? Lives on a campus Who is “gay” or “lesbian”? Lives same-sex 2 person household? Sexual or other patterns? Self-identification as gay / lesbian? Who is “depressed”? § Each criteria may meet the goals of a particular Received a diagnosis Pattern of behaviors hypothesis or empirical question. Describes self as from MH professional ? and feelings? § Of course different choices may lead“depressed”? to very Presents at Doctor’s different samples office for general § Some criteriamalaise? are easy to assess but may be only approximate § Others may require relatively difficult assessments

31 Foundations of Research Who do you want to generalize to: Your “Sampling Frame”. n What is known about your larger population? n Are there Census or survey data? Ø E. g. , are there “population” data on depressed people? Ø Do we know the demographic profiles of Facebook users? n n Data about your target population will help you determine how well your sample represents that population. What is its size, sub-groups, location…. Ø Where / how can I best recruit members of the population? Ø Will some sub-groups require different recruitment methods than others? Ø Will different recruitment methods be biased in favor of some subgroups? ü Internet surveys may be biased against older people. ü Studies that use monetary incentives pull for poorer people.

Foundations of Research Overview: From research question to sample What is the research question? ü Are we describing some natural process? ü …testing a theory? General population or targeted sub-population Sampling frame What is the population of interest? ü What population does your research address? ü Whom do you want to generalize to? The sample Category of participant criterion? ü Demographic or “Status” criteria? ü Behavioral criterion? ü Self-Identification, attitudes or beliefs? Operational definition of enrollment criteria? ü Inclusion & Specific measures that define who does / does not qualify for enrollment. Exclusion Criteria: Actual recruitment? ü Concrete (operational) processes to recruit and enroll participants. 32

Foundations of Research From theory to sample: Asthma among African-Americans. Study structure & research question: EXAMPLE ü Adherence to a medication regimen is key to health among people with asthma. ü Medication adherence is generally low, particularly among African. American adolescents, who have high rates of asthma. ü Self-determination theory proposes that autonomous motivation (being self-directed), self-confidence, and relatedness (family routines & parental support) underlie adherence. ü This study tests the hypothesis that three variables comprising self -determination theory will be associated with patients’ adherence to medications. ü Because young African-Americans have a significant health burden from asthma, the study focuses on them. Bruzzese, J. , Idalski C, Lam, P, Deborah A. ; Naar-King, S. (2014) Adherence to asthma medication regimens in urban African American adolescents: Application of self-determination theory. Health Psychology, Vol. 33. 5 (May 2014): 461 -464. Article here. 33

Foundations of Research From theory to sample: Asthma among African Americans. 34 Population of interest? ü Young African-Americans who suffer from poorly controlled asthma. EXAMPLE Category of participant criterion? ü Demographic or Status criteria § African-American adolescents ü Self-Identification / attitudes § Poorly controlled asthma. not a criterion in this study. ü Behavioral criterion § Already participating in long-term asthma control study. Operational definition of enrollment criteria? ü “Adolescent”: Age 10 – 18. ü “Poorly controlled”: At least one asthma-related hospitalization or two asthma-related emergency department visits in the last 12 months. Actual recruitment? ü N = 162 participants recruited from the hospital’s outpatient immunology clinic after an asthma-related clinic visit or hospitalization

Foundations of Research 35 Results EXAMPLE § Having asthma regulation embedded in the family routine was the only predictor of medication adherence. Multiple regression analysis (all variables are tested simultaneously) Shutterstock. com

Foundations of Research 36 Research sampling q Defining your target population q Probability & Non Probability sampling methods. Shutterstock. com

Foundations of Research Major forms of sampling 37 Probability (Random) Sampling § Recruit (or select) participants to maximize the representativeness of the sample to a known population. § Uses some form of random selection. § Requires that each member of the population has a known (often equal) probability of being selected. § Most externally valid approach to sampling general populations Non-Probability Sampling § Use available samples for convenience, or targeted outreach to unusual or small populations. § Selection may be either systematic or haphazard, but is not random. § Often the most externally valid approach to unusual, small, or extreme groups, or groups where little is known. § When used only for convenience it is the least externally valid.

Foundations of Research 38 Watch that word ‘random’! Participant Selection Sample Random Selection or a Random Sample refer to how we recruit participants; who is in the sample. Participant Assignment Experimental Treatment or Procedures Manipulation Results Group A Procedure Treatment Outcome Group B Procedure Control Outcome (Group C) (Procedure ) (Alternate Treatment? ) (Outcome) Random Assignment is how we (should) assign participants to different groups.

Foundations of Research Probability / Random Sampling 39 • Core feature: all members of the study population have an equal (or known) chance of being sampled • Procedure: Choose participants in a systematic, random fashion. • e. g. , every 100 th student ID, • Every 1000 th person on a voter registration record. • Random digit dialing for telephone surveys. • Advantages: eliminates obvious biases of convenience sampling • Limitations: • May under-sample unusual / hard to reach participants • Some may be unavailable in, e. g. , telephone lists, computer files.

Foundations of Research Basic Forms of random sampling • Simple Random Sampling: Select a specific % of a target population; all members of population have about equal chance of selection. • Multi-Stage: Randomly select population units (census tracts, households, schools. . ), then randomly select individuals within unit. • Stratified: Random within population sub-blocks, e. g. , gender (randomly select 50 women and randomly select 50 men), ethnicity, etc. • Cluster: Random within (potentially convenience) clusters, e. g. , specific locations or “venues”, events, times of day, etc. 40

Foundations of Research Simple Random sampling Objective: Attempts to truly represent the general population; absolute minimal selection bias. Procedure: Recruitment method where all members of the population have ≈ chance of being selected: n Examples: Polls or surveys using random digit dialing. 41

Foundations of Research Simple Random sampling Objective: Attempts to truly represent the general population; absolute minimal selection bias. Procedure: Recruitment method where all members of the population have ≈ chance of being selected: n Polls (e. g. , Gallup) or surveys using random digit dialing. Examples: ü Dialing random phone numbers eliminates any selection bias on the part of the researcher; calls are made by a computer. ü Everyone who has a phone has an = probability of being selected. ü Of course not everyone has a phone, many people block calls not in their contact list, and so on. ü These and other demographic trends induce an unknown degree of sampling bias. 42

Foundations of Research Simple Random sampling Objective: Attempts to truly represent the general population; absolute minimal selection bias. Procedure: Recruitment method where all members of the population have ≈ chance of being selected: n Examples: n Polls (e. g. , Gallup) or surveys using random digit dialing. “American Community Survey”; census based random sample of households. ü The census is designed to assess literally every American household. ü A small sample (2. 5%) of census respondents receive a longer survey of demographics and social patterns. ü Despite being a mandatory, highly systematic sample, not everyone completes the either the census or for the sub-sample. ü Both the poor or homeless and the very wealthy can be difficult to contact. 43

Foundations of Research Simple Random sampling Objective: Attempts to truly represent the general population; absolute minimal selection bias. Procedure: Recruitment method where all members of the population have ≈ chance of being selected: n Polls Examples: n Census, “American Community Survey”… Advantages: Most representative sampling frame for the general population Disadvantages: Any recruitment method excludes some people (no telephone, no stable address, etc. ). Face-to-face (actual contact, not telephone or internet) data collection is very expensive. 44

Foundations of Research Multi-stage random sampling Objectives: Develop a focused & efficient random sample. Use random sampling “stages” to reach hidden, stigmatized or other hard to reach groups. ü Simple random sampling is optimal. ü However, it is biased when relying on telephone, internet or similar contact methods. ü A simple random sample for face-to-face interviews or recruitment for a larger research study (e. g. , a public health intervention) is prohibitively expensive. 45

Foundations of Research 46 Multi-stage random sampling Objective: Focused & efficient random sample. General population Procedure: Use successive levels of random selection to narrow the sample. 1 st level random selection ü From the General Population, randomly select sub-population blocks. 2 nd level random selection ü From those blocks randomly select smaller groups… ü And smaller groups withing those blocks. . . ü To arrive at a efficient random sample. 3 rd level random selection Final sample

Foundations of Research 47 Multi-stage random sampling Selecting a random sample of college students: Rather than trying to randomly sample the entire population of students, we narrow our approach. ü Randomly select subpopulation blocks. General population Universities Classes ü Randomly select smaller groups within them… ü Randomly select students within the target classes. . . ü To arrive at a efficient random sample. Students Final sample

Foundations of Research 48 Multi-stage random sampling “Real World” example: NIDA* household surveys of drug use. General population ü Randomly select a moderate # of census tracts. Step 1. ü Randomly select modest # of households within each target tract… Step 2. ü Interview the first adult who answers the phone in each household. . . ü To arrive at a efficient random sample. Step 3 Final sample *National Institute on Drug Abuse

Foundations of Research Multi-stage random sampling Multi-stage sampling is also useful for “hard to reach” population where we have no sampling frame: § For the student or drug study we know what the population looks like: ü We have census data or university & class lists. § This gives us known population blocks that we can randomize around. § For a “hidden” population where we have no sampling frame… ü We do not know the size or geographic distribution of the group. ü We do not have simple population blocks available. § …we must often resort to convenience sampling in venues (locations, events) where the population may be available. § This can lead to substantial bias due to non-random selection. 49

Foundations of Research 50 Multi-stage random sampling How to we randomize within the constraints of convenience (venue) sampling? Population of unknown size & distribution. This example is from the “CITY” study of HIV risk among youth. Randomize Venues ü In each of 4 cities, randomly select bars, clubs & other venues attended by young gay men. * ü Randomly select days & times to recruit in selected venues. Randomize Day & Time Randomly select people Final quasiü Randomly approach every random sample 4 th person who enters the venue for an interview * Use qualitative interviews / direct observation to determine which venues are relevant.

Foundations of Research Multi-stage random sampling “CITY” study of HIV risk among youth. § Many important research questions involve people who would not be found in simple random selection methods. § With “hidden” or stigmatized groups sampling must be very targeted. § Targeted sampling can be biased toward members who are easiest to contact. § This approach adds random selection to what ordinarily would be a (biased) convenience sample. 51

Foundations of Research Multi-stage random sampling “CITY” study of HIV risk among youth. § This approach adds random selection to what ordinarily would be a (biased) convenience sample. § Of course the sample is biased by larger issues: ü Not all gay youth attend venues relevant to them. ü Concern over being identified as gay may lead some # of men to refuse the study. § Despite its limitations, this example shows how creative sampling approaches can provide a less biased sample of a difficult group to reach. 52

Foundations of Research Stratified or cluster sampling Objective: Represent every key segment of the population. Procedure: Ø Decide which population segments are important; § E. g. , ethnic groups, § Geographic areas, § Self-identification. This decision is based on your hypothesis or empirical question. Ø Randomly select from each segment. § Proportionate: Sampling fraction from each segment should approximate the overall population. ü ü This is the distribution of ethnicity in the U. S. A stratified sample would randomly select from each ethnic group to approximate this distribution. 53

Foundations of Research 54 Stratified or cluster sampling Objective: Represent every key segment of the population. Procedure: ØDecide which population segments are important; ØRandomly select from each segment. § Proportionate: Sampling fraction from each segment should approximate the overall population. § Dis-Proportionate: Over-sampling population groups to ensure you have large enough samples of small groups. This estimates the distribution of legal vs. illegal immigrants. To directly compare groups we over-sample illegal immigrants.

Foundations of Research Probability Sampling overview 55 Core features: n Random selection of participants from the population n Most externally valid approach. n Assumes: ü A clear sampling frame. Summary ü All segments of the population are available. Variations: Each member of the entire population has an ≈ chance of being selected. Ø Simple a. b. Select segments of the population, e. g. , census tracts, registered voters. Each segment member has an ≈ chance of being selected. Ø Cluster a. b. Identify clusters, e. g. sports fans at a sports bar. Each cluster member has an ≈ chance of being selected. Ø Stratified a. b. Identify strata, e. g. , ethnic groups, gender, age groups. . Randomly select a proportion of each strata. Ø Multi-stage

Foundations of Research Non-Probability Sampling 56 Useful for populations that: ü Cannot be randomly sampled; “hidden” or difficult to reach. ü No sampling frame available, such as census data, describing its size, composition, etc. Examples: drug users, recent immigrants, gay men… Likely to misrepresent the population ü It may be difficult or impossible to detect this misrepresentation. ü Can be over-sensitive to incentives: paying participants attracts more poor people. ü “Respondent Driven” sampling (RDS) allows for “targeted” population estimates.

Foundations of Research Types of Non-probability Samples ü Haphazard ü Modal instance ü Venue – time / space ü Multi-frame ü Snowball / Respondent driven ü Web ü Quota ü Heterogeneity 57

Foundations of Research Haphazard Sampling “Man on the street”; recruiting the most easily available participants. n n n Literally recruiting in public places for, e. g. , brief interview n College psychology majors. n Medical / therapy clients in a clinic. Often used for quick interview / attitude studies on current topics. Advantage: Participants are readily available. Problem: No evidence for representativeness. 58

Foundations of Research n Modal Instance Sampling Recruit a “typical case” of a target population. n n A member of a target population group; ü …a ‘typical’ college student with debt. ü An injection drug user, homeowner, etc. A person affected by a major event; ü A 9/11 survivor. . . ü A witness to a natural disaster. § Often used to describe an event or way of life, or to generate hypotheses for later research. Advantage: Direct, personal description. Problems: Potentially strong self-selection bias in who volunteers for such personal disclosure “Social desirability responding” – presenting oneself in a positive light – or biased recall may compromise accuracy of answers. 59

Foundations of Research n Haphazard & Modal Instance Sampling 60 Both Haphazard & Modal Instance Sampling are used for Case Studies. n n Researchers elicit in-depth personal accounts of an event or life pattern. Multiple Case Studies interview a set of individuals who express the phenomenon, e. g. , members of a key group… ü In the Qualitative Research module we will discuss how in-depth interviews can be analyzed. n These methods, plus direct observation, are often used by journalists. ü An interesting example is a New York Times discussion of the “down low” phenomenon among African-American, bisexually active men. ü A journalist followed several young men as they went through their “straight” and gay lives, documenting how they approached different relationships and social settings. ü While not a large sample, this in-depth method provides a rich and detailed description of a specific life pattern.

Foundations of Research n Venue and Time / Space Sampling 61 Assumes that members of a specific population group are well represented in particular places & times (“venues”). n The GLBT community has a range of specific places or venues where people congregate or are available. n n Occupational support groups, bars / clubs, sports teams, political groups… Ethnic groups may also have specific venues; n Barbershops, hair salons or churches are occasionally used to sample African. Americans.

Foundations of Research n n 62 Venue and Time / Space Sampling Assumes that members of a specific population group are well represented in particular places & times (“venues”). Venue sampling often uses “Intercept” methods to reach participants. n n n Outreach workers use a standard recruitment script to approach potential participants. Data may be collected on site via a brief interview, such as shopping mall intercepts. Often the contact is used to collect (or distribute) contact information for later participation

Foundations of Research n n 63 Venue and Time / Space Sampling Assumes that members of a specific population group are well represented in particular places & times (“venues”). Venue sampling often uses “Intercept” methods to reach participants. Time / Space randomization lessens bias due to choice of venue: n Randomly approach different venues at different times n Randomly select participants within the venue (e. g. , every 4 th person…) These strategies must be based on a clear epidemiological or theory question.

Foundations of Research 64 Example of Venue sampling Recruiting gay or bisexual men for HIV research can present challenges. Example § Simple recruitment methods for any targeted population include: • Newspaper or internet ads, David J Mc. Kirnan • Flyers in health clinics or popular stores. . § Many gay/bisexual men – particularly younger & minority men – do not respond to simple methods… • …due to distrust or disenfranchisement from the health care system, • Unwillingness to disclose sexual orientation in other than ‘gay friendly’ settings, • The perception that research is irrelevant to them or may harm them. • Direct personal contact within gay/bisexual venues such as clubs can help break down barriers to recruitment.

Foundations of Research Outreach / venue sampling § Project MIX was a national safer sex intervention study sponsored by the Centers for Disease Control. Example § It had a complex sampling frame: • ⅓ each African-American, Latino & Caucasian, • ½ HIV infected / uninfected within each ethnic group. § Outreach workers recruited in multiple venues; • Bars and clubs • Public areas; parks and neighborhoods where men congregated • Community events, private ‘house’ parties, etc. § Outreach workers were indigenous, i. e. gay/bisexual men from the community. • They were able to approach men in the target groups. • They could explain the study and foster trust & cooperation. 65

Foundations of Research 66 Outreach / venue sampling Example To broaden the sample, recruitment flyers show different ethnicities, and cite a range of potentially eligible behaviors… Outreach workers complete a Contact Sheet for each person who responds to the study in a given venue, to standardize later enrollment. Images used with permission, David Mc. Kirnan, Project MIX Principle Investigator.

Foundations of Research Outreach / venue sampling 67 Example Note how discretion is build right into every aspect of recruitment methods and study design. ü Since Howard Brown Health Center is known as a GLBT organization, being affiliated with it may cary stigma for some people. Researchers using any recruitment method need be aware of their ethical responsibilities… Images used with permission, David Mc. Kirnan, Project MIX Principle Investigator.

Foundations of Research n n Targeted Multi-frame Sampling 68 Often used to sample a specific, hard to reach group…. . with no census or similar data to help us develop a sampling frame. n n n There are no clear “population blocks” that we may use for a multistage sample; The population may be spread among many venues or locations…. . . and different segments of the population may be more (or less) sensitive to any specific recruitment approach.

Foundations of Research n n 69 Targeted Multi-frame Sampling Often used to sample a specific, hard to reach group…. . with no census or similar data to help us develop a sampling frame. Based on preliminary qualitative work we develop multiple sampling “frames”: n Interviews, n focus groups n direct observations… n Direct outreach in specific places, such as street or park locations n Newsletters, internet lists, Facebook, web sites, chat rooms n Organizations or meeting places. This is the most common & valid form of convenience sampling.

Foundations of Research 70 Targeted Multi-frame Sampling n Sample a specific, hard to reach group n No census or similar data for sampling frame. n Uses multiple (convenience) sampling “frames”. Example: recruiting gay men for HIV research We use multiple ways to approach men or for them to contact us. Multi-frame sample n Direct outreach in bars, clubs, street. n Community events (festivals…) n Newspaper ads. n n n Flyers in bars & stores. Medical clinics. “Snowball” / word of mouth.

Foundations of Research Targeted Multi-frame Sampling 71 ü Each sampling “frame” is a convenience approach – we typically cannot randomly select participants. ü By using multiple frames we can recruit a broad cross-section of the population. ü This also allows us to “test” different sampling approaches or venues: Ø For HIV research: Do riskier men come from one type of sampling frame? E. g. bar venues… Ø Are there ethnic or other differences in the participants who are recruited in one type of venue or another…? Ø These analyses can help us better understand our sample and study results.

Foundations of Research n An initial set of participants are recruited according to a specific set of inclusion / exclusion criteria: n n n Snowball / Respondent Driven sampling (RDS). 72 Characteristics of the initial participants – the “seeds” of the network – help determine who subsequent participants will be. The seeds may be, e. g. , injection drug users (inclusion criterion) who do not live outside a specific geographic area (exclusion criterion). Initial participants are paid to recruit others, who are paid to recruit still others … all using the same in(ex)clusion criteria. Over “waves” of recruitment RDS recruits a network of “linked” people; n A network of, e. g. , musicians, gamblers, sexual partnerships… n Population of an organization or school… With enough waves of recruitment, Snowball / RDS can produce accurate population estimates for a given subpopulation.

Foundations of Research § Snowball / Respondent Driven sampling (RDS). Snowball / RDS can also provide insights into links among participants: § Since participants recruit each other, we can track who is linked to whom. § Study measures can determine the characteristics of people who are linked (e. g. , sexually) with each other. § We can also assess people who are “nodes” in social networks: who knows (recruits) a lot v. few people? ü What influence do those people have? § Problem: Sensitive to incentives; § Advantage: Access unusual or “hidden” people related by a common behavior or venue. § Using participants to recruit others they know may yield a more representative sample. § Researchers can collect data on who recruits whom to examine the structure of social networks 73

Foundations of Research RDS coupon examples 74 § These are examples of cards used in an RDS recruitment among injection drug users. § Initial “seeds” are interviewed, then given 5 cards to distribute to people who meet the eligibility requirements. § 2 nd “wave” participants receive the same card set. § Participants get $30 for the study interview, and $20 for each person they refer. Heckathorn, D. D. & Magnani, R. (2004). Snowball and Respondent-Driven Sampling. In: Behavioral Surveillance Surveys: Guidelines for Repeated Behavioral Surveys in Populations at Risk of HIV

Foundations of Research RDS; chain description This shows linkages within the RDS sample collected with the cards. We can see how people from different geographic, ethnic and gender groups interact (and potentially transmit HIV), …and social “nodes”: people who influence many others (and may have a key role in transmission). Advanced research can even track the strains of HIV moving through these networks. Heckathorn, D. D. & Magnani, R. (2004). Snowball and Respondent-Driven Sampling. In: Behavioral Surveillance Surveys: Guidelines for Repeated Behavioral Surveys in Populations at Risk of HIV. 75

Foundations of Research Here is another example of social network sampling: Bearman et al. , Romantic ties among adolescents 76 With a number of smaller chains This examines the social network of “Romantic Ties” among students at a specific High School. And a small % in 2 to 4 person chains n n A substantial majority of students are in an extended, linked chain of relationships. Using network sampling, several “seeds” access most of the population These findings suggest a clear potential for Sexually Transmitted Infections (STIs). Bearman, P. et al. , American Journal of Sociology, Volume 110 Number 1 (July 2004): 44– 91. Click image for article.

Foundations of Research Quota sampling. ü Select people Similar to cluster sampling, except you cannot randomly sample each non-randomly according to quotas population segment. ü Must have clear theory / research question to pick relevant population characteristic(s). ü Proportional quota sampling • Represent major characteristics of a population. If gender is important, and the proportion of women : : men in your population = 65% : : 35%, the sample must meet that quota. ü Non-proportional quota sampling • Sample enough members of each group to test hypothesis, even if the sample is not proportional. (e. g. , recruit 50 women & 50 men, even though the real proportion is 65: : 35). • Helps assure that you have good representation of smaller population groups. 77

Foundations of Research n Web sampling. Typically highly targeted samples P Gay / bisexual men… P Adolescents… P “Gamers”… n Typically access through existing venues: P Users of specific web sites P List-serves, e-mail lists P Active recruitment on Facebook or similar pages Problem: Inherent bias in computer literacy(? ) Advantage: ü Cheap large national sample ü Access unusual or “hidden” people who reach others via internet 78

Foundations of Research Heterogeneity sampling. • Sample every sector of a population -- at least several of everyone -- without worrying about proportions. • At least some members of each geographic area… • …racial, ethnic or income group… • …behavioral group (voters & non-voters…) • Assume that a few people are a good proxy for the group. Examples: focus groups or qualitative interviews about products, social issues. . . Problem; Cannot be sure a few people really represent their sub-group. Advantage: At least some representation of all subgroups. 79

Sampling overview Foundations of Research 80 Who do you want to generalize to? n Summary n n Who is the target population? n broad – external validity n narrow – internal validity How do you decide who is a member? n demographic / behavioral criteria? n subjective / attitudinal? What do you know about the population already – what is the “sampling frame”. Is a Probability or random sample possible? n “Hidden” population? n Socially undesirable research topic? n Easily available via telephone, door-to-door? n Sampling frame adequate to choose selection method?

Foundations of Research Overview, 2 Types of Non-probability Samples Ø Haphazard Summary Ø Modal instance Ø Venue – time / space Ø Multi-frame Ø Snowball / Respondent driven Ø Web Ø Quota Ø Heterogeneity 81

Foundations of Research Overview, 82 3 Probability sampling ü Most externally valid ü Assumes: ü Clear sampling frame ü Population is available Summary ü Less externally valid for hidden groups. ü Generally less externally valid Non-probability sampling n targeted / multi-frame n snowball n quota, etc. ü “Convenience” rather than random. ü Best when: ü No clear sampling frame ü Hidden / avoidant population. ü Multi-frame or snowball samples: ü Much more representative than other convenience approaches, ü But much less convenient….