Foundations of Research 13 Survey Research Part 1

Foundations of Research 13. Survey Research Part 1 This is a Power. Point Show ü Open it as a show by going to “slide show”. ü Click through it by pressing any key. ü Focus & think about each point; do not just passively click. © Dr. David J. Mc. Kirnan, 2014 The University of Illinois Chicago Mc. Kirnan. UIC@gmail. com Do not use or reproduce without permission Center for Epidemiologic Studies National Institute of Mental Health 1

Foundations of Research 13. Survey Research Part 1 We will address five topics in the Survey modules: Part 1 will address the first four. Topic areas & formats General issues in Survey research Sources of bias (or fraud…) in survey research Examples of surveys Testing Hypotheses with surveys 2

Foundations of Research 3 Survey Research ✓Topic areas & formats General issues in Survey research Sources of bias (or fraud…) in survey research Examples of surveys From: http: //comicsalliance. com/dc-comics-market-research-new-52/

Foundations of Research What do surveys measure? 4 § Knowledge § Information re: current events, political or consumer choices § Awareness: e. g. , of Public health resources, government decisions. . . § Attitudes and Beliefs § Preferences or evaluations: e. g. , attitudes toward racial or ethnic groups, consumer preferences. . . § Beliefs about political or social events: “which party provides the strongest security for the U. S…. ? ” § Feelings or moods: quality of life, depression / anxiety, marital satisfaction. . . § Behavioral intentions; Intent to vote, financial plans, exercise goals. § Self-reports of previous or on-going behavior; voting in the last election, alcohol and drug use, exercise patterns.

Foundations of Research Survey research; General uses of surveys Survey methods have a wide range of applications, ü from single-item consumer satisfaction (“How useful did you find this web site”)… ü …to full-fledged, theory-driven behavioral research. For convenience we will consider 5 categories: § Descriptive research § Testing hypotheses § Testing the generalizability of experimental results § Predicting an event or outcome § Pragmatic / applied questions 5

Foundations of Research Uses of surveys; descriptive research § Descriptive research § Epidemiology is the study of how behaviors, disease states, or similar issues are distributed across the population. § Epidemiology uses many methods, such as standard crime or disease reporting. § Even Google search data can be used to track topics such as interest in healthier foods. Number of Google searches for “gluten” and “probiotic” over time. (Click image for article) 6

Foundations of Research Uses of surveys; descriptive research § Descriptive research § Epidemiology is the study of how behaviors, disease states, or similar issues are distributed across the population. § Epidemiology uses many methods, such as standard crime or disease reporting. § Google Trends and other sites are valuable adjuncts to health monitoring for tracking, e. g. , heroin overdose. § Search data can also be used for less serious reasons… 7

Foundations of Research Counting Google searches to assess social interest 8 Does the Chicago Cubs’ World Series quest inspire national interest? Google Search data during the MLB Division playoffs, 2016: § Interest in the LA Dodgers is limited to California & Nevada; § Interest in Cleveland is limited to Ohio; ? § Interest in the Toronto Blue Jays is limited to Canada; § The Cubs truly arouse national attention.

Foundations of Research Uses of surveys; descriptive research § Descriptive research § Epidemiology is the study of how behaviors, disease states, or similar issues are distributed across the population. § Epidemiology uses many methods, such as standard crime or disease reporting. § Google Trends and other sites are valuable adjuncts to health monitoring for tracking, e. g. , heroin overdose. § Epidemiological studies often use direct survey methods, such as phone or face-to-face survey interviews. § Knowledge of, e. g. , how to access health care… § Feelings or moods, such as the rate and distribution of depression… § Behavioral patterns, such as alcohol or drug use or gun ownership… § Or archival data, such as disease reporting. 9

Foundations of Research The origins of epidemiology 10 § In London of 1854 a cholera outbreak raged through several poor neighborhoods of London. § Sewage and other effluvia that ran through gutters created a dreadful smell (a “miasma”) that was blamed for the outbreak. § The concept of infectious disease transmission through water supplies was not well understood. § Dr. John Snow, one of the physicians charged with stopping the epidemic, noted a particularly fetid cesspool in front of 40 Broad St. , proximal to a water pump used by the neighborhood. § He decided to empirically map the cholera cases in the area. § He proposed that water from the pump, not the miasma, was the cause of the outbreak. § He was generally disbelieved, but convinced the town governors by his evidence. Image: http: //leabridge. org. uk/waterworks/timeline/1 852 -1880. html Only much later would tracking of disease outbreak be labeled ‘epidemiology’. Click for Wikipedia article. Dr. John Snow’s Cholera map & the closing of the Broad St. pump.

Foundations of Research 11 The origins of epidemiology Dr. John Snow’s Cholera map & the closing of the Broad St. pump. of cases to be near the pump at 40 Broad St…. Click for Ted talk by Steven Johnson. § …and to radiate out from there. § As he noted in his 1855 book: "I had an interview with the Board of Guardians of St. James's parish, on the evening of Thursday, 7 th September, and represented the above circumstances to them. In consequence of what I said, the handle of the pump was removed on the following day. ” § By carefully describing the distribution of cases and the circumstances around the pump, Snow was able to empirically demonstrate a likely cause. § His hypothesis was supported by the epidemic quickly subsiding once the pump handle was removed. S. Johnson, The Ghost Map (2007), Riverhead Books. § Snow’s map showed the bulk

Foundations of Research Uses of surveys; descriptive research 12 § Descriptive research § Epidemiology § Political / social description is what we often think of as surveys. § Opinion polls about society, the government, or current events, e. g. Gallup Polls, or systemic studies by Pew Memorial Trust. § The Consumer Confidence Index is a highly standardized poll that is used for basic economic decision making. § The Census, of course, is our national information source.

Foundations of Research Uses of surveys; descriptive research § Descriptive research § Epidemiology § Political / social description. § Testing hypothesis § Assessing blocking variables § We often assess blocking variables to test how a given attitude or behavior varies across important social groups. e. g. , gender, age group, ethnicity, geographic location… 13

Foundations of Research Uses of surveys; descriptive research § Testing hypothesis § Assessing blocking variables § Here is a 15 -year trend in trust in political leaders, blocked by self-reported political affiliation Click for poll 14

Foundations of Research Uses of surveys; descriptive research 15 § Descriptive research § Epidemiology. § Political / social description. § Testing hypothesis § Assessing blocking variables § We often assess blocking variables to test how a given attitude or behavior varies across important social groups. § Correlational studies § A key form of analysis is examining the association among different variables. e. g. , what are the correlates of dieting…

Foundations of Research Uses of surveys; descriptive research 16 § Testing hypothesis § Assessing blocking variables § Correlational studies § A key form of analysis is examining the association among different variables. § The theoretical model we have been using is a good example of a correlational study framework. § In a survey we may develop measures of each construct, and test the model through correlation analyses.

Foundations of Research Uses of surveys; descriptive research § Descriptive research § Epidemiology § Political / social description. § Testing hypothesis § Assessing blocking variables § Correlational studies § Examine generalizability of experimental results § E. g. , The Consumer Reports survey on therapy we discussed in quasiexperiments § Predict event or outcome; § E. g. , election polling § Pragmatic / applied, § E. g. , marketing, or consumer surveys. 17

Foundations of Research Surveys; populations § Who do we want to generalize to? § Our sampling frame is based on our hypothesis or empirical question. § Sampling: breadth internal validity tradeoff § Key dimensions: § Demographic § ethnic / age / gender groups, § “all Americans”… § Behavioral § “likely voters”, alcohol users, home buyers. . . § Self-identification § Republicans / Democrats, § “students”… See Design and sampling overview See also: diminishing validity of political polling. 18

Foundations of Research How do we ask questions in a survey? § Survey questions (“items”): operational definition of your phenomenon. o Cast topic (stress, “morality”, attitude…) as specific, concrete statement or question. o Similar to Dependent Variable in an experiment. § “Closed-ended” items: o Rating scale, agree / disagree, checklist… o Highly structured, specific question; “top-down”. o Clearest operational definition. § “Open-ended” o Fill in the blank, open writing, listing… o More general & person based; “bottoms-up”, o More difficult to capture specific phenomenon. 19

Foundations of Research Question Formats: Closed-ended items n Specific rating scale or a highly structured prompt. n These are most reliable for concrete, specific behaviors or attitudes. n An attitude can be assessed in several ways: n Direct (face valid) assessment Research methods is a wonderful course… n Does not 1 2 3 4 5 6 7 strongly agree at all agree Behavioral (content valid) indictors How many times this semester have you skipped class? How many hours per day do you spend reading the material? n n 20 0 ______ 2. 5 ______ Researchers typically use the Mean (…average) of several related items to create an attitude scale. Scales are more reliable measures than are single items.

Foundations of Research n n 21 “Closed-ended” items, cont. Example: The Center for Epidemiological Studies Depression inventory (CESD). The M score of these 9 items is often used as a depression scale… Moods & Feelings Below is a list of different feelings. Circle the number that shows how many days you felt each of these over the PAST WEEK. Rarely or A Little none of of the Time the time (less than 1 day) A moderate Most or all of amount of the time (1 or 2 days) (3 - 4 days) (5 - 7 days) I was bothered by things that usually do not bother me. 0 1 2 3 I felt I could not shake off the blues even with help from my friends or family. 0 1 2 3 I had trouble keeping my mind on what I was doing. 0 2 3 I felt depressed. 0 1 I felt that everything I did was an effort. My sleep was restless. 0 I was happy. 0 1 I enjoyed life. 0 1 I felt sad. 0 1 2 2 3 Sum 1 of item ratings / 9 3 1 2 These are 2 items 3 final 3 score 3 3 “reversed” in the

Foundations of Research n n n“Closed-ended” 22 items, cont. Or we may count the number of symptoms. A scale like this typically has a cut point: Moderate depression is defined as 4+ symptoms. Below is a list of different feelings. Circle the number that shows how many days you felt each of these over the PAST WEEK. Rarely or A Little none of of the Time the time (less than 1 day) A moderate Most or all of amount of the time (1 or 2 days) (3 - 4 days) (5 - 7 days) I was bothered by things that usually do not bother me. 0 1 2 3 I felt I could not shake off the blues even with help from my friends or family. 0 1 2 3 I had trouble keeping my mind on what I was doing. 0 1 2 3 I felt depressed. 0 1 I felt that everything I did was an effort. My sleep was restless. 0 I was happy. 0 1 I enjoyed life. 0 1 I felt sad. 0 1 2 2 3 1 2 3 3 # of symptoms: items 3 rated 2 or 3 Center for Epidemiologic Studies National Institute of Mental Health Moods & Feelings

Foundations of Research n“Closed-ended” 23 items: standard scales n n Scores on the CES-D have been shown to predict scores on longer and more systematic depression inventories. This allows a brief, survey-sized measure to be used as an indicator of depression. As we have seen, the scale can yield Continuous measures of depression, • • n Standard scales like the CESD are used in many studies & diverse populations, so their properties are well understood. Mean (average) ratings, or symptom counts raging from 1 to 9, …good for correlational analyses (how strongly does depression correlate with drug use…). Or a categorical measure (depressed – non-depressed), based on the scale cut-point, • . . . appropriate for quasi-experiments that, e. g. , compare the depressed to the nondepressed group on a variable such as risk taking.

Foundations of Research Evaluating our measures: Reliability and Validity. Reliability § If we are assessing a stable characteristic (IQ, personality, temperament, core values…) participants should show similar scores over time, or even across the life span. § If they do not, our measure may be unreliable. § For a more transient characteristic (mild depression) we expect scores to change over time. Validity 24 § Our survey or scale must actually measure what we designed it to. § There are several ways we think about validity, each getting at a different element…

Foundations of Research n Tests for Reliability Test - retest; n n n 25 Are responses consistent over time? Assume a stable attribute; e. g. , “temperament”. If the measure is reliable, participants should show similar scores across time, e. g. , at baseline and after a year. If scores do change over time, there are two possibilities: 1. Our measure is not reliable – there is a lot of “error” variance in scores – so we should reconsider how we are assessing the construct. 2. The very (hypothetical) construct we are studying may not be valid (e. g. , “temperament” may not be a single, stable construct…), and we should reconsider our theory. n Split-half; Do different sets of items give use the same scores? n n Multi-item scales are designed with overlapping or converging items. If a 10 -items scale such as the depression inventory is reliable, the first 5 items should yield scores similar to the last 5.

Foundations of Research n 26 A scale or item appears “on its face” to measure what it is designed to: n n n “How dependent are you on heroin? ” “How much did you enjoy…”. An item is “intuitively valid”; clearly addresses the topic. A strongly face valid item (heroin use…) may push for socially desirable responses (“No no, not me…”). Content validity n Assesses all key components of a topic or construct: n n Validity Face validity n n Descriptive research: Simple skill index; assess computer skills by having a job applicant write a simple program “Depression” may be best assessed by measuring Knowledge, Attitudes & Moods, and Behavior. A content valid scale would address all of these. Exams; for research methods we should test all the core skills for research design… Predictive validity n Validly predicts a hypothesized outcome: n n e. g. , I. Q. is a moderately good predictor of college & job success, criminality, etc. The “Big Five” personality inventory is a moderate to good predictor or responses to different challenges or social contexts.

Foundations of Research Do people who use predictive tests in business know or care about their scientific validity? § Some widely used predictive measures – that themselves have become big business – predict little or nothing. § The Myers-Briggs Type Indicator (MBTI) is used to predict job success in 89 of the Fortune 100 companies. § Knowing which of the 16 MBTI “types” you are may feel good to you and your prospective boss, but has no reliability or validity. § Click for a good overview of reliability & validity, and the MBTI. 27

Foundations of Research n Descriptive research: Validity (2) 28 Construct validity n Test whether the hypothetical construct itself is valid (differs from other constructs, corresponds to measures or outcomes it should. . ). n n Test if the Measure addresses the construct it was designed for n n E. g. ; “anxiety” and “depression” and “anger” may not be separate constructs, but may all be part of “negative affectivity”. e. g. , measures of social support (“do you have people who care for you”) often strongly influenced by depression, a separate construct… “Ecological” validity n n n Measure corresponds to how the construct “works” in the real world External validity of assessment device. See Focus Module 10, for External / Ecological Validity.

Foundations of Research 29 “Closed-ended” items: standard scales n Because the CES-D has multiple items developed over many studies the scale has high Test. Retest Reliability. • n Scores taken at one time generally are consistent with scores from another time. The scale also has high Split. Half and Inter-item reliability. • n The scale has high Face Validity… • n Different items correlates highly with other (sets of) items. Items clearly “map onto” or directly assess the concept of depression. … Content Validity… • Different aspects of depression – self perception, behavior, sleep – are all assessed. • and Predictive Validity. . . • Scores correlate with (predict. . . ) both larger, clinic-based assessments of depression and depresson-related outcomes such as job performance.

Foundations of Research 30 “Closed-ended” items: standard scales n Because the CES-D has multiple items developed over many studies the scale has high Test-Retest Reliability. • n Scores taken at one time generally are consistent with scores from another time. The scale also has high Split. Half and Inter-item reliability. • n The scale has high Face Validity… n … Content Validity… Different items correlates highly with other (sets of) items. § and Predictive Validity. . . §. . . but mixed Construct Validity: • Depression scores tend to correlate with scales of anxiety, anger, and social isolation. o This may be a meaurement problem: survey measure of any negative mood correlates with other negative mood scales, o Or that the hypothetical construct of depression overlaps with other constructs such as anxiety and anger.

Foundations of Research n Chief virtue: clear operational definition of our study variable(s). n n n Closed-ended items, summary Items are specific & concrete; we know exactly what the participant is responding to. Face or content valid items can be written that correspond exactly to our empirical questions or hypotheses. n Items or scales can be easily tested for internal reliability. n Quantitative scales can be used directly for statistical analyses. Chief liability: potential insensitivity. n Items are typically brief & simply worded; potentially superficial. n Quantitative surveys are “top down”; n n n The participant often does not have the ability to indicate what issues are of personal interest. Unlike in qualitative research, attitude or mood scales may not be sensitive to participants’ personal perspectives. Surveys that mix quantitative with qualitative items can enhance sensitivity, although qualitative items can be difficult to analyze. 31

Foundations of Research Survey formats; Open-ended items 32 Textual / qualitative responses; n More sensitive to the participant: What have you enjoyed most about your methods class so far? Please list the three things that first come to mind when you think of Research Methods in Psychology. . . § Highly sensitive measures of, e. g. , social support, may have the participant write the initials of the 5 people closest to him/her, then describe the forms of support each provides. § This approach is easily combined with quantitative measures, via rating scales for each person… § By analyzing themes that emerge in participants open-ended descriptions (e. g. , of their voting decisions, what they consider socially supportive. . . ) researchers can discover how important social processes work “on the ground”, rather than imposing a hypothesis or expectation on participants’ answers. § As we saw from the module on descriptive research, these open-ended data can be used to develop quantitative measures or analyses.

Foundations of Research 33 Survey formats; Open-ended items Textual / qualitative responses; n More sensitive to the participant n More difficult to collect and interpret: § People simply do not like to write. § “Fill in the blank” or “Please write a brief paragraph…” items often get minimal responses. § Social groups differ in their writing ability or comfort. § Data may thus be biased by those with higher eduction or more willing to write.

Foundations of Research 34 Survey formats; Open-ended items Textual / qualitative responses; n More sensitive to the participant n More difficult to collect and interpret: § People simply do not like to write. § Analysis often involves subjective judgments on the part of the researcher. § Researchers use computer programs and their own judgment to specify the major themes in a body of textual data, by developing and applying thematic codes. § Thematic codes are then used to show associations among different issues or topics, e. g. , in specifying the personal characteristics that participants most commonly associate with social support. § Since coding must involve researchers’ judgments, it can run the risk of researchers “discovering” in the data themes they expected to see in the first place. § There are, however, a variety of methods researchers use to improve and test the reliability of coding.

Foundations of Research 35 Mixed survey formats Here is an example of a mixed question format from a survey of women’s sexual practices. n Closed-ended attitude scale n Open-ended qualitative description n Simple behavioral index. Personal Safer Sex Guidelines How strict are your personal guidelines or rules for safer sex (e. g. , condom use, “safe relationships, ” etc. )? 1 Not at all Strict 2 3 Somewhat Strict 4 5 6 Very Strict 7 Extremely Strict What are your rules for safer sex? Have you ever refused to have sex with someone to stay safe? 0 1 2 3 Never twice once or times a few times many

Foundations of Research ✓ ✓ 36 Mixed survey format § Mixed survey formats allow researchers to “triangulate in” on a topic or empirical question. § Here the researcher is assessing different components of women’s decision making. § The general attitude scale assesses participants’ subjective sense of the importance of personal safety. § The open-ended item allows women to make a personal statement that the researchers may not have expected or assessed. o For example, many participants cited “a single close relationship” or “non-sexual relationship” as their primary safety strategy. o A proportion of participants wrote in “no rules”, an unanticipated and important response. o Of course leaving the item blank cannot be interpreted: does it mean “no rules” or “did not what to write an answer…”.

Foundations of Research 37 Mixed survey format § Mixed survey formats allow researchers to “triangulate in” on a topic or empirical question. § Here the researcher is assessing different components of women’s decision making. ✓ § The general attitude scale assesses participants’ subjective sense of the importance of personal safety. § The open-ended item allows women to make a personal statement that the researchers may not have expected or assessed. § The behavioral item provides a concrete index of whether maintaining safety has been problematic enough to influence an important behavior. § Responses to these items do not necessarily cohere perfectly, but the combination allows the researcher to more sensitively portray participants’ behavior than a single item or type of item.

38 Foundations of Research Marketers have used multi-format surveys to rap with customers for years! Click to Dig!

39 Foundations of Research Note: § Comics were first beginning to reflect larger social issues in the late 1960’s. § Marketers were genuinely uncertain about emerging topics of interest among their young readers. § (“Black people”? ) From: http: //ifanboy. com/articles/great-moments-in-comics-history-1970 -dccomics-reader-survey/

Foundations of Research SUMMARY n n 40 Survey topics & item types Surveys assess: n Knowledge n Attitudes or preferences n Ongoing or intended behavior Closed-ended formats Psychology 242 is a wonderful course… n Highly structured, easy to analyze n Potentially insensitive Open-ended formats …list the three things that first come to mind… n More sensitive to the participant n Potentially ambiguous or difficult to analyze Surveys typically… n Use multiple items n Employ several formats.

Foundations of Research Survey Research Topic areas & formats ✓General issues in Survey research Sources of bias (or fraud…) in survey research Examples of surveys 41

Foundations of Research n Self-report questionnaire n n Forms of survey administration 42 “Paper and pencil” or internet-based; n Primarily closed-ended, structured questions n Limited open-ended items All data collection increasingly computer-based n Assume at least moderate reading level n Cheap & easy to administer n Internet: Representativeness very dubious Face-to-face interview n “Door step”, formal research center, or telephone n Allows in-depth qualitative questions n Many studies combine questionnaire & interview formats n (Telephone version becoming obsolete)

Foundations of Research n 43 Cost / population access n Different methods are more / less likely to reach certain populations, e. g. : Ø Ø Ø n General issues in surveys Disfranchised / poor populations often not reached by internet or telephone Cell phones & avoidance of telemarketers less availability for telephone surveys Stigmatized populations less available for face-to-face interviews, more available via internet. Participant sophistication n n Participants may not be able to accurately report certain topics n Attitudes toward stem cell research from readings. n What factors are most important to your choice of political candidate. . n Describe the amounts and types of proteins you eat during a typical week. . . “Rationality bias”; many questions (incorrectly? ) assume a rational reason for behavior: n Why do you have unsafe sex. . . n What is your chief reason for using alcohol each night…

Foundations of Research n n Social Desirability Responding 44 Clear face-valid items addressing embarrassing topics yield less valid responses n How often are you dishonest with your friends? n Have you ever cheated on an exam. . ? High social desirability wording elicits inaccurate responses… n Do you support protecting our Nation’s forests for future generations ? (Does “yes” mean you an “environmentalist”? ). n Do you feel there are ways your husband could be closer. . . ? (Does “yes” make you are unhappy in your marriage”? ). n Populations differ in social desirability responding; that difference may be a confound in studying group differences § Women report more suicidal thoughts, but may be more willing to disclose, creating a possible confound… n Desirability can be minimized by: n Anonymous surveys n Assurances of confidentiality Computer administration (no personal interaction) n Careful wording / pilot testing of items n

Foundations of Research 45 Social desirability responding Click image for NY Times article n Do people lie on surveys? Men routinely report more sex partners than do women. n If the sample is unbiased by gender, number of partners should balance for men & women. n Social desirability hypothesis: n n Women underestimate partners n Men overstate partners n Much of the difference due to: ü A high proportion of women who report 1 partner ü A few men who report many partners. n Possible sample bias (confound? ) in who responds to such surveys? Click for article from phys. org

Foundations of Research n General issues in surveys: Time Frames Rare(er) events require a long time frame to assess n n When was your last doctor’s visit… These questions asses the last time you left a romantic relationship… ü Longer term recall can be surprisingly unreliable § Recall of last doctor visit is highly unreliable when checked against medical records n Shorter time frame yields more reliable responding ü Memory is better for more recent effects § “Exit interviews” from medical visits are far more reliable than even 2 -week retrospective measures. ü Current, concrete behaviors are more accurately reported than are behavioral trends. § In general, how often do you miss a dose of your medication Less reliable than… § Let’s go over each of the past 7 days and tell me if you took or missed your medication dose. 46

Foundations of Research n General issues in surveys: Question Order Questions trigger participant’s memory or attention, and can bias questions that follow, e. g. : Do you think Social Security & Medicare payments have kept up with inflation. . then Do you favor or oppose Democratic efforts to expand Medicare payments. . . n Bias can be limited by counterbalancing questions. n Using different question orders in different versions of the survey. 47

Foundations of Research n Summary n 48 Survey administration n Internet increasingly important as self-report method n Face-to-face interviews more common in clinical research n Time frames & question order can influence responses Population access & sophistication n Some groups are difficult to reach n Creates threat to External validity n n General Issues Summary Assumption that participants understand survey materials often questionable. Social desirability responding n n Inhibited responding threatens Internal Validity May represent a confound if groups differ in desirability set.

Foundations of Research Survey Research Topic areas & formats General issues in Survey research ✓Sources of bias (or fraud…) Examples of surveys 49

Foundations of Research n n Bias / Fraud in survey research Social research is increasingly important to political & cultural debates. n Effects of gay marriage… n Political “approval” ratings… n Scientific consensus on global warming… n Research on working mothers … Pressure for confirmatory results encourages bias or outright fraud (see this week’s article on Opinion Polls). n n In the study structure n Items used n Sample In the interpretation of results n “Cherry picking” n Simple distortion 50

Foundations of Research n n Example n n n 51 Opposition to gay marriage n n Example of a fraudulent survey: Does having a gay/lesbian parent harm children? In early court cases gay marriage opponents cited religious doctrine as the basis for continued discrimination against GLBTQ marriage. In several cases Judges struck down religious doctrine as a basis for disallowing gay marriage, citing the separation of church and state. Opponents thus sought evidence of that gay marriage causes civil harm to justify continued discrimination. The prospect that gay marriage may cause harm to children emerged as a key issue. Multiple well conducted studies here and a policy statement by the American Psychological Association here report no harm to children whose parent(s) are gay. April De. Boer and Jayne Rowse … after closing arguments in their challenge to Michigan’s marriage restriction. (Click image. ) Mandi Wright/Detroit Free Press, via Associated Press In marked contrast, a 2012 Survey by Mark Regenerus appeared to show that Children in households with a gay/lesbian parent(s) fare much worse than children of heterosexual parents.

Foundations of Research n Example n 52 Example of fraudulent survey use The Regenerus survey was cited extensively in a nationally recognized case in Detroit (as well as other cases, up to today…). Gay marriage opponents lost, in part due to the debunking of the Rogenerus study. Click image for a more sympathetic view of the Rogenerus study April De. Boer and Jayne Rowse … after closing arguments in their challenge to Michigan’s marriage restriction. (Click image. ) Mandi Wright/Detroit Free Press, via Associated Press § Funded & cited widely by gay marriage opponents § Study sample and interpretation of results wildly biased § Has been disavowed by Renenerus’ own Academic Department and American Sociological Association. § Many children classified as being raised in gay-led households did not, in fact, reside with a gay parent for at least a year click.

Foundations of Research n 53 Sources of survey fraud; question wording can elicit a response desired by the researcher; n n Bias in survey research: Leading or biased items How much do you support the administrations’ actions to protect you and your children from terrorists… Wording can “normalize” a response, e. g. , n When do you feel that it is O. K. to cheat on an exam? . . when I really do not know the material. . when others are doing it. . when I think the exam is unfair n Vague wording can be interpreted in a biased fashion Is there anything your husband could do to be more intimate with you? n “Push” polls: a survey can be used to actually create an attitude. http: //www. marketingresearch. org/issues-policies/bestn practice/push-polls-deceptive-advocacypersuasion-under-guiselegitimate-polling

Foundations of Research n n Push Polls: 54 Push polls attempt to induce or change attitudes in the guise of a poll, rather than simply assess attitudes. They change attitudes in two ways: 1. While taking the poll participants are given “information” (typically distorted or even fraudulent ‘facts’) designed to shift opinion, 2. The “results” of the poll are publicized to influence opinion more broadly. Push Poll disinformation example. Ø The 2000 Republican primary in S. Carolina is one of the most extreme examples of using a wildly dishonest (and racist) push poll to poison opinion. Ø As part of a smear program operatives aligned with the G. Bush campaign blanketed the state with ostensible polls regarding John Mc. Cain (click the image for an interesting article on those politics).

Foundations of Research Push Polls: Push Poll disinformation example. Ø Mc. Cain, a former prisoner of war in Vietnam and his wife had adopted a child from Mother Theresa’s orphanage in Bangladesh. Ø The push poll took this information and twisted it into these and similar smear questions: 1. Studies have shown that John Mc. Cain is mentally unstable because of his time spent in prison camps during the Vietnam War. Would this make you more likely or less likely to vote for him? 2. John Mc. Cain has a mixed-race daughter that he had with a black prostitute. Would this make you more likely or less likely to vote for him? Ø There were no “data”: people were simply hired to ask the questions to any voter who reported a preference for Mc. Cain. Ø The Mc. Cain campaign, leading my a wide margin at the beginning of the South Carolina race, never recovered from these attacks, and went on to lose the nomination. 55

Foundations of Research n n Push Polls: 56 Push polls attempt to induce or change attitudes in the guise of a poll, rather than simply assess attitudes. They change attitudes in two ways: 1. While taking the poll participants are given “information” (typically distorted or even fraudulent ‘facts’) designed to shift opinion, 2. The “results” of the poll are publicized to influence opinion more broadly. Using polls to disseminate misleading information. Ø An effective way to change public opinion is to publish “data” about what most people think. Ø Like it or no, our own views are strongly influenced by the majority. Ø Push polls designed to produce one pre-ordained outcome can influence us by giving a gloss of science to highly manipulative statements of “the majority opinion” or “what most people think”.

Foundations of Research Push Polls: Using polls to disseminate misleading information. Ø Items are developed that “trap” the participant into endorsing a specific view. Ø When the “data” are released the biased wording is ignored. Ø Publicity about the “findings” (e. g. , by politically biased news organizations) are used to further create or change attitudes. Ø Many political & social organizations use this strategy to… • Ostensibly measure attitudes objectively • Use the “results” to influence popular opinion. Ø Example of a highly biased survey: The Republican National Committee ‘Future of American Health Care’ survey. • The survey was distributed in several counties as part of a fund raising letter. • It is clearly a “push poll” designed to create fear of health care reform. • It got limited distribution, but is a great example of a Push Poll. 57

Foundations of Research n n “GOP health survey” push poll (2009 – 2010) Some of these items are simple lies – or manipulative statements – designed to induce anti-health care attitudes… Other are powerful (and dishonest) fear manipulations ✓ ✓ ✓ 58

Foundations of Research n n Forms of survey bias: Provide leading or emotionally manipulative information to induce an attitude rather than simply measure it, to provide politically useful “data”. . Questions that, if you accept their assumptions, can only be reasonably answered one way… 59

Foundations of Research n Biased surveys: Democratic example An example from the Democrats, that is also used for fund-raising. 60

Foundations of Research Manipulative presentation of questionable information Simple emotional manipulation Distorted description that may be changed in presentation of findings “Who could disagree” item. Democratic biased survey (2007) 61

Foundations of Research Summary: Manipulating attitudes by surveys 1. Ask manipulative or highly leading questions Summary 2. Find high levels of agreement (and potentially change participants’ attitudes). 3. Publicize – and often distort or overstate – the “findings” via highly biased news sources 4. News reports themselves lead to attitude change among people who are uncertain or uninformed. 62

Foundations of Research Survey Research Topic areas & formats General issues in Survey research Sources of bias (or fraud…) ✓Examples of surveys 63

Foundations of Research 64 Examples of surveys & data, 1 Consumer reports survey of mental health care question population Self-Identified group: sample Self-selected convenience sample: data Attitudes & behavior findings • Satisfaction with therapy. • Differences between types of therapy. U. S. mental health care users Readers who got therapy & returned the survey, n=4000 Self-report questionnaire, cross-sectional High satisfaction for most treatments Descriptive & hypothesis tests

Foundations of Research 65 Examples of surveys & data, 2 “Monitoring the future” youth studies question population Demographic group: sample Random sample: data Knowledge, attitudes and behavior findings Mostly descriptive • Social behavior • Academics • Alcohol & drug use • Health. All U. S. youth, 15 -> 21 years old. Sample of High School health classes, n=3000 -> 5000. Face--to--face interviews & questionnaires, longitudinal (biyearly) Assess yearly trends/shifts in drugs, grades, emotional well being

Foundations of Research 66 Examples of surveys & data, 3 Gallup, Time/CNN, other polls question populations Demographic: - Eligible voters, Political opinions, Lifestyle information Social attitudes, e. g. , managed care - Target age groups Self-identified: “Democrats”… Behavioral: Voters, ACA users… General: - U. S. adult population samples National, random data Knowledge, Attitudes, Behavior findings Descriptive Digit dial telephone, n=150 to >500 Brief interview, cross-sectional Ratings of politicians, Consumer preferences Approach to Affordable Care Act

Foundations of Research 67 Examples of surveys & data, 4 Exit polls question Election outcome, possibly stratified by state / region population U. S. electorate sample Probability data Self-reported behavior findings Descriptive/ predictive National and/or local electoral district Stratified random sample of electoral districts. Self-report interview, cross-sectional Increasingly inaccurate predictions See reading on shifts in use of polling data in U. S. politics

Foundations of Research 68 Examples of surveys & data, 5 ”Social Issues Survey” of Chicago gay / lesbian community question population Self-identified sample Targeted multiframe data Attitudes & behavior findings Descriptive & hypotheses - Stress & coping - Alcohol & drug use - Responses to HIV / AIDS Self-identified gay, lesbian, & bisexual adults in Chicago. community newspapers, organizations, & mailing lists, n=3500 Self-report questionnaire, cross-sectional - High experience of discrimination - Less stress & alcohol-drug use than expected

Foundations of Research 69 Examples of surveys & data, 6 National Institute on Drug Abuse Household survey of Alcohol and Drug use question population National sample Random Multistage: data Knowledge, attitudes & behavior findings Alcohol-drug use and problems, treatment use, health effects. All U. S. adults 1. Census tract 2. Household, 3. Any adult member Face - to - face Interview, successive cross-sectional (each 5 years) Age & regional differences in substance use, trends over time in use & problems n>4000 Typically descriptive Data often used for hypothesis-oriented secondary analyses (i. e. , as archival data).

Foundations of Research n Surveys typically use multiple items to measure each hypothetical construct n Summary n n n Summary: Testing Hypotheses Correlations among items tell us if they are reliable in measuring the same construct. We use Mediating Analyses to n Test hypotheses about correlations between constructs n Build or test theory Cross-sectional analyses are difficult to interpret n Causal direction? n 3 rd variable problem Longitudinal analyses help us determine causal direction 70