David Geddes Ph D Senior Vice President Partner

David Geddes, Ph. D. Senior Vice President & Partner Fleishman-Hillard Institute for Public Relations Measurement Commission 21 tips for conducting evaluation research on a boot lace

Today’s objectives § Advice on doing it yourself § Advice on being a good client 2

Where to get help § University departments § Social work, Sociology, Psychology, Political Science, Business, and others § Contacts via board of directors § American Marketing Association § Market Research Association § Market research firms § Linked. In (www. linkedin. com) § National Opinion Research Center (University of Chicago) http: //www. whatisasurvey. info/ 3

http: //www. italladdsup. gov/community_partners/dc_teach mehow_02. asp § Introduction to evaluation research § Process evaluation examples § Telephone survey with annotations § Intercept and selfadministered survey 4

Build a community § Vision § Lutheran Social Services Research Community § Share best practices § Methods § Facebook or other collaboration tool 5

1 Set your objectives § Program objectives § Specific information needs 6

2 Do your background research 7

3 Think about different types of evaluation and measurement 1. Formative evaluation § Helps planning 2. Process evaluation § How well is the program operating? 3. Impact evaluation § Building awareness and understanding? 4. Outcome evaluation § Concrete program or “business” metrics 8

4 Don’t assume you know audience attitudes and opinions § You need formative research 9

5 Set up your process metrics 10

6 Measure impacts § Awareness § Knowledge § Attitudes and perceptions § Intended actions 11

7 Identify survey targets Known respondents Population = Sample frame § Members § Donors § Clients § Employees § Civic organization leaders Defined population Sample frame derived from population § General public § Congregation § members § Potential donors § Potential adoptive parents 12

8 Pick the right survey method to reach targets § Telephone § Mail § Online § Mall or store intercept 13

Telephone surveys Advantages § Speed, in some cases § Control § Probe and clarify § Cost if you do it yourself Disadvantages § Cost if you go outside § Hard to reach some socio-economic groups 14

Telephone surveys § Case: General public, 10 minute survey of 500 adults in Saint Louis metro area. § $8, 000 = $16 cost per interview § Review cost parameters § Case: Survey of 200 adult children or guardians of seniors who receive services from LSS in St. Louis metro area. LSS provides names and phone list. § $4, 000= $20 cost per interview § Review cost parameters 15

Telephone surveys § Case: Survey of 200 married adults in the St. Louis metro area aged 18 to 45 who are at least somewhat open to the idea of adoption. 10 minute survey. 30% incidence based on other surveys. § $7, 000 = $35 cost per interview 16

Doing your own telephone surveys § Sample list § Staff § Interns, volunteers, students, etc. § Call management system § Data recording system § Discipline § Persistence 17

Going outside for telephone surveys § Smaller local firms § Be flexible on timing 18

Mail surveys Advantages § Lower cost § Easy if you have a mailing list § Easier to do yourself Disadvantages § Slow § Lower control § Lower response rates § Need a mailing list 19

Doing your own mail surveys § Clear questionnaire layout § Sample list § Staff § Interns, volunteers, students, etc. § Lots of manual paper handling § Discipline § Patience 20

Online surveys Advantages § Speed, in some cases § Control § Cost if you do it yourself (Survey. Monkey) Disadvantages § Purchase a sample list § Unrepresentative of some socioeconomic groups 21

Online surveys § Case: General public, 10 minute survey of 500 adults in Saint Louis metro area using an online survey tool. § $3, 300 = $6. 60 cost per interview § Review cost parameters 22

Doing your own Online surveys § Survey. Monkey programmer § Sample list § Staff § Interns, volunteers, students, etc. 23

Mall intercept surveys Advantages § Control § Probe and clarify § Cost if you do it yourself Disadvantages § Slow § Need mall or store cooperation § Unrepresentative of some socioeconomic groups 24

Doing your own mall intercept surveys § Cooperation of mall or Wal-Mart § Staff § Interns, volunteers, students, etc. § Discipline § Patience. . Figure on one interview/hour 25

8 a Select an appropriate sample size § What are possible sources of error? § How far off could we be? § What sample size do we need? 26

9 Give advance notice If you know the target respondents § Newsletter § E-mail § Postcard Benefit: improves response rate 27

10 Remember you are asking for their time This applies to… § Pre-survey letters or cards § Introduction to the survey § And the questions themselves § Don’t ask meaningless questions 28

11 Write a good introduction to the questionnaire § Introduce yourself § “Research” not “survey” § Survey topic: social services § Not a sales call … not a fundraising call § You want their opinion / input 29

12 Move from general to specific 30

13 Lead with an interesting question § Engage the respondent § Ask interesting questions and people will talk 31

14 Make judicious use of open-ended questions § Example: § “Based on what you have read or heard, what are your personal perceptions of Lutheran Social Services? [PROBE] Anything else? ” § 2 to 3 per questionnaire maximum § Finish with a “final thoughts” question 32

15 Use good response scales Building a scale § Clearly distinguish good and bad § 1 Agree § 2 Disagree 33

Use good response scales Building a scale § Add a polite positive and negative § 1 Strongly Agree § 2 Agree § 3 Disagree § 4 Strongly disagree 34

Use good response scales Building a scale § Add a top positive response § 1 Very strongly agree § 2 Strongly Agree § 3 Agree § 4 Disagree § 5 Strongly disagree 35

Use good response scales Building a scale § Add a don’t know/no opinion § 1 Very strongly agree § 2 Strongly Agree § 3 Agree § 4 Disagree § 5 Strongly disagree § 6 Don’t know / no opinion 36

Use good response scales Building a scale § Add a mid-point if you really need it. . . and balance the scale § 1 Very strongly agree § 2 Strongly Agree § 3 Agree § 4 Neither agree nor disagree § 5 Disagree § 6 Strongly disagree § 7 Very strongly disagree § 8 Don’t know / no opinion 37

16 Make sure you cover all possible responses § Multiple choice questions 38

17 “No opinion” does not equal “Don’t know” 39

18 Always include a “refused” option 40

19 Be careful with numeric responses § “About how much do you spend on gas each month” $_____ § Which of the following best describes the amount you spend on gas each month: § Under $25 § $26 to $50 § $51 to $100 § Over $100 41

20 Be careful about “do you floss your teeth” items § Socially desirable responses § Responses that may imply guilt 42

21 Test your questionnaire 43

Wrap-up • Begin with your objectives and information needs • Define the people you want to survey • Select an appropriate survey method • Write good questions • Test • Launch the survey • Analyze the results 44

Exercise 45