Building Coalition and Evaluator Evaluation Capacity to Measure
Building Coalition and Evaluator Evaluation Capacity to Measure Community and Systems Change Local Evaluation Consulting: What Does it Take to Change Evaluator and Community Collaborative Evaluation Practice ? American Evaluation Association Conference November 4, 2011 Ann Webb Price, Ph. D. aprice@communityevaluationsolutions. com.
DRUG FREE COALITION OF HALL COUNTY
Georgia Student Health Survey II Conducted each October, this survey is designed to collect information about a variety of topics including alcohol, tobacco, and other drugs; school violence; school climate; and nutrition. Data analysis will concentrate in the Four Core Measures as required by the DFC grant. These include: • Age of onset of drug use; • Frequency of use in the past 30 days; • Perception of risk of harm; and • Perception of disapproval of use by adults/parent.
Assessment of Community Substance Abuse Create Pjct Guidelines Collect Data Set up Template Complete Report Create WBS Define Indicators Review Other Reports Evaluate Data Create Project Plan Select Sources Create Chart Format Write Assessment Create Schedule Assign Sources Input Data Create Comm Mgt Plan Create Request Format Create Risk Mgt Plan Create Follow-Up Format Request data Review data Follow-up as Necessary Review for Completeness
Problem Statement: There is an underage drinking problem in Hall County. Measure 1: According to the 2007 Georgia Student Health Survey (GSHS II), 30% of 12 th grade students have used alcohol in the last 30 days Measure 2: According to local Youth Focus Groups, alcohol is the most widely used substance But why? But why here? Theory of Change: By a comprehensive community effort involving multiple strategies, our community will significantly reduce youth substance use. CSAP Strategies/Activities Outcomes Short Term Alcohol is easily accessible. Measure 1: According to the 2007 GSHS II: 44% of 6 th 51% of 8 th 69% 10 th 79% of 12 th Somewhat or strongly agreed that alcohol was easy to get Measure 2: According to the 2007 GSHS II: Age of onset was 12. 9 As teens mature, they perceive that parents see alcohol as a rite of passage. Measure 1: According to the 2007 GSHS II: 6 th Grade: 76% Strongly Agree and 8% Somewhat Agree that their parents would disapprove if they used alcohol. 8 th grade: 70% Strongly Agree; 17% Somewhat Agree 10 th grade: 64% Strongly Agree; 20% Somewhat Agree 12 th grade: 56% Strongly Agree; 22% Somewhat Agree Measure 2: 2010 Youth and Parent Focus Groups Youth use alcohol at home. Access/Barriers: Increase the number of adults who agree not to provide alcohol and commit to “Face It”, sign online commitment form and access personalized “Face It form”, Posted “Face It” signs in businesses. Number of families who pledge to “Face it” and not provide youth alcohol Measure 1: Number of families who sign online commitment form Measure 2: Parent Focus Groups Change Consequences: Encourage consistent enforcement of adult provider laws Increased collaboration with law enforcement Measure 1: Law enforcement attendance at coalition meetings Measure 2: Key Informant Interviews Change Physical Design: Distribute/post signs in retail outlets about the laws about adults buying for youth and Hall data about the % of youth who get alcohol in their homes/friends homes Change/Modify Policies: Strengthen existing adult hosting ordinances Measure 1: According to the 2007 GSHS II: 5% of 6 th, 20% of 8 th, 35% of 10 th, and 46% of 12 th graders use at home Measure 2: 2010 Youth Focus Groups Youth use alcohol at the home of friends. Measure 1: According to the 2007 GSHS II: 4% of 6 th, 22% of 8 th, 48% of 10 th, and 60% of 12 th graders use at home of their friends Measure 2: 2010 Youth Focus Groups Number of signs distributed; stores posting signs Measure 1: Evaluation Process Data Measure 2: Parent surveys at community events (Relay for Life, etc. ), Determine current social hosting laws and level of enforcement Measure: Archival Records Intermediate Increased number of families making Face It commitments Measure 1: Number of families who sign online commitment form Measure 2: Parent Focus Groups Increased number of arrests for adult provider violations Measure 1: Gainesville Police arrest data Measure 2: Sheriff’s data Increased number of adults who say they have seen a sign; adults who say they have heard about the media campaign Measure 1: Evaluation Process Data Measure 2: Parent surveys at community events (Relay for Life, etc. ), Assess change in social hosting laws and levels of enforcement Measure: Archival data Long Term National Core Measures: Decrease in 30 day use Increase in age of first use Increase in students’ perception of harm Increase in students’ perception of parental disapproval Measure 1: GSHS II Measure 2: Youth Focus Groups Georgia Priority Outcomes: Decrease teen alcohol-related arrests Decrease in teen alcohol-related car accidents Measure 1: Sherriff’s Office Measure 2: Hospital Data
Year 1 Accomplishments ØParticipated in SAMHSA required training ØDeveloped a coalition logic model and alcohol LM ØDeveloped a written evaluation plan ØEstablished parent messaging workgroup ØProduced 8 billboards bearing 6 different parent prevention messages ØEstablished a website and developed media relations
Year 1 Accomplishments Established an active steering committee Ø Met consistently, with good membership representation Ø Recruited members from all 12 required sectors Ø Established relationships with important local and state-level prevention organizations Ø Networked with other DFC across Georgia Ø Partnered with the local county commission to strengthen existing alcohol ordinances Ø
Year 2 Accomplishments Completed a strategic planning process Ø Developed and approved coalition by-laws Ø Established standing committees Ø Established relationships with important local and state-level prevention organizations Ø Attended additional CADCA’s training and Leadership Academy Ø Helped to coordinate two community forums about prescription drug use Ø Jointly sponsored the Hidden in Plain Site room Ø
Year 3 Accomplishments Selected as partner with Medical Association of Georgia on a Prescription drug prevention effort Ø Active steering committee and work groups Ø Coalition Coordinator selected for select leadership training Ø Officers completed first full year Ø Dashboard developed Ø
Moving in to Year 4 and 5 Ø Connecting prevention efforts to our outcomes Ø Getting coalition to adhere to a plan, LM, something, anything! Ø Buy-in for prescription drug effort Ø Breaking up is hard to do Ø New leadership/new members
2011 AEA Conference Theme From AEA President's letter…. “Challenges of values and valuing in evaluation, particularly the plurality of values represented by different evaluation purposes and audiences, key evaluation questions, and quality criteria. ” Ø Which stakeholder interests should be given priority in an evaluation study, and how is this determination made? Ø
Evaluation Best Practices Evaluation Guidelines defined by the Joint Committee on Evaluation Standards in Education: Ø Ø Usability - ensure that an evaluation will serve the information needs of intended users Feasibility- ensure that an evaluation will be realistic, prudent, diplomatic and frugal Propriety - ensure that an evaluation will be conducted legally, ethically and with due regard for the welfare of those involved in the evaluation, as well as those affected by its results Accuracy- ensure that an evaluation will reveal and convey technically adequate information about the features that determine worth or merit of the program being evaluated
Challenges faced by local Evaluator We drank the Cool-aid but how do we bring it home? Diffusion of training…… Ø Balancing what the client wants vs. what the funder wants and what partners want Ø Its about the budget Ø Let’s assume we have good data. What is our method of analysis for the moving target? Ø
Challenges faced by local Evaluator Four core measures and lots of other data- so what? Ø What about measuring the coalition leadership, development etc. ? The budget again Ø Coalitions tend to follow the shiny object Ø Needs (assessment) 1 st Ø
Challenges faced by local Evaluator Ø Learning to speak their language Ø Is that new idea, opportunity, prevention effort on the map (LM)? Ø Evaluation on a moving escalator other wise known as developmental evaluation Ø Personalities again
From your perspective, what is the key challenge or barrier to collecting and using systems change data? � From a data perspective: identifying and collecting quality data on perpetually moving targets � From a people perspective: resistance from coalition members � From a field perspective: lack of simple methodology
From your perspective, how can systems change data help local leaders and partnerships improve their work? � It can help them focus their efforts on the real problems in their community rather than the substance du jour � Help draw the lines between what they do and their target outcomes
- Slides: 20