Goal Attainment Scaling Measuring and Monitoring Project Objectives
Goal Attainment Scaling: Measuring and Monitoring Project Objectives December 11, 2007 Ella Taylor Tom Udell National Consortium on Deaf-Blindness The Teaching Research Institute Western Oregon University
Objectives for Webinar 1. Understand what GAS is including benefits and uses. 2. Understand the steps for developing a GAS. 3. Understand how to implement a GAS to track progress.
Let face it my friends, We’re not exactly rocket scientists.
What is Goal Attainment Scaling (GAS) • A tool for tracking and recording the achievement of project activities. • Regular use of a GAS across objectives and activities can be used to determine if − (a) timelines are being met, − (b) milestones or bench marks are met on a regular basis, − (c) resources need to be reallocated, − (d) new strategies are needed due to a shift from an intended outcome, or − (e) new targets need to be developed with project officer.
Benefits of GAS • Monitors improvement and achievement. • Can be applied to all project outcomes and activities at any and all stages of implementation. • Enables comparison between very different activities. • Aligns with OSEP’s Grant Performance Reports. • Provides proposal reviews a “snapshot” of targets across all activities.
GAS Scores or Degrees of Achievement • Level +2: Much higher than anticipated outcome. • Level +1: Higher than anticipated outcome • Level 0: Met anticipated outcome. • Level -1: Less than anticipated outcome • Level -2: Much less that than anticipated outcome
Comparison between very different activities. • Difficult to compare outcomes of… a. 85% b. 25 c. December 1 st • Giving a numeric value to each enables comparison… a. 85% becomes 0 b. 25 becomes -2 c. December 1 st becomes +1
Example of GAS (in back of Handout) Degree of Achievement Objectives/Activities Anticipated Outcomes Current -2 -1 0 +1 +2 Objective 4: Implement an array of TA and personnel training activities to build the capacity of state and local agencies to meet the needs of children/youth and their families. Meet 90% of identified anticipated outcomes for the objective’s activities. 80% 85% 90% 95% 100% 4. 1 – Implement state-specific TA and training activities that promote the implementation of IDEA and the use of evidence-based practices. 80% of states participate in state specific TA 60% 70% 80% 90% 100% 4. 2 – Implement multi-state TA and training activities that promote the implementation of IDEA and the use of evidence-based practices. 90% of states participate in 1 or more Area activities ( Meetings, Webinars, LISTSERVs) 80% 85% 90% 95% 100% 4. 3 – Implement national TA and training activities that promote the implementation of IDEA and the use of evidence-based practices. 95% of states participate in two or more national activities (PDM, Topical Workshops, Webinars, FST) 90% 95% 100%
Steps in Developing a GAS • Step 1: Develop objectives for the project. • Step 2: Develop activities needed to met or complete each objective. • Step 3: Set an anticipated outcome for each activity. • Step 4: For each outcome, set targets for each degree of achievement.
Step 1: Develop objectives for the project. • Driven by the RFP priorities the project must address. • Statements describing each aspect of what a project will do. • Lead to achieving the project’s overall outcome or goal. • Outline accomplishments that are critical for the project to be considered successful. • Should be stated in terms of observable performance.
Do I develop degrees of achievement for objectives? • Average degree of achievement for anticipated outcomes. • Percentage of anticipated outcomes met.
Step 2: Develop activities needed to meet or complete each objective. • Clear and specific language. • Discrete steps. • Logically lead to accomplishment of the objective.
Step 3: Set an anticipated outcome for each activity. • Set a standard of quality and/or quantity. • Stated in observable and measurable terms. • Acceptable evidence that the activity was completed.
How do I determine criteria? • Use previous project data. • Consider funding level. • Consider available resources. • Use project staff experiences. • Use advisory committee input.
Do I set up the GAS for a quarterly, annual or 5 year review? • Dependents on what decisions you need to make and what data you need to make decisions. • Criteria for anticipated outcomes should be based on review period.
Step 4: For each outcome, set targets for each degree of achievement. • For each identified outcome targets are set for the degree of achievement scale. • Level 0: Met anticipated outcome is simply the criteria set within the outcome. • Levels -1 and -2 are lower degrees of success or achievement using the same measurement or criteria type. Conversely, • Levels +1 and +2 are higher degrees of achievement that go beyond what is satisfactory for meeting the outcome and completing the respective activity.
Implementation • Summarize and record data. • Identify degree of achievement. • Reflection. • Planning.
Incorporating outcomes into project activities… Activity Needs Assessment Beginning: (Objectives) Identifying what needs to be done and selecting OPIs related to the needs. (Grant activities) PROJECT GOALS Evaluation (Measures & Outcomes) End: Documenting and evaluating your efforts to determine your effects. Middle: (1) Planning and delivering TA to meet the needs. (2) Planning on how you will evaluate effort.
Reporting Example Effort Language • Number of TA providers who received training about goal attainment scaling (n = 50) • Number of workshops conducted focusing on strategies to improve the use of goal attainment scales (n = 4)
APR Example Effect Language • Mean rating of TA providers reporting increased knowledge about goal attainment scales (m = 4. 0 (5. 0 scale) [APR 40/50] • Mean rating of TA providers reporting progress in implementing strategies to use goal attainment scaling (m = 3 (5. 0 scale))[APR 30/50)]
APR (ED 524 B report) language Project Objective: Assess the impact of TA & D activities Performance Measure Target Actual 5 4 Type Provide 5 workshops on Project goal attainment scaling Explanation of progress: While four trainings were provided that detailed specific strategies for increasing the receptive and expressive communication of students who are deaf-blind, we missed our target by one training due to a cancellation caused by snow.
APR (ED 524 B report) language Project Objective: Assess the impact of TA & D activities Performance Measure Target Actual Type Impact of TA on service Project 40/50 46/50 providers measured as an increase in knowledge Explanation: All data based on evaluation conducted after delivery of technical assistance. Analysis is based on calculation of means and percentages using a five-item Likert scale ranging from either Strongly Disagree(1) to Strongly Agree (5) for change of knowledge or No Progress (1) to Substantial Progress (5) for change of skill. For purposes of this report, means were changed to whole numbers by adding 10 (i. e. , 4. 0/5. 0 equals 40/50).
APR (ED 524 B report) language Project Objective: Assess the impact of TA & D activities Performance Measure Target Actual Type Impact of TA on TA providers measured as an increase in skill/implementation Project 30/50 29/50 Explanation: All data based on evaluation conducted after delivery of technical assistance. Analysis is based on calculation of means and percentages using a five-item Likert scale ranging from either Strongly Disagree(1) to Strongly Agree (5) for change of knowledge or No Progress (1) to Substantial Progress (5) for change of skill. For purposes of this report, means were changed to whole numbers by adding 10 (i. e. , 4. 0/5. 0 equals 40/50).
Example of GAS (in back of Handout)
- Slides: 26