323 A State Personnel Development Grants SPDG Webinar

323 A State Personnel Development Grants SPDG Webinar on Grant Performance Report for Continuation Funding Jennifer Coffey Office of Special Education Programs US Department of Education Washington, DC January 2015 1

Today’s Agenda Today’s Topic – Grant Performance Report for Continuation Funding q q Overview of Performance Reporting Developing Performance Measures Completing Section A of the ED 524 B Completing Sections B&C of the ED 524 B Program Measure 1 – Revised Guidance Documents Evidence-based Professional Development (EB-PD) Rubric EB-PD Worksheet 2

Annual Grant Performance Report (APR) O S E P 3 Ü An annual report of your activities and performance in meeting the approved objectives of the project and responsible use of federal funds Ü Required for all active grants, including those in no cost extension (NCE) Ü OSEP reviews the report to determine if substantial progress has been made in order to receive continued funding or a NCE

Requesting a no-cost extension (NCE) O S E P 4 On the Grants Management page (http: //www. signetwork. org/content_pages/139): Requesting No-cost extensions At the time the no-cost extension is requested, about 30 days before the end of the grant (must be before 10 days prior to the last day of the performance period), the Project Officer will need an updated continuation report emailed to them. The Project Officer will also need to know: (1) the amount the grantee has remaining in their budget, (2) the activities the grantee wants to continue to conduct that align with approved objectives, (3) how much of the budget will be used for each activity, and (4) why the grantee was not able to spend the entire budget within 5 years. NCE Reporting Form, This is the form OSEP provides to states to capture information about the no cost extension (NCE).

Overview O S E P Ü Recognize strong project objectives that can be associated with high quality performance measures Ü Develop relevant, measurable, outcome oriented performance measures related to your objectives that maximize the potential for meaningful data reporting and positive outcomes Ü Complete the ED Grant Performance Report (aka. APR) using form ED 524 B. 5

Why Is This Important? O S E P 6 High quality objectives and measures … Ü Make it easier for you to measure your progress for the purpose of grant management Ü Allow you to report progress easily and quantitatively Ü Establish targets (both short-term/annual & longterm) Ü Allow OSEP staff to gather evidence of program effectiveness Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www. tadnet. org

Goal – Objectives - Measures O S E P 7 Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www. tadnet. org

Project Objectives O S E P 8 What are you trying to accomplish? Objectives should answer this question. Preferred format for objectives: Begin the objective with a verb and define a desired outcome or condition

High Quality Project Objectives O S E P 9 Ü Relevance How relevant is the project objective to the overall goal of the program and/or the goal of your project? Ü Applicability How applicable is the project objective to the specific activities that are being conducted through your particular project? Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www. tadnet. org

High Quality Project Objectives O S E P 10 Ü Focus How focused is the project objective? Ü Measurability Are there concepts in the project objective that lend themselves to measurement? If so, is measurement feasible? Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www. tadnet. org

Project Objectives -- Examples O S E P • Increase the meaningful participation of families in the Rt. I/PBIS processes in participating LEAs. • Improve the academic and behavioral outcomes of students with autism in participating schools.

Performance Measures O S E P 12 How are you measuring your progress in meeting your objectives? Performance measures should answer this question.

Performance Measures O S E P 13 Ü Measurable indicator used to determine how well objectives are being met. Ü How will progress be assessed? Ü How much progress will constitute success? Ü How will it be known if an objective or part of an objective has been achieved? Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www. tadnet. org

Performance Measures O S E P 14 Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www. tadnet. org

2 Types of Performance Measures O S E P 15 Ü Program All grantees funded under the SPDG must report on the PDP program performance measures established by OSEP. Ü Project Each grantee reports on the approved project performance measures established to meet their project objectives.

Program Performance Measures O S E P Ü Program Measures established by OSEP for the SPDGs. Measures apply to all grants funded under the SPDG. Results on these measures are reported to Congress under the Government Performance and Results Act of 1993. Program Measures Webinar: http: //www. signetwork. org/event_calendar/events/1257 16 Please see the Program Measures Web page for more information and recorded Webinars: http: //www. signetwork. org/content_pages/205

Types of Performance Measures O S E P Ü Project Measures that the grantee establishes to meet their project objectives § Project performance measures can address both the process of working towards an objective and the outcome related to meeting the objective § Ensure a mix of both process and outcome measures, but most will be outcome Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www. tadnet. org 17

High Quality Performance Measures O S E P 18 High quality performance measures show ÜWhat will change ÜHow much change you expect ÜWho will achieve the change ÜWhen the change will take place Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www. tadnet. org

Project Performance Measure Examples O S E P 19 Process measure (e. g. ) SPDG staff (who) will hold 4 (how much) trainings with IHE faculty on how to integrate the transition curriculum into their syllabi (what ) during the first and third years of the grant (when).

Project Performance Measure Examples O S E P 20 Outcome measure (e. g. ) By the end of the third year of the grant (when), 80% of SPDG professional development participants (who) will demonstrate 100% reliability (how much) when using the selfassessment rubric established to evaluate implementation of the transition program (what).

Project Performance Measure Examples O S E P 21 Outcome measure (e. g. ) At the end of their third year of training (when), 90% (how much) of partner schools (who) will demonstrate a 15% improvement in the math scores of 4 th grade students (what).

Common Problems Ü Activities are NOT performance measures O S E P Ü If the best response is “Yes, we did that, ” it is likely an activity (not a performance measure) Ü Activities: Ü Establish a stakeholder group Ü Hold an advisory board meeting Ü Evaluate the project 22 Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www. tadnet. org

Common Problems O S E P 23 Ü Performance measures need to be measurable Ü Examples with measurement problems (activities rather than outcomes) Ü Will maintain collaborative partnerships with parent organizations Ü Increase the sustainability of the personnel development program Taken from the Center for Evaluation and Education Policy (CEEP) presentation at www. tadnet. org

Need additional information on writing performance measures? O S E P 24 Ü All grantees are strongly encouraged to seek training on writing performance measures. Ü For further information on developing performance measures and logic models, see http: //www. tadnet. org/pages/589

Summary O S E P 25 Ü Projects should have a few clear objectives that explain what the project is doing to support the overall goal(s) Ü Each objective should have a few, specific performance measures to demonstrate how progress toward meeting the objective will be measured Ü Both program and project performance measures are included in the ED 524 B

Completing the ED 524 B O S E P 26 The ED 524 B is a required reporting form with specific instructions. Ü The form is used by all ED grants and has been approved by the Office of Management and Budget (OMB). Project Directors must follow the directions listed in the Dear Colleague letter and ED 524 B Instructions provided by OSEP. Ü Word or PDF versions of the forms are available at http: //www 2. ed. gov/fund/grant/apply/appforms. html

X x O S E P 27

ANNUAL PERFORMANCE REPORTS Reporting Period: For first year grants, the date is the beginning of the project year to -----. O S E P For grants in years 2 -4, it is the date from the end of the previous reporting period to ------. 28 2015

ANNUAL PERFORMANCE REPORTS Budget Expenditures: O S E P 29 Report the expenditures during the “Reporting Period. ” Must be data or information from the business or grants office.

ANNUAL PERFORMANCE REPORTS Performance Measure Status: This will be checked “No” since OSEP is asking for data for the reporting period covering all years of the grant, not for this budget period. O S E P The date entered here will be the due date for your Final Performance Report; which is 90 days after the end of the grant. Signatory must have authority to sign on behalf of the institution since the grant is from the Department to the institution and not to an individual. The Authorized Representative signs; not the Project Director. 30

OMB No. 1894 -0003 Exp. 02/28/2011 EXECUTIVE SUMMARY SHEET O S E P H 323 A - - - *** Provide highlights of the project's activities and the extent to which the expected outcomes and performance measures were achieved during the reporting period. Do NOT include the project abstract. 31

PROJECT STATUS CHART H 323 A - - - O S E P 32

PROJECT STATUS CHART H 325 T - - - O S E P 33 You will begin with Program Measure 1 as your 1 st project objective. Program Measure 2 will be your 2 nd objective, and so on. After these program measures, you will then have your project’s objectives.

PROJECT STATUS CHART Here you identify if the performance measure is a PROGRAM measure, “PRGM, ” or a PROJECT measure, “PROJ. ” O S E P PRGM PROJ Note: Program measure refers to one of OSEP’s 4 program/GPRA measures for the SPDGs. Project measures are unique to your grant. 34

PROJECT STATUS CHART QUANTITATIVE DATA Depending on your measure, enter either a raw number, or a ratio and percentage. O S E P Enter the target number identified in the performance measure and then the actual data for this year. If complete data are not available for the measure, enter “ 999” (if no baseline) or “NA” in the “Raw Number” or “%” column, as appropriate. Provide an explanation at the bottom of the page under “Explanation of Progress. ” 35

Information to Include in the Explanation of Progress Section O S E P 36 Ü Describe the data provided (e. g. , what data collection methods were used, when were the data collected, how was a sample drawn, are there missing/incomplete data, what was the response rate, was a reliability measure taken). Your Project Officer should be able to understand interpret the number in the chart from your description in this section. Ü What changes in the data occurred since last APR (i. e. , trend)? Ü What activities were undertaken to achieve the targets? Ü If targets were not met, what are possible reasons? Ü How will activities that failed to meet targets be improved?

Additionally… O S E P 37 A “template” is provided for the program measure descriptions in the Program Measure Example Continuation Report (http: //www. signetwork. org/content_pages/20 5)

PROJECT STATUS CHART QUALITATIVE DATA If measure requires the collection of qualitative data, report the performance measure and type (PROG or PROJ) and then, enter “N/A” under the Raw Number and Percentage columns. O S E P N/A N/A In the “Explanation of Progress” section of the page, referencing the performance measure by number, report applicable qualitative data along with other information about how these data were collected, targets and activities –refer to previous slide for additional content requirements. 38 N/A

Final Page of the Report H 325 - - - O S E P 39 Section B: Refer to the instructions for Section B in the ED 524 B Instructions Section C: Include additional information (recruitment material, syllabi, evaluation instruments, journal articles)

Section B – Budget Information O S E P 40 This section is never blank! A table can be helpful! A. Provide actual expenditures for this reporting period (through B. Estimate anticipated expenditures for the rest of this budget period and balance remaining, if any. 2/29/2012) C. Explain why you did not expend funds at the expected rate. D. Indicate how you plan to use the unexpended funds (carryover) in the next budget period.

Section B – Budget Information O S E P E. Describe any significant changes to your budget resulting from modifications of project activities. F. Describe any changes to your budget that affect your ability to achieve your approved project activities and/or project objectives. G. Describe any anticipated changes in your budget for the next budget period that require prior approval from the Department. Any questions … Talk to your Project Officer 41

Section C – Additional Information O S E P 42 Ü Provide a list of current partners on your grant and indicate if: Ü Any partners changed during the reporting period. If there were changes, please describe both the changes and any impact that resulted in your ability to achieve approved project objectives and/or project activities. Ü Any partners are anticipated to change during the next budget period. If so, please describe both the changes and any impact the change might have on your ability to achieve approved project objectives and/or project activities. Ü Describe any changes that you wish to make in the grant’s activities for the next budget period that are consistent with the scope and objectives of your approved application.

Section C – Additional Information O S E P 43 Ü If requesting changes to the approved Project Director and/or other key personnel, please include the person’s name, title, and contact information. Indicate his/her proposed start date, and percentage of time working on the grant, and attach a resume or curriculum vitae to the annual performance report being submitted. Ü Do not report on any key personnel changes that were already made during the current or previous budget period(s). Ü Note: Departmental approval must be requested and received prior to making key personnel changes. Ü Provide any other information about your project including unanticipated outcomes or benefits

Submitting the ED 524 B O S E P 44 Ü Submit the ED 524 B at http: //www. g 5. gov/ Ü Instructions for using G 5 are in the continuation packet. Ü Signed ED 524 B Cover Sheet must be scanned and emailed in PDF format to your Project Officer. Ü Special cases require regular email submission of the 524 B and signed cover sheet in PDF format to your Project Officer rather than submission through G 5 – ÜFinal Performance Reports or APRs for grants in their last performance period, or no-cost extension ÜGrants that have been front-loaded (forward-funded) last year sometimes cannot be uploaded.

Exemplar APR O S E P 45

46 O S E P THE REVISED PROGRAM MEASURE 1 FORM AND GUIDANCE Program Measures Web page: http: //www. signetwork. org/content_page s/205

§ SPDG Professional Development Rubric 5 Domains, each with components § Selection § Training § Coaching § Performance Assessment/Data-based decision making § Facilitative administration/Systems intervention § Components from the National Implementation Research Network, Learning Forward (NSDC), Guskey, Trivette § Each component of the domains will be rated from 1 - 4 47

48 Component Themes § Assigning responsibility for major professional development functions (e. g. , measuring fidelity and outcomes; monitoring coaching quality) § Expectations stated for all roles and responsibilities (e. g. , PD participants, trainers, coaches, school & district administrators) § Data for each stage of PD (e. g. , selection, training, implementation, coaching, outcomes)

SPDG Initiatives and Evidence-based Professional Development § Each initiative you are reporting on should have its own worksheet. § You will only receive credit for work you have already done. § Just lay out the program measure target and data for the current year (do not provide multiple years’ targets for the same initiative). 49

50 Grantee Benchmarks § § 1 st year of funding: baseline § 3 rd yr: 70% of components will have a score of 3 or 4 § 4 th yr: 80% of components will have a score of 3 or 4 § 5 th yr: 80% of components will have a score of 3 or 4 (maintenance yr) 2 nd yr: 50% of components will have a score of 3 or 4

Revisions to the Worksheet and Rubric § § Expectations more clearly defined. § All of SIGnetwork was given an opportunity to comment on the Worksheet changes. Should assist both reporting and evaluation of your reporting. § Please let John or Jennifer know if you have a recommended change to the rubric. 51

Why components received a score of 1 or 2 § § Activities had not been completed. Activities described were in the planning stage (the description included information about activities that will take place in the future). 52

53 The Description was Incomplete § The description was vague and did not have enough detail to determine what activities had been completed. § Required element descriptions were superficial and lacked sufficient detail. § The description did not include information on all of the professional development component elements. § The description merely repeated the expectations in the rubric for the professional development component.

The description was not specific to the professional development component. § The description was detailed, but not appropriate for the professional development component (e. g. , the state provided a detailed description that did not match the component and did not include required elements). § The description was unrelated to the component and required elements. § The grantee used the same narrative to describe activities related to multiple professional development components. 54

Please contact your OSEP Project Officer with any questions about your APR. 55