Getting Started With Your ATE Evaluation ATE PI

Getting Started With Your ATE Evaluation ATE PI Conference October 24, 2012 This material is based upon work supported by the National Science Foundation under Grant Nos. 0802245 and 1204683. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NSF.

’s Mission To promote the goals of the ATE program by - partnering with ATE projects and centers to strengthen the program's evaluation knowledge base - expand the use of exemplary evaluation practices - support the continuous improvement of technician education throughout the nation

Session Objectives - Understand the role evaluation should play in your ATE project - Have clear expectations for your evaluator - Know where to turn if you need more guidance about your ATE evaluation

Evaluation Purposes Formative Evaluation Learn how the project could be improved as it is implemented Summative Evaluation Provide evidence of the quality and impact of your achievements Accountability Document what you did with the grant money

Formative Evaluation The most important purpose of evaluation is not to prove, but to improve. ―Daniel Stufflebeam

Formative Evaluation Project staff… - make time for evaluation - are open to hearing negative findings - use feedback to inform decision making

Formative Evaluation Evaluator… - learns about the project, its context, and stakeholders - gathers data in sound, efficient ways - provides timely feedback - guides project staff in understanding and using results

Summative Evaluation What is the project’s quality? What are the project’s impacts?

Summative Evaluation What is the project’s quality? What are the project’s impacts? - changes in knowledge, skills, abilities, attitudes, performance, practices, or policies - show up at the individual, program, institution, organization, regional, or national level - occur in the short term, long term, or in between

Accountability Document… - What you did Who you reached Who you worked with What you produced (outputs)

Accountability Create a project vita Mission Goals Funding Activities Products Staff Students Contributors & collaborators - Consultants - www. evalu-ate. org/about_us

Evaluation Purposes & Uses Formative Evaluation How is the project doing and how can it improve? Summative Evaluation What is the quality and impact of the project? Accountability What was done, who was served, who contributed, what are the outputs?

Evaluation Purposes & Uses Formative Evaluation How is the project doing and how can it improve? - Feedback to internal stakeholders - Reflection and redirection Summative Evaluation What is the quality and impact of the project? Accountability What was done, who was served, who contributed, what are the outputs?

Evaluation Purposes & Uses Formative Evaluation How is the project doing and how can it improve? - Feedback to internal stakeholders - Reflection and redirection Summative Evaluation What is the quality and impact of the project? - Accountability What was done, who was served, who contributed, what are the outputs? Summative report to key stakeholders Fast. Lane/annual report to NSF ATE Impacts book Annual ATE survey New proposals Promotion and dissemination

Annual Reports to NSF Report Sections - Participants - Activities & findings - Publications & products - Contributions www. fastlane. nsf. gov

Annual Reports to NSF See also: Evalu. ATE’s summary of Fast. Lane report components evalu-ate. org/resources Keyword: Fast. Lane

Annual ATE Survey For example, # of students and their demographic characteristics # of professional development participants # of materials developed www. evalu-ate. org/annual_survey

ATE Projects/Centers Impact Book & Website Student impact data Workforce impact data www. atecenters. org www. ateprojectimpact. org

Results of Prior NSF Support “ “ The Project Description must begin with the subsection on Results of Prior Support…. This subsection must contain specific outcomes and results including metrics to demonstrate the impact of the activities undertaken including evidence of the quality and effectiveness of the project's deliverables. —ATE Program Solicitation

Be an Informed Consumer Joint Committee on Standards for Educational Evaluation The Program Evaluation Standards www. jcsee. org American Evaluation Association Guiding Principles for Evaluators www. eval. org Canadian Evaluation Society Competencies for Evaluation Practice www. evaluationcanada. ca

What should we expect to receive from our evaluator? Participant in the 2010 ATE PI Conference Getting Started Workshop ―

Evaluation Deliverables - Detailed evaluation plan - Opportunities to review and provide feedback on draft plans and materials - Interim reports - Annual reports - Knowledge and insights you could not have obtained on your own

Reporting PROGRAM OFFICER EV REP AL OR T EVALUATOR PI OTHER PROJECT STAKEHOLDERS, E. G. : partners, advisors, participants, host administrators

Reporting PROGRAM OFFICER EV REP AL OR T EVALUATOR PI OTHER PROJECT STAKEHOLDERS, E. G. : partners, advisors, participants, host administrators

Reporting PROGRAM OFFICER EV REP AL OR T EVALUATOR PI OTHER PROJECT STAKEHOLDERS, E. G. : partners, advisors, participants, host administrators

Reporting PROGRAM OFFICER EV REP AL OR T EVALUATOR PI OTHER PROJECT STAKEHOLDERS, E. G. : partners, advisors, participants, host administrators

Reporting Ask your program officer how he/she prefers the evaluation results to be conveyed. Ways of reporting evaluation results to your program officer EV REP AL OR T PI Complete original report Original executive summary Evaluators’ summary in annual report PROGRAM OFFICER

Reporting Ask your program officer how he/she prefers the evaluation results to be conveyed. Ways of reporting evaluation results to your program officer EV REP AL OR T PI Complete original report Original executive summary Evaluators’ summary in annual report PROGRAM OFFICER ASK!

Using Evalu. ATE www. evalu-ate. org

Newsletters Feature articles by experienced ATE PIs and evaluators Q&A Definitions & discussion of evaluation terms Resource highlights

Resource Library Search by - Evaluation task - ATE topic - Keyword - Author

Evaluator Directory Search by - location - discipline - project type - keyword

Annual Survey form FAQs Reports

Webinars Upcoming events Materials from past events: - Recordings - Slides - Handouts

Next Webinar Register at www. evalu-ate. org/events

Conference Activities Concurrent with this session PI and evaluator workshop on evaluation Tonight Showcase #209 -210 Tomorrow Afternoon Birds of a Feather Session: Beyond Satisfaction and Short-Term Self-Reports: Evaluating the Impact of Your ATE Grant

Thank You!

Advice from the Trenches Elaine’s Golden Rule: Elaine Craft ATE PI & Evaluator “ “ The one who has the gold rules! i. e. , don’t disperse grant funds to contractors or partners until obligations have been met

Advice from the Trenches Karl Kapp ATE Evaluator “ “ Set up regular data collection milestones or checkpoints so that data is being collected as part of the normal process of carrying out the grant and doesn't have to be ‘bolted’ onto the process of the grant after it is underway.

Advice from the Trenches Jane Ostrander ATE PI & National Visiting Committee chairperson “ “ The evaluator is the expert in evaluation but you are the expert on your project. If you feel that the evaluation is not providing the information you need, then talk to your evaluator. Do it immediately. … You can change evaluators but first, communicate.