FINANCIAL EDUCATION EVALUATION TOOLKIT APPLYING EVALUATION CONCEPTS AND

FINANCIAL EDUCATION EVALUATION TOOLKIT® APPLYING EVALUATION CONCEPTS AND AN ENHANCED EVALUATION TOOLKIT USER EXPERIENCE

BILLY J HENSLEY, PHD PEGGY MULDOON, AFC®

What is research? 3

Research is… • The systematic investigation into, and study of, materials and sources in order to establish facts and reach new conclusions. – OR IN OTHER WORDS… • To investigate or discover proof 4

What is evaluation? 5

What is Evaluation? • The systematic collection of information about the activities, characteristics, and results of programs to make judgments about the program, improve or further develop program effectiveness, inform decisions about future programming, and/or increase understanding. (Patton, 1997) 6

In other words… • • Assessment Is something working well or not How do we improve Should we continue 7

Research vs Evaluation • RESEARCH • EVALUATION – Knowledge intended for use – Program- or funderderived questions – Judgmental quality – Action setting – Often not published – Production of generalizable knowledge – Researcher-derived questions – Paradigm stance – More controlled setting – Published 8

In other words… Research seeks to prove and evaluation seeks to improve. 9

What is Financial Education Program Evaluation? • The implementation of a financial education intervention and comparing learner achievements with program goals and objectives to determine the success or failure of the educational program. 10

Primary Objectives of Evaluation • Exploring the operations or process of a financial education intervention, and • Exploring the learner outcomes or achievements as a result of participation in the intervention. 11

Formative Evaluation • Does the financial education program or • • curriculum address participant needs? Is the information presented relevant to the participants? Are the materials being implement as intended? A good formative evaluation provides data to support these areas, ultimately helping educators make program improvements. 12

Summative Evaluation • Is there a significant change in participants’ • • outcome before and after participation? Do participants demonstrate better outcomes than nonparticipants? Do participants experience long-term benefits? Documented program outcomes help funding agencies measure the worth of programs and allocate more funds to stronger educational programs. 13

Identify the Evaluation Stakeholders & Team Evaluation Stakeholders are individuals who share an investment or interest in the findings of an evaluation. 14

What is our Program or Project Trying to Do? • A logic model usually is a visual representation of the relationships between the resources invested, the activities that take place, and the benefits or changes that result. • The logic model depicts the programming process in graphical form to help clarify what activities should be implemented to promote the financial education outcomes as intended. 15

Logic models inform the evaluation process by providing guidance about appropriate outcomes, outputs, and indicators of high-quality activities. 16

17

What Outcomes Should Be On The Logic Model and Evaluation? Measureable Reasonably Achievable Specific 18

The lower the level, the easier it is to document, but the weaker the evidence to justify the program. 19

Sample Program Outcomes 20

LIMITS • He have several limits to what we can do and measure… – Outside factors – Quality of partners – Access to high-quality programs, resources and information – Etc. 21

What we can control • • • Quality of content Consistency across the brand Message Platforms Accessibility of staff and materials Etc. 22

What factors contribute to impact? 23

The Three Questions • Who is the organization serving? • How well is the organization serving clients? • Is the organization ready to successfully innovate, evolve and/or scale? 24

Q 1 Demographic Information • Even basic demographic information can be very difficult to access for some organizations. – Capacity – Resources – Client interaction type 25

Q 1 Demographic Information Level 2: Basic Demographic Information Who is the Organization Serving? Level 1: Little to No Data Collected 26

Q 2 Serving Clients How Well Is the Organization Serving its Clients? Level 4: Outcome Data Collection and Use Level 3: Dosage / Output Data Collected 27

Q 2 Serving Clients • Level 3 – This is focused on the programming and instruction (“Showing up” data) • Do people enjoy the class, teacher, etc. • Is the program working well, do people like the website, etc. • Downloads, Time on Site, etc. • Level 4 – Is the “IT” making a difference • Are grades higher, did behavior change, etc. – Difficult to get to this level for most people – If can prove program is working, most may not need to go to the next level(s) 28

Reminder • Summative Evaluation: Did it work? • Formative Evaluation: Why did it work? 29

Q 3 Innovate, Evolve, or Scale Is the Organization Ready to Successfully Innovate, Evolve, and/or Scale? Level 7: Second (or 3 rd) High Rigor Formal Evaluation Level 6: High Rigor Formal Evaluation Level 5: Formal evaluation projects 30

Q 3 Innovate, Evolve, or Scale • Level 5 – Low rigor; frequently internal – Usually no RCT • No benchmark group • Level 6 – External – RCT • Level 7 – Formal evaluation to test Level 6 evaluation 31

Evaluation Capacity Range Level 7: Multiple High Rigor Evaluations Level 6: High Rigor Evaluation Level 5: Formal evaluation projects Level 4: Outcome Data and Use Level 3: Dosage / Outputs Level 2: Basic Demographics Level 1: Little to No Data 32

Capacity & Concerns • Unfortunately… – Very few opportunities for nonprofits or small organizations to have a thorough conversation on evaluation strategy – If you are considering evaluation after your program is running, you are already behind. • Goals and objectives must be the framework that guides the program and the evaluation. • 5% of RCTs show strong impact • 5% show they made things worse • 90% show more progress needs to be made 33

Another capacity concern…expertise 3 or more full time staff devoting 100% of time to evaluation 2 full time staff devoting 100% of time to evaluation Senior level person has 100% of their time committed to evaluation tasks 9% Junior / mid-level person has 100% of their time committed to evaluation tasks Senior leader/staff responsible for evaluation tasks Evaluation tasks part of a program staff member’s duties No staff formally assigned to data and evaluation tasks 34

Evaluation Capacity Matrix Level of Evaluation Capacity Data and Evaluation Infrastructure Staffing Multiple high rigor evaluations 3+ full time staff spending 100% of time on evaluation Advanced Moderate Low High rigor evaluation 2 full time staff spending 100% of time on evaluation Formal evaluation project Senior person spends 100% of time on evaluation Outcomes Junior / mid person spends 100% of time on evaluation Dosage / outputs Part of senior person’s job Basic demographics Part of program staff person’s job Little to no data No assigned staff 35

High Functioning Nonprofits • Most nonprofits offer programs and services, but… • Few are high functioning enough to have a sophisticated operation of: – Fundraising – Accounting and Finance – Evaluation • Gets the least amount of work and focus 36

NEFE’s External Evaluation Commitment • We have the expertise to provide evaluation • • capacity building to educators and nonprofits at a basic level. We ask grantees to demonstrate evaluation capacity to assure high quality research projects. We have committed resources to help educators and nonprofits reach a higher level of evaluation capability. 37

Evaluation Toolkit 38

Evaluation Toolkit Overview • • Identify learning objectives. Create a new evaluation. Write custom introduction. Choose from the question bank or write your own questions. Write custom thank you. Email the evaluation to your learners. View and analyze the results of the assessment. Print or download your evaluation to share with stakeholders. 39

Create an Evaluation 40

Create an Evaluation from a Template High School Financial Planning Program (HSFPP) http: //www. hsfpp. org 41

Create a Custom Blank Evaluation 42

Name Your Evaluation • Name your evaluation • Describe it (optional) • You can edit it at a later time 43

Write a Custom Introduction 44

Request Participant’s Names 45

Write a Custom Thank You 46

Use the Question Bank or Create a Custom Question 47

Search Questions By Keyword 48

Search Questions Using Filters 49

Apply Filters for Your Audience Filter by: • • • Question Type: Knowledge, Behavioral, etc. Age: Elementary, Middle School, High School, Adult Question Format: Multiple Choice, T/F, Likert Scale, Select All That Apply, Short Answer Concept: Budgeting, Saving, Spending, Credit, etc. Benchmark Surveys: National Financial Capability Study, etc. 50

Question Bank (Available Questions) 51

Create Custom Questions 52

Share Your Evaluation 53

Student View 54

Student Thank You Screen 55

Access Your Evaluation 56

Results Screen 57

Accessible on Mobile 58

Discussion • Beyond what we do internally: – How do we formally incorporate principles of evaluation into our daily work? – Are the users of our sites, tools and programs meeting the objectives we have for them? – Do all of our sites, tools and programs have objectives? – What do well? – Where can we improve? 59

Questions 60