Performance Accountability in the World of WIOA Introduction

Performance Accountability in the World of WIOA

Introduction l WIOA brings Massive Accountability Changes – Especially for Texas l Changes in Key Accountability Concepts l l l l Participant Exit Exclusions Replaces Existing Common Measures Creates Significant new Reporting Requirements Creates Highly Structured Process for Target Setting & Negotiations Implementation of WIOA’s Accountability Provisions is a Work-in-Progress 2

Overview of Topics l l WIOA Performance Accountability Background Performance Accountability Constructs l l l Introduction to the New Common Measures Employment/Earnings Outcomes l l l Definitions / Calculations Comparison of Old & New Data Effectiveness Serving Employers Data is the Key l l Definitions / Calculations Comparison of Old & New Data Education Outcomes l l Participant / Exit / Exclusions Required Data & Allowable Data Sources Target-Setting & Statistical Modeling Implementation 3

Key Terms & Abbreviations l l l l Title I / Adult, Dislocated Worker, Youth (Adult, DW, Youth) Title II / Adult Education & Family Literacy (AEL) Title III / Wagner-Peyser (WP) Title IV / Vocational Rehabilitation (VR) Temporary Assistance to Needy Families (TANF) Supplemental Nutrition Assistance Program – Employment & Training (SNAP E&T) Career & Training Programs (C&T) l l Primarily Adult, DW, Youth, WP, TANF, SNAP E&T Generally also includes special initiatives Essentially All but AEL, VR, and Child Care Jobs for Veterans State Grants (JVSG) 4

Key Terms & Abbreviations (continued) l Department of Labor (DOL) l l Department of Education (ED) l l l l l Employment Training Administration (ETA) Office of Career, Technical, & Adult Education (OCTAE) Rehabilitation Services Administration (RSA) “The Departments” DOL & ED “The Secretaries” – The Secretaries of DOL & ED Participant Individual Record Layout (PIRL) OCTAE’s National Reporting System (NRS) Notice of Proposed Rule-Making (NPRM) Information Collection Request (ICR) Quarter – (Qtr or Q) YYYYQX – Qtr X in Calendar Year YYYY (e. g. 2004 Q 1) 5

Key Terms & Abbreviations (continued) l l l l l Work. In. Texas. com (WIT) – TWC’s job matching system; also used as the primary system for recording Wagner-Peyser services, included by TVC staff TWIST – TWC’s integrated case-management system used by most programs other than VR & AEL TEAMS – TWC’s AEL case-management system Re. Hab. Works (RHW) – TWC’s VR case-management system On-the-Job Training (OJT) Reemployment & Employer Engagement Measures (REEMs) Measurable Skills Gain (MSG) Effectiveness Serving Employers (ESE) Educational Functioning Level (EFL) 6

WIOA Performance Accountability Background

WIOA l l WIOA Signed into law July 2014 WIOA Replaces Workforce Investment Act & Includes 6 Programs l l l Common Performance Accountability & Reporting l l Title I Adult, Dislocated Worker, Youth (Adult, DW, Youth) Title II Adult Education & Family Literacy (AEL) Title III Wagner-Peyser (WP) Title IV Vocational Rehabilitation (VR) § 116(a) establishes “performance accountability measures that apply across the core programs to assess the effectiveness of States & local areas. . . in achieving positive outcomes for individuals served by those programs” Other Federal Programs Impacted by WIOA l l SNAP E&T now requires use of WIOA-esque measures Draft TANF Bills proposing WIOA or WIOA-esque measures 8

Where We Are: WIOA Regulations l WIOA Required Draft Regulations by April 2015 l l l DOL & ED timely published 2, 400+ pages of proposed regulations for 60 Day Comment Period Nearly 2, 900 sets of comments submitted Final Regulations Issued Required January 2016, but not issued until June 2016 l Final Package 4, 100 pages in length l l l Preamble Estimated Impact/Cost Comments & Responses Regulations 298 pages on Performance Accountability Complete/Final? l Numerous holes to be filled by “Further Guidance” 9

Where We Are: Reporting l Secretaries supposed to develop Joint Reporting Specifications by July 2015 l l l July 2015 – Draft Information Collection Request (ICR) published for 60 Day Comment Period, receiving 112 Sets of Comments April 2016 – 2 nd Draft ICR published for 30 Day Comment Period, receiving unknown # of comments June 2016 – “Final” ICR with Joint Specs published l l Numerous References to more Guidance to Come Apparently the Departments are already working on amendments Joint Specs only include the elements jointly in common across all 6 WIOA programs DOL, OCTAE, RSA each require additional data l l WIOA didn’t specify a due date for these items Requirements issued under separate means 10

Where We Are: Reporting l DOL-specific ICR l l l September 2015 – Draft published for 60 Day Comment Period, receiving 68 sets of comments April 2016 – 2 nd Draft published April 2016 for 30 Day Comment Period receiving unknown sets of comments June 2016 – “Final” DOL Specifications Published l l RSA-specific ICR l l l Numerous References to more Guidance to Come April 2015 – Draft RSA-911 ICR published for 60 Day Comment Period June 2016 – Somehow RSA finalized the RSA-911 without going through a 2 nd draft ICR with Comment Period OCTAE-specific ICR l September 2016 – Finally proposed a new version their National Reporting System for public comment 11

Performance Accountability Constructs & Key Reporting Issues

Reportable Individual vs Participant l Most important definition in WIOA Accountability? l l l Not Everyone? No l l l Definition of “Participant” Nearly all Participants are included in WIOA measures Some People are only “Reportable Individuals” Others are excluded for factors outside the system’s control WIOA Rule 677. 150(b) Reportable Individual is “an individual who has taken action that demonstrates an intent to use program services and who meets specific reporting criteria of the program including: l l l Individuals who provide identifying information; Individuals who only use the self-service system; or Individuals who receive information-only services or activities” 13

Participant l WIOA Rule 677. 150(a) defines as a Participant as a person determined eligible for service AND met specific other programmatic requirements: l AEL – completed 12+ contact hours EVEN IF THEY CROSS PYs l l VR – has an approved/signed Individual Plan for Employment (IPE) & has begun to receive services l l Very different from old State Measure – Not all Consumers are Participants – No IPE? Not a Participant Adult/DW/WP – has received Staff-Assisted qualifying services other than Information-Only Services l l l TWC will not apply the “Cross PY” provision to AEL grantees this year VERY different than old Customer definition – excludes those receiving only Self-Service & Information-only Service (even by staff) TWC is applying this definition for other C&T Programs Youth – has received an Objective Assessment, development of an Individual Service Strategy (EDP) & any of the 14 WIOA Youth program elements identified in WIOA § 129(c)(2) 14

Information-Only Services? l Information-Only Services are not Specific to the Person being Assisted – They are not dependent on assessing or understanding a person’s: l l l Skills Education Career Objectives Barriers to Employment Other specific needs General List of Information-Only Services: l l l All Orientations Computer Work Station Usage (22) LMI (29/LMI, 191/SLM) WOTC Eligibility (139/TCE) Referral to Educational Service (141/RES) Referral to WIOA or Other Training – note that this is the REFERRAL only, not the actual provision of training (the training itself would be Qualifying) 15

14 Youth Elements l l l l Tutoring, Study Skills Training, Instruction, & Evidence-based dropout prevention & recovery strategies Alternate Secondary School Service/Dropout Recovery Services Paid & Unpaid work experience including employment, pre-apprenticeship, internships/job shadowing, & OJT Occupational Skill Training Education offered Concurrent & in the Context of Workforce Preparation activities and training for a specific occupation or occupational cluster Leadership Development Supportive Services Adult Mentoring (which must last at least 12 months) Follow-up Services (which again, must last at least 12 months) Comprehensive Guidance/Counselling Financial Literacy Entrepreneurial Skills Training Career Awareness, Career Counseling, Career Exploration, etc. Activities to help prepare for/transition to Postsecondary Education & Training 16

Customer/Consumer vs Participant l Data reported will look very different for VR and C&T l VR served 102. 9 K Consumers in SFY 16 but only 75. 3 K (73. 2%) are Participants under the new definition l l l 11, 104 Consumers received Blind Services of whom 7, 219 (65. 0%) are considered Participants 91. 7 K Consumers received General VR Services of whom 68. 3 K (74. 4%) are considered Participants C&T Programs served 1. 27 M Customers in SFY 16, but only 830 K (65. 44%) are Participants under the new definition Youth Program served 8, 166 Customers in SFY 16, but only 5, 799 (71. 01%) are Participants under the new definition JVSG data were generally not affected because JVSGservices involves staff-assistance (generally beyond Information-Only service) 17

When Does Exit Occur? l Departments Developed Separate Standards l VR – Exit is upon Closure of File in accordance with RSA regulations l l AEL/C&T – Exit is on Date of last Qualifying Service, but is determined based on lack of Qualifying Service for 90 days l l (Unless Closed due to Supported Employment in an Integrated Environment, but NOT in Competitive Integrated Employment) Self-Service & Information Service do not Extend Participation Planned Gap in Services can prevent exit IF the Participant is expected to return on a specific date for specific services If Participant does not return as planned Exit is retroactive back to the date of the last Qualifying Service Common Exit not allowed across OCTAE, RSA, & DOL l DOL requires Common Exit across their programs l l TWC intends to use Common Exit across all C&T programs ED Claims to be evaluating Common Exit & TWC would likely want to Pilot such a system 18

Impact of Lack of Common Exit? l Undercuts the Idea of an Integrated System l l WIOA Envisions Stronger Integration & Coordination of Services Focus should be on the PARTICIPANTS & their outcomes l l Siloed Exits & Siloed Reporting puts the Focus on the Program Has the Potential to Distort Outcomes l l l Most WIOA measures focus on specific Post-Exit Qtrs If a Participant is coenrolled across AEL, C&T, and/or VR with exits in different quarters, the Participant could be reported differently for each program Biggest Potential Impact is probably on Earnings Outcomes l Example: AEL Participant Exits late 2015 Q 2 continuing Work Search in C&T becoming employed mid 2015 Q 4; Wages in 2015 Q 4 are for a “partial” quarter. Wages for 2016 Q 2 (2 nd qtr after C&T Exit) are more likely to be a “full” quarter 19

Who is Included in Performance? l ALL Participants without a Valid Exclusion: l Institutionalization (criminal or health) l l l l Participants Receiving AEL while Incarcerated are included in Measurable Skills Gain, but not employment, earnings, and credential outcomes Health/Medical condition expected to last at least 90 days that precludes continued participation in the program or employment Death Reserve Forces called to Active Duty Determination of Ineligibility (correcting an earlier mistake) Those in Foster Care who exited due to mandated move from the area (only applies to those 14 -21) BIG CHANGE - Lack of a Valid SSN No Longer Excludes Participants l l 40, 118 out of 94, 391 PY 16 AEL Participants, did not have SSNs recorded in TEAMS (42. 5%) Difference between 55% and 31% Employed Q 2 Post-Exit 20

Consequences of Failure l Greater Consequences for Failure under WIOA l l 2 nd Consecutive Year of Failure by a State results in a 5% Reduction to the Governor’s 15% Statewide Funds 3 rd Consecutive Year of Failure by a local Board results in: l l WIOA Definitions of Failure l State is Considered to Have Failed if l l l Appointment of a New Board Prohibition in the use of certain Providers & Partners found to have achieved a poor level of performance Other Significant Actions as Deemed Appropriate Average % of Target is <90% for all Measures in any PROGRAM Average % of Target is <90% for all Programs in any MEASURE % of Target on any single Measure is <50% States have Flexibility in Setting Standards for Locals & other Subrecipients Legislature sets Failure at Missing Target by >5% 21

Introduction to the New Common Measures

Origin of New Common Measures l Integrated Performance Initiative l DOL-funded grant from 2004 led by Washington State with Goal to Develop Measures l l l l Of Importance to Elected Officials, Policy-Makers, & Stakeholders Acceptable to WF, K-12, & Post-Secondary Education Initial Workgroup of 7 States produced 1 st Proposal validated/revised with 10 other States Proposal validated/revised AGAIN thru discussions with Think-tanks & other Experts Results Championed by National Governors’ Association in all WIA Reauthorization drafts Ultimately adopted in WIOA with few changes l Measurable Skills Gains added in by OCTAE 23

New Federal Common Measures l New Common Measures in WIOA Statute 1. 2. 3. 4. 5. 6. Employed Q 2 Post-Exit (Emp. Q 2) Employed Q 4 Post-Exit (Emp. Q 4) Median Earnings Q 2 Post-Exit (Med. Earn. Q 2) Measurable Skills Gain (MSG) Credential Rate Effectiveness in Serving Employers (ESE) l l MSG & Credential Rate don’t apply to Wagner-Peyser l l l Totally Undefined in Statute Not a Training Program Youth versions of Employed Q 2 & Q 4 also count Enrolled in Education (Emp/Ed Q 2, Emp/Ed Q 4) Changes are most substantial for VR 24

New State Common Measures l Federal Measures were a Missed Opportunity l Difference between Youth & Non. Youth Measures is counting Enrollment in Education l l Federal Measures also don’t include “Retention” l l Why is going back to school “Good” for a 22 or 24 year old, but not a 25 or 30 or 50 year old? Employment in Q 4 is not the same as retaining employment State WIOA measures correct these issues by: l l Focusing on Employment or Enrollment in School/Training Dropping the Q 4 measure in favor of a Retention measure 25

Post-Exit Outcomes Employment/Earnings/ Educational Enrollment

Employed Q 2 Post-Exit l DENOMINATOR – All Exiters without a valid Exclusion l l NUMERATOR – Employed Q 2 Post-Exit l l Unsubsidized Employment Only Good l l Not just those Unemployed at Participation Picks up instances where we help an employed person get a 2 nd Job Bad l l Same Pre- & Post-Exit status considered “successful” (still employed in original job) Adds 1 Quarter of Lag to the Measurement Period l l l Minimize Affect of Added Lag by Running Data on Employed Q 1 ~90% of those Employed Q 1 are also Employed Q 2 Reported to OCTAE, RSA, & DOL (all but Youth) 27

Let’s Look at Some Data!!! Program Entered Employment or % Rehabilitated Employed Q 2 Post-Exit C&T 78. 4% 69. 9% JVSG 72. 9% 65. 3% AEL 45. 5% 31. 7% VR (different periods) 59. 0% 57. 9% l C&T differences not obvious l l l Entered Employment was 741. 2 K out of 945 K Exiters Employed Q 2 Post-Exit was 565 K out of 808 K Exiters Difference is in definition of Participant & new measure includes employed OR unemployed at Participation AEL differences mostly due to no “Lack of SSN” exclusion – Excluding non-SSN Participants raises performance to 55% VR used to focus on status at Exit vs. Post-Exit l Differences may also be due to lack of way to record Post-Exit employment in RHW 28

Employed Q 4 Post-Exit l l DENOMINATOR – All Exiters without a valid Exclusion NUMERATOR – Employed Q 4 Post-Exit l l Not a “Retention” Measure l l l Unsubsidized Employment Only Not a Subset of those Employed Q 2 Good? Nothing Good about it Bad l Meaningless Measure l l Counts those not employed Q 1 -3 Post Exit if Employed Q 4 Same Pre- & Post-Exit status considered “successful” Adds 1 Quarter of Lag to the Measurement Period Reported to OCTAE, RSA, & DOL (all but Youth) 29

Let’s Look at Some More Data!!! Program Employment Retention Employed Q 2 Post-Exit Employed Q 4 Post-Exit C&T 84. 6% 69. 9% JVSG 84. 1% 65. 3% AEL 58. 3% 31. 7% NA VR 88% 57. 9% 56. 1% l REMEMBER Employed Q 4 is NOT a Retention Measure l It is essentially the same population as the Employed Q 2 measure followed-up upon 2 quarters later 30

Median Earnings Q 2 Post-Exit l Median Earnings is the Earnings Level at which half of those employed earn MORE & half earn LESS l l l $5, 000, $5, 050, $6, 000 – Median is $5, 050 $4, 900, $5, 000, $6, 266 – Median is $5, 500 Good l l l Shorter Lag than old Average Earnings (Q 2 vs. Q 3) “Median” reduces influence of outliers 5 Exiters with $5, 000, $6, 500, $150 K Earnings l l l Bad l l l Average is $34, 500 Median is $6, 000 Performance tends to be about 1/3 rd the old measure Smaller Numbers are less meaningful to Stakeholders Reported to OCTAE, RSA, & DOL (including Youth) 31

Let’s Look at Some More Data!!! Program Average Earnings Median Earnings (Average Earnings in Q 2/Q 3 of those employed in Q 1/Q 2/Q 3) C&T $15, 895 $4, 956 JVSG $18, 253 $6, 616 AEL NA $4160 VR $12, 468 $4, 319 l Old Measure Only included those employed in Q 1, Q 2, & Q 3 l l l Tended to include more people with Stable Employment For JVSG this was 16, 782 Exiters New Measure includes anyone with any Wages in Q 2 Post-Exit l For JVSG this was 18, 643 Exiters 32

Employed/Enrolled Q 2 Post-Exit l DENOMINATOR – All Exiters without a valid Exclusion l l NUMERATOR – Employed or Enrolled in Education Q 2 Post-Exit l l l Recognizes Enrollment in Education as a Positive Outcome Bad l l Unsubsidized Employment Only Education or Advanced Training Good l l Not just those Unemployed or Not in School at Participation Same Pre- & Post-Exit status considered “successful” We don’t have much post-exit Enrollment Data Yet Adds 1 Quarter of Lag to the Measurement Period Reported to LBB (VR, AEL, C&T) & DOL (Youth Only) 33

Employed/Enrolled Q 2 -4 Post-Exit (Retention) l l DENOMINATOR – All Exiters without a valid Exclusion Employed or Enrolled in Education in Q 2 Post-Exit NUMERATOR – Employed or Enrolled in Education both Q 3 & Q 4 Post-Exit l l Good l l RETENTION!!! Deeper look at Success Bad l l l Unsubsidized Employment Only Education or Advanced Training Can be any combination of Employment & Education Same Pre- & Post-Exit status considered “successful” Adds 1 Quarter of Lag to the Measurement Period Reported to LBB Only (VR, AEL, C&T) 34

Let’s Look at Some More Data!!! Program Employment Retention Employed Q 4 Post-Exit Employed/ Enrolled Q 2 -4 Post-Exit C&T 84. 6% 69. 9% 84. 7% JVSG 84. 1% 65. 3% 81. 9% AEL 58. 3% NA 83. 5% VR 88% 56. 1% 85. 7% 35

Educational Outcomes

Credential Rate l DENOMINATOR – Exiters without valid Exclusion who: l l Were in Training excluding OJT or Customized Training; OR Were in Education at or above Secondary School Level at Date of Participation or During Period of Participation l l NUMERATOR – Within 1 Year of Exit Achieved: l l Recognized Post-Secondary Credential; or High School Diploma or Equivalent AND was also either: l l l What does that mean for English as a Second Language in AEL? We don’t know – OCTAE never seemed to anticipate it Employed in any of the 4 Quarters Post-Exit employed; or Enrolled in Post-Secondary Education/Training at some point during the Year following Exit Does not appear to consider moving from Secondary to Post. Secondary Enrollment or Enrollment in Integrated Education & Training during Period of Participation OCTAE seems to be trying to limit Numerator 37

Credential Rate (continued) l Examples of Recognized Post-Secondary Credentials – l l l Good l l Associate’s, Bachelor’s, Graduate or Post-Graduate Degree Occupational Certificates (awarded by either educational institution or independent education or training provider upon successful completion of a course of study & exam) Occupational Licensure (generally awarded by a governmental entity) Occupational Certification (often issued by 3 rd party organization based on demonstrated competencies) Having a Non. Youth Credential measure Bad l Lots of Lag – One Year Post Exit to Achieve Credential l l Diploma Requires Employed or in Post-Secondary in Year after Exit Reported to RSA, OCTAE, DOL & LBB (VR, AEL, C&T) 38

Let’s Look at Some More Data!!! Program Educational Achievement (C&T) / Achievement of GED (AEL) Credential Rate C&T 80. 0% (10, 440 Exiters) 55. 4% (9, 513 Exiters) AEL 52. 0% 22. 1% VR NA 30. 0% l C&T has Big Changes – l l Failure to Achieve due to “Entered Employment” no longer an exclusion OJT & Customized Training no longer put Participant in Denominator AEL now looks at those who reached ASE level rather than just those who took all 4 Parts of the GED test VR doesn’t have a good means to track Credentials, especially Post-Exit 39

Measureable Skills Gain l DENOMINATOR – All PARTICIPANTS without a valid Exclusion in Education or Training during the POP l l Even if Education/Training not paid for by the program (other than perhaps AEL – we’re still looking at that) NUMERATOR – Achievement of a MSG during the PROGRAM YEAR (July-June) Good – Having an interim measure of progress Bad – Completely Arbitrary Period l l l Harder to Attain MSG if enrolled in education/training late in the Program Year YOU DON’T WORRY ABOUT THAT – YOU FOCUS ON NEEDS OF PARTICIPANT WE WILL FIGURE OUT A WAY TO ADJUST TARGETS 40

Measureable Skills Gain l Measurable Skills Gains Include: l Educational Functioning Level (EFL) Gain – l l Attainment of Credential (including Diploma/Equivalent) Transcript/Report Card: l l Comparison of Pre-Test & Post-Test Scores Exit from Program with enrollment in Post-Secondary Education/Training during the SAME PROGRAM YEAR Sufficient # of Credit Hours & meeting State’s Academic Standards 12 Hours in a Semester or 12 hours over 2 consecutive Semesters Training Milestone – Satisfactory or better progress report towards established milestones such as completion of OJT or One Year of Apprenticeship Skills Progression: l l Successful Completion of Exam required for a particular Occupation Progress in attaining Technical/Occupational Skills as evidenced by trade-related benchmarks such as knowledge-based exams 41

Common or Uncommon? l l WIOA requires Common Measures OCTAE is proposing different standards for Credential Rate & Measurable Skills Gains using NRS l l l Only want to count non-Diploma/Equivalent Credential if in Integrated Education & Training Only want to count MSG if achieved thru EFL or achievement of Diploma/Equivalent (not other Credential) RSA is proposing different standard for MSG by redefining the Joint PIRL data elements in the RSA 911 l Changes relatively mild (referencing VR Program Policies) but this reduces commonality 42

Effectiveness in Serving Employers

Effectiveness Serving Employers (ESE) l Undefined in Statute l l Secretaries required to consult with States, Locals, Employers, Stakeholders to develop Secretaries couldn’t settle on anything l l Good l l l Proposed 3 measures & directed states to select 2 to pilot Having National Measures of Effectiveness Serving Employers Applying measures across programs rather than separately Bad l l l DOL & ED also referring to this as Employer Satisfaction Problems with all 3 proposed measures Not clear how they can be applied for LVER even though DOL wants an LVER-specific set of measures 44

ESE #1 Market Penetration l % of Employers Served l l Denominator – Total # of Establishments Numerator – # of Establishments that received service l l l l Employer Information & Support Services Workforce Recruitment Assistance Strategic Planning/Economic Development Activities Untapped Labor Pool Activities Training Services (including Incumbent Worker Training) Rapid Response/Business Downsizing Assistance Problems l l Which Services are Most Meaningful? Which are Easiest to “Get Credit For? ” 45

ESE #2 Repeat Business l % of Employers Served that are Repeat Customers l l Denominator – Total # of Establishments Receiving Service at any time in last 3 years Numerator – # of Establishments that received service this year AND at any time in the last 3 years l l Same Services as Market Penetration Problems l Creates a Disincentive to Serve Smaller Employers l l >93% of Employer Served with 1000+ Employees were repeat customers >75% of Employers Served with 100 -249 Employees were repeat customers <60% of Employers Served with <10 Employees were repeat customers Creates a Disincentive to Serve New Employers l l Denominator includes anyone served this year, even if for 1 st time Numerator might not include those served twice in the current year 46

ESE#3 Retention with Same Employer l Looks at Employers in Q 2 and Q 4 to confirm match l l l What if multiple Employers in Q 2 and Q 4? l l Any match counts Good l l l Denominator – Those Employed Q 2 Numerator – Those Employed Q 4 at Same Employer as Q 2 Could have been an amazing measure Would have allowed system to compare retention of our job seekers with all employees at an employer or industry Bad l l Counts people who don’t get new jobs Exit-based instead of based on quarter of new hire l Effectively anywhere from 3 Quarters to 5 Quarters of Retention 47

Data is the Key

WIOA is Highly Data Intensive l States Required to Capture & Report Far More Data l Statute Lays Out Data Gathering Requirements l l l Statute Lays Out how States Report Data l l Dozens of Specific Data Elements Applies to “Formula” and “Statewide” Funded Participants Not just Common Measures, Common Reports Statute Prescribes how the Departments will Use Data to Set Targets & Evaluate Performance l l The Departments have to Develop Statistical Models to Set Targets during Negotiations and RESET Targets at the end of the Year (Down or UP) The Models are built with & run on Data We are not going to be judged against an Arbitrary Target that OCTAE, RSA, or DOL create Success will be based on the results we achieve given WHO we serve & the conditions in which we serve them 49

Data Requirements l l The Departments Developed a Set of Joint Performance Reporting Specs “Participant Individual Record Layout” – (PIRL) l The PIRL contains the basic Elements required to be reported on EVERY Participant l l l 76 Different Elements (though some are conditional) The Departments also Developed Specs for the Common Performance Reports The Departments each ADDED to the Joint PIRL l l l DOL Created a “DOL-PIRL” that adds several hundred elements RSA incorporated the 76 Joint PIRL elements into an updated RSA-911 OCTAE is largely doing their own thing thru their NRS 50

Joint PIRL Elements l PIRL is made up of 4 Main Sections l Participant Information l l Tells who was served Tells the status of the those served Helps the Statistical Model account for it Participation Information l l Tells which programs contributed to Success (or lack thereof) Identifies the Types of Services provided Employment/Earnings Outcome Information l Education Outcome Information l 51

Participant Information l SSN is not reported to DOL/OCTAE but is Critical l l l Date of Birth Sex Race Ethnicity Employment & Educational Status at Participation l l No SSN? Harder to report outcomes Not A Factor in the new Common Measures, but of value in Statistical Model Barriers to Employment l l Statute incredibly Specific about these Elements Statute requires Statewide Reports to break all data out by Barrier to Employment 52

Barriers to Employment l WIOA requires reporting detailed information on Barriers to Employment on ALL Participants l l l l Displaced Homemakers Low-income individuals Indians, Alaska Natives, and Native Hawaiians Individuals with disabilities Older individuals Ex-offenders Homeless individuals Youth who are in or have aged out of the foster care system Individuals who are English language learners, individuals who have low levels of literacy, and individuals facing substantial cultural barriers Eligible migrant and seasonal farmworkers Individuals within 2 years of exhausting lifetime eligibility under part A of title IV of the Social Security Act Single parents (including single pregnant women) Long-term unemployed individuals Other groups identified by the Governor 53

Participation Information l We have to identity each Program that the Participant was Enrolled in l That’s across DOL, OCTAE, RSA and even other Partner Programs: l l l Adult, Dislocated Worker, Youth, Wagner-Peyser AEL VR Youth. Build & Job Corp Also Key Period of Participation information Date of Participation l Date of Exit l Exclusion Reason (if any) l Service Types l Eligible Training Provider & Program of Study (if in training) l 54

Employment/Earnings Outcome Data l Have to Report Employment & Earnings by Quarter for 4 Quarters Post Exit l l Why 4 Quarters when WIOA measures focus on Employment in Quarters 2 & 3 & 4? l l Includes Identifying Source of Information Because achievement of Diploma/Equivalent also required Employment or Enrollment in Education in year Post-Exit Sources of Data l l Primarily Wage Records from Texas, U. S. Government & other States Other Records of Employment l l l Used when Wage Records not Available (not sure if can combine) Have to be recorded separately for each Post-Exit Quarter All IT Systems have to be modified to allow data to be recorded 55

Education Outcome Data l l l Type/Date of each MSG Type/Date of each Credential Achieved Whether Employment in Q 2 Post-Exit was related to Training Post-Exit Enrollment in Post-Secondary Education/ Training Sources of Data l l l Primarily Administrative Records manually recorded Working to develop Data Matching with Texas Higher Education Coordinating Board All IT Systems have to be modified to allow data to be recorded 56

Target Setting & Statistical Modeling

Target Setting l WIOA § 116 lays out target setting in detail: l l l States Negotiate Targets for 2 years at a time based on assumptions regarding Casemix & Economic Conditions Targets are retroactively adjusted Down or UP based on the ACTUAL Casemix & Economic Conditions WIOA § 116 specifies factors to be considered in State Negotiations: l l How state targets compare to one another Government Performance Results Act Goals l l l Long-Term National Goals set by the Secretaries Degree to which the proposals support Continuous Improvement Casemix & Economic Conditions accounted for thru the Statistical Models 58

Statistical Models l l Estimate the level of performance that “should” be achieved for a given Casemix & set of Economic Conditions Developed using historic data to establish norms Each Measure has its own Statistical Model Development involves estimating the impact that each factor has on an outcome l l Essentially if the % of the Casemix that is _______ were to increase by X%, how much is the outcome likely to go up or down? All Probability Based 59

Statistical Models l Not All Models have the Same # of Factors (or the Same Factors) l l l Adult Employed Q 2 Post-Exit has 50 Factors DW Employed Q 2 Post-Exit has 50 Factors Youth Employed/Enrolled Q 2 Post Exit has 51 Factors WP Employed Q 2 Post-Exit has 31 Factors Some of the Difference is due to Data Availability Even when Models have a Factor in Common, the “impact” of the Factor can be Different l Impact of a 1 point increase in % of Participants who are TANF Recipients: l l l -. 055 points for Adult -. 439 points for DW -. 106 points for Youth 60

Adult Employed Q 2 Post-Exit Model l There are 50 Factors in this Model l l l 12 for Participant Demographics (sex, age, race, ethnicity) 6 for Participant Education Level 4 Related to Pre-Participation Employment 9 Related to Barriers for Employment 7 Related to Types of Services Provided 9 Related to Economic Factors 3 Miscellaneous Factors 61

Demographic Factors with Biggest Impact Demographic Factor General Impact of a 1 Point Increase in the Percent of Participants on Performance Age 66+ -. 68 Points Associate’s Degree +. 32 Points More than One Race -. 31 Points Limited English Proficiency -. 23 Points Asian +. 21 Points Less than High School -. 17 Points Earnings in Q 2 Pre-Participation +. 15 Points Individuals with Disability -. 14 Points l l Serving Higher % of Participants with Barriers to Employment will Reduce Performance Target Failure to Document Barriers will generally result in HIGHER Targets 62

Economic Factors with Biggest Impact Economic Factor General Impact of a 1 Point Increase in an Economic Factor on Performance Employment in “Other Services” Sector - 1. 66 Points Unemployment Rate - 1. 4 Points Employment in Construction - 1. 29 Points Employment in Information, Finance, or Technology +. 62 Points Employment in Leisure, Hospitality, or Entertainment -. 61 Points l Bad Economy will generally Reduce Performance Target 63

Implementation

How will Implementation Occur? l l WIOA § 116 provides that Performance Accountability Begins PY 16 Actual Implementation more Complicated l l WIOA requires use of Statistical Models in Target Setting Departments Decided to Delay Implementation of those Measures without sufficient Data to Develop Models l l l Departments delaying Effectiveness in Serving Employers DOL delaying MSG & Youth Median Earnings JVSG delaying all WIOA Measures (didn’t like available models) OCTAE delaying all but MSG (even though they don’t have a statistical model) RSA delaying ALL WIOA measures Departments Initially Suggested that this would be a 2 Year Delay – They’re wrong 65

Current Year End Performance Periods Measure Exiters From Exiters To Employed Q 2 Post-Exit 2015 Q 3 2016 Q 2 Employed/Enrolled Q 2 Post-Exit 2015 Q 3 2016 Q 2 Median Earnings Q 2 Post-Exit 2015 Q 3 2016 Q 2 Employed Q 4 Post-Exit 2015 Q 1 2015 Q 4 Employed/Enrolled Q 2 -Q 4 Post-Exit 2015 Q 1 2015 Q 4 Credential Rate 2015 Q 1 2015 Q 4 Retention with Same Employer 2015 Q 1 2015 Q 4 From To Measurable Skills Gain 2016 Q 3 2017 Q 2 Market Penetration 2015 Q 1 2015 Q 4 Repeat Business 2015 Q 1 2015 Q 4 Measure 66

Statistical Model Development l l Need 2+ years of Data to develop Models (preferably more) Departments Instructed States to not Report on those Exited Prior to 7/1/16 (i. e. no reporting prior to 2016 Q 3) Measure Date when 2 Years of Data Available Statistical Models? Employed Q 2 Post-Exit (AEL & VR) PY 18/Fall 2019 PY 20/Spring 2020 Employed/Enrolled Q 2 Post-Exit (AEL & VR) PY 18/Fall 2019 PY 20/Spring 2020 Median Earnings Q 2 Post-Exit (AEL & VR) PY 18/Fall 2019 PY 20/Spring 2020 Employed Q 4 Post-Exit (AEL & VR) PY 19/Fall 2020 PY 21/Spring 2021 Credential Rate (AEL & VR) PY 19/Fall 2020 PY 21/Spring 2021 Measurable Skills Gain (All) PY 17/Fall 2018 PY 19/Spring 2019 Retention with Same Employer (All) PY 19/Fall 2020 PY 21/Spring 2021 Market Penetration (All) PY 19/Fall 2020 PY 21/Spring 2021 Repeat Business (All) PY 19/Fall 2020 PY 21/Spring 2021 67

State & Local Accountability l l l TWIC required new WIOA-based measures to be Reported for PY 15/FY 16 (just concluded) Legislature/Governor will require new WIOA-based measures to be Reported for PY 17/FY 18 TWC is applying WIOA-based measures for Local Boards in PY 16/FY 17 (i. e. NOW) per DOL guidance TWC is following OCTAE’s lead in AEL Accountability using only MSG but coupling it with Participants Served RSA has delayed WIOA reporting for VR reporting to PY 17/FY 18 (i. e. NEXT YEAR) l System still accountable at State Level for the historic measures for one more year 68

Reporting System Changes l Board performance has historically been run thru “TWIST Web Reports” l l l Reports based on a Common Measures Data Model using the Old Common Measures rules for participation, exit, exclusion Can’t run WIOA measures using Old Common Measures Data Model IT Systems can’t run both sets of Performance Data the same Weekend l l Plan is to develop a schedule for Running Old vs New each Month Might run alternating weeks Might run one of them only once a month Even during week where one set was not run, the last version of that data will be still be available via reports 69

Reporting System Changes l AEL Grantees have historically had access to reports thru TEAMS l l These Reports have not been perfectly aligned with federal requirements & are definitely not aligned with WIOA TWC plans to modify these reports to match WIOA’s requirements l l Complicated by different in State & OCTAE implementation timeframes Also complicated by OCTAE’s NRS instructions not fully aligning with the Joint Performance Reporting Specs (maybe) Currently using Ad Hoc processing for federal/state reporting VR likely to be a hybrid reporting system for some time l l l RSA now moving to quarterly RSA-911 (from annual) RSA-911 currently doesn’t have access to wage records, requiring Ad Hoc processing for federal/state reporting Will work with VR Management on appropriate reports during interim transition & implementation 70

Local Board Accountability l WIOA Specifies the Measures that must be Contracted l l l WIA did as well but TWC had a Waiver That Waiver was not Approved for use in WIOA TWC required to contract 11 Siloed Measures this Year l l l Adult Employed Q 2 Post-Exit Adult Employed Q 4 Post-Exit Adult Median Earnings Q 2 Post-Exit Adult Credential Rate DW Employed Q 2 Post-Exit DW Employed Q 4 Post-Exit DW Median Earnings Q 2 Post-Exit DW Credential Rate Youth Employed/Enrolled Q 2 Post-Exit Youth Employed/Enrolled Q 4 Post-Exit Youth Credential Rate 71

Local Board Accountability (continued) l TWC strongly Objects to Use of Siloed Measures l l l Not Aligned with concept of Integration Boards responsible for more than Adult, DW, & Youth Contracting siloed measures for each Program would mean contracting 32 measures l l l l 3 for WP Veterans 3 for WP Disabled Veterans 4 for SNAP E&T 5 for Choices 2 for REEMS 1 for Child Care 72

Local Board Accountability (continued) l TWC decided to minimize siloing l l l l 11 WIOA Mandatory Measures 4 WIOA-based Integrated (all Participant) Measures 2 REEMs Average # of Children Served per Day - Discretionary Care Choices Full Work Rate Long Range Goal is to find a way to automatically Coenroll all Participants into Adult, DW, or Youth Some Challenges Exist l l Have to address Male Selective Service Requirement Can’t automatically Coenroll if Statistical Adjustment Model doesn’t sufficiently account for casemix changes 73

Review of Topics l l WIOA Performance Accountability Background Performance Accountability Constructs l l l Introduction to the New Common Measures Employment/Earnings Outcomes l l l Definitions / Calculations Comparison of Old & New Data Effectiveness Serving Employers Data is the Key l l Definitions / Calculations Comparison of Old & New Data Education Outcomes l l Participant / Exit / Exclusions Required Data & Allowable Data Sources Target-Setting & Statistical Modeling Implementation 74