Ongoing STC Research and Evaluation Projects Scott Gibbons

Ongoing STC Research and Evaluation Projects Scott Gibbons & Christina Yancey Chief Evaluation Office, USDOL 2015 STC Summit

Presentation Overview 1. Summary of a recently completed study 2. Summary of 2 active studies 3. What are the approaches being tested? – How are we measuring their effectiveness? – How can the findings from these studies help you? 4. Some Discussion Topics and Questions 2015 STC Summit 2

Recently Completed Employer Survey • Slides will be available in conference materials • Key Findings (KS, MN, RI, WA) – High levels of satisfaction, some barriers to participation • Topics and Areas Investigated – Characteristics of Employer Participation in STC • Example: Industry, Size, Tenure, – Employer Opinions & Perceptions of STC – Employer Awareness of STC – Policy Recommendations to Increase Usage 2015 STC Summit 3

Ongoing STC Research and Evaluation • STC Demonstration Project on Employer Uptake (to be completed in 2017) – Random Control Trial (RCT) and Quasi-Experimental Design (QED), Iowa and Oregon – Testing approaches to increase employer uptake – 12 months of experiment complete in fall 2015, now follow up • Supplemental STC Report (to be completed in early CY 2016) – Mandated by Middle Class Tax Act – Synthesis of recent work, best practices, challenges – Interviews with employers, State staff (CT, RI, MI, TX, OH, WA, OR, FL) 2015 STC Summit 4

Increasing Employer Uptake of STC • Demonstration project in Iowa and Oregon – Can we increase employer uptake of STC? – Can we determine what the impact of different kinds of treatments is on uptake of STC? • An RCT design to test controllable approaches A QED design for less-controllable approaches The QED design reflects the kinds of things many states did or anticipate doing with their STC grants 2015 STC Summit 5

2015 STC Summit 6

What Approaches Increase Employer Uptake? • Increasing information about STC – Direct mailings to employers: Special mailing, notice of claim, tax notices, others – Advertising, “Banners” and links on web sites – Links up on Job listing portal, tax and wage filing, employer area of UI web site • Direct outreach to employers – – – Focus group presentations Media Discussions through industry or affinity groups Educating politicians and representatives Materials provided during 1 -on 1 or group meetings 2015 STC Summit 7

How Do We Measure What Works? • Extract UI claims data (STC, non-STC) and employer agreement data for analysis – Analyze trends and patterns in STC usage – Extent of usage, user characteristics, industry, time period – Analyze changes in numbers of employer agreements • Track responses to outreach – Track web site traffic linked from different forms of outreach, hits indicate responses to the outreach – Track outreach from mailings, personal contacts • Perform surveys of employers, state staff – Satisfaction, awareness, perceptions of benefit 2015 STC Summit 8

How Do We Measure What Works? Many of these approaches can be used now, in your states, to assess important characteristics of STC usage: • What industries are most common among STC users? • Where, geographically, are the establishments located? • Are increases in agreements or decreases in utilization linked to economic factors or conditions such as declining industry employment or output? • What are the characteristics of the individuals filing STC claims? • What are the trends in patterns in counts of claimants, claims and of employer agreements, now and over time? • Do STC claims and regular UI claims increase and decrease at the same time or different times, what factors influence this? • Monitoring for people who transition from STC to regular UI • Monitoring to see if regular UI claims, benefit payments decrease when employers begin STC 2015 STC Summit 9

Planning and Logistics: Ensuring Studies have Resolving Sufficient Power Each data point on the graph represents the number of STC plans initiated in the indicated quarter and the prior three quarters for employers in the proposed treatment and control regions in Oregon. 2015 STC Summit 10

Predicted control and comparison group STC usage and minimum detectable effect estimates for treatment group STC usage, assuming usage during 12 -month study period will be equal to current usage Control 14, 329 Treatment 14, 329 Number of new STC employers Control and comparison Treatment (predicted) (control + MDE) 17 35 12, 331 14, 373 12, 330 15, 059 51 40 Number of employers State Iowa Oregon Study RCT (Portland) QED 80 68 Proportion of STC use Control and comparison Treatment (control (predicted) + MDE) 0. 0012 0. 0024 0. 0036 0. 0028 0. 0056 0. 0045 Predicted control and comparison group STC usage and minimum detectable effect estimates for treatment group STC usage, assuming STC usage during 12 -month study period will be one-half of current usage Number of eligible employers State Iowa Oregon Study RCT (Portland) QED Control and comparison 14, 329 Treatment 14, 329 8 12, 331 14, 373 12, 330 15, 059 26 20 Number of new STC employers Control and Comparison Treatment (predicted) (control + MDE) 22 47 41 2015 STC Summit 11 Proportion of STC use Control and comparison Treatment (control (predicted) + MDE) 0. 0006 0. 0018 0. 0014 0. 0033 0. 0027

Motivation Interventions Goals Strategies Intervention Activities RCT Avert lay-offs Minimize unemployment Maintain business operations Retain employer–employee attachments Awareness Desire Knowledge Ability Reinforcement Emails Postal mailings Quarterly wage reports—banner Employer job postings —banner Notice of claims— letters Webinars (OR) QED (OR) RCT plus Employer group presentations One-on-one employer presentations Supplementary emails from business organization External Factors a. b. Results Economic climate in state, county, industry, and firm; financial costs to state and employers State political and social climate (legislation; Federal grants) 2015 STC Summit 12 Outcomes Short Term Phone inquiries Email inquiries Web hits QR web hits (OR) Outcomes Long Term Plan applications Plan participation Use of STC relative to Unemployment Compensation

Considerations When Reviewing or Applying Research Findings • Are the results generalizable? • What factors would effect your success – Things in your control versus out of your control – Spatial or temporal variance in quality, intensity, frequency of treatment? – Changing conditions in economy, industry? – Similar dynamics between employers and State? – Finding a rationale that resonates 2015 STC Summit 13

Discussion Topics • Applying evaluation principles to your work – Has your agency attempted to increase uptake and measure whether the efforts were successful? • Leveraging DOL’s evaluations – What aspects of this work would you like to know more about (e. g. , research design, approaches, data, measures)? • Disseminating findings – How can we share the results of these studies in a way helpful to you? 2015 STC Summit 14