Developing a Local and State Customer Satisfaction Strategy

Developing a Local and State Customer Satisfaction Strategy An Overview of an Integrated Customer Satisfaction Approach Barry A. Goff and Ronald W. Schack, The Charter Oak Group, LLC For The US Department of Labor and the Workforce Excellence Network July, 2001 The Charter Oak Group, LLC

Overview n n n Requirements under the Workforce Investment Act Using a One-Stop perspective Integrating state and local strategies Analysis and reporting Customer Satisfaction as a part of a performance management measurement system and continuous improvement July, 2001 The Charter Oak Group, LLC 2

The Workforce Investment Act n [WIA Section 136(b)(2)(B)]: “the customer satisfaction indicator of performance shall consist of customer satisfaction of employers and participants with services received from the workforce investment activities authorized under this subtitle. ” July, 2001 The Charter Oak Group, LLC 3

The Workforce Investment Act, continued The legislation [Section 136(b)(3)(A)(I)] also requires that there be state-adjusted levels of performance for customer satisfaction that Can be expressed in an objective, quantifiable, and measurable form; and Show the progress of the state toward continuously improving in performance July, 2001 The Charter Oak Group, LLC 4

The American Customer Satisfaction Index 3 Questions Ø Ø Ø Overall satisfaction Compared to expectations Compared to the ideal 1 -10 scale for each ACSI index: 3 questions weighted, averaged together, and converted to a 0 -100 scale July, 2001 The Charter Oak Group, LLC 5

CS Indicator Questions American Customer Satisfaction Index: Utilizing a scale of 1 to 10 where “ 1” means “Very Dissatisfied” and “ 10” means “Very Satisfied” what is your overall satisfaction with the services provided from_____? Considering all of the expectations you may have had about the services, to what extent have the services met your expectations? “ 1” now means “Falls Short Of Your Expectations” and “ 10” means “Exceeds Your Expectations. ” Now think of the ideal program for people in your circumstances. How well do you think the services you received compare with the ideal set of services? “ 1” now means “Not Very Close To the Ideal” and “ 10” means “Very Close to the Ideal. ” July, 2001 The Charter Oak Group, LLC 6

ACSI Scores/Benchmarks July, 2001 The Charter Oak Group, LLC 7

ACSI Scores/Benchmarks July, 2001 The Charter Oak Group, LLC 8

Basic Methodology n n State-level telephone survey of Title IB exiters with ACSI questions first Local-level survey conducted with exit cohorts in a consistent manner within a state making every effort to avoid bias July, 2001 The Charter Oak Group, LLC 9

The lead-in question Statement of confidentiality Which of the following services did you receive from the [One-Stop]? Help in career planning Assistance with your job search Recommendations for job training Help in resume writing or interviewing Etc. First I want you to rate your overall experience with the services. I will read three statements and ask you to rate your experience for each. July, 2001 The Charter Oak Group, LLC 10

Guidance: Sampling for State Survey • Required Participants: exiters as defined for core indicators (Those with exit date or 90 days without service) • Required Employers: those who participated in substantial services that may include labor exchange, OJT, Rapid Response activities, Job Fairs, etc. • Random sampling where sufficient numbers exist • Minimum acceptable response rate is 50% in the first year and 70% for subsequent years • At least 500 completed surveys from participants and 500 from employers where total population allows July, 2001 The Charter Oak Group, LLC 11

Guidance: Other Parameters for State Survey • • Participants: Data collected within 60 days after exit Employers: Data collection within 60 days after service Quarterly data collection Report with WIA quarterly report using reporting format in specifications Send letters indicating surveying will take place Require those providing services to indicate that customer satisfaction surveying is a regular practice in which they may be asked to participate Telephone methodology Statement of confidentiality July, 2001 The Charter Oak Group, LLC 12

First level enhancements to the State-level CS Survey n n Attach demographics Attach to individual respondents’ administrative records, while maintaining confidentiality Ask additional program-relevant questions Ask open-ended questions (e. g. , “what did you appreciate the most” and “what did feel could be improved? ”) July, 2001 The Charter Oak Group, LLC 13

Requirements for local surveys n n n Must be a consistent strategy across the state Does not have to be the same local strategy in different states Does not have to use the ACSI 3 ACSI questions can be used without license but can not be referenced as ACSI If ACSI is used locally a license must be purchased for each local area in the state July, 2001 The Charter Oak Group, LLC 14

Local Point of Service Surveys n n n n Surveys can be collected “at the door” Like telephone surveys include open-ended questions Ask the three questions first as in your survey for federal performance reporting Use alternating surveys, short and long, so that demographics can be collected once or twice a year New technologies can help, including scanning technologies that compress data entry time Cost is very low Warning: May not capture training participants Because they are in-process surveys, they can not be used for WIA local performance July, 2001 The Charter Oak Group, LLC 15

Integrating State and Local Customer Satisfaction Efforts July, 2001 The Charter Oak Group, LLC 16

Integrating the CS Indicator into a Customer Service Strategy Federal Report State Level Quest -ions for Title I (strategic planning) Local level (program management & continuous improvement) July, 2001 CS Indicator Service data from admin records (e. g. , process and intermediate outcomes Questions for Other Programs Other service specific questions (e. g. , timeliness) The Charter Oak Group, LLC 17

A Customer Feedback Model 1. 2. 3. 4. July, 2001 Conduct state-wide telephone survey with expansion of numbers for local areas to provide data for reporting state and local satisfaction for performance Conduct local surveys post-program/service surveys using lower-cost methods for lower intensity services (e. g. , written, computer) Conduct on-site (point-of-service) surveys periodically to obtain in-process feedback Combine with other data The Charter Oak Group, LLC 18

Use the state survey and add to the sample n Advantages n n n n July, 2001 Scores at local and state level completely comparable Sufficient numbers at state level to conduct additional breakouts by program, geography, etc. Highly coordinated Avoids over-surveying customers Cost-efficient Most likely to avoid bias Allows limited benchmarking n Disadvantages n n The Charter Oak Group, LLC May limit sense of ownership at local level Does not address quality of local service processes Customization limited Higher costs than most other methods 19

Develop a local strategy with alternate form and methodology n Advantages n n July, 2001 n Highly customizable Determine quality of local services especially is using point-of-service strategy High level of buy-in from local level Develop benchmarks among local WIBs Disadvantages n n The Charter Oak Group, LLC Difficult to relate local and state results May over-survey customers Difficult to coordinate with state May be more costly depending on strategy 20

Developing questions n n n n Identify customer needs Determine system goals Determine service strategies for different customer segments to reach those goals Review service designs Identify critical points of service Identify the purpose Identify any customer segments about which you are concerned July, 2001 The Charter Oak Group, LLC 21

Developing questions, continued n n n Review questions based on the proposed analysis and report strategy Get input from partners Test questionnaire Develop collection strategy (e. g. , sampling, timing, logistics) Decide on who will use it and how it will be used Develop analysis and reporting strategies to fit audiences and their needs July, 2001 The Charter Oak Group, LLC 22

Service Quality Model: A tool for developing questions n n n n n Reliability: consistency and dependability of performance Responsiveness: willingness, readiness, timeliness of service Competence: skill and knowledge of service personnel Access: ease of contact, approachability Courtesy: politeness, respect, consideration, friendliness Communication: using understandable language, making understandable explanations Credibility: trustworthiness, believability, honesty Security: freedom from risk, danger, or doubt Understanding: knowing customer needs Tangibles: physical aspects and context of service July, 2001 The Charter Oak Group, LLC 23

Additional Question Comparison n How satisfied (1 -10) were you with each of the following: n n n Help in learning Information about occupations Training in job skills (or) Availability of staff Timeliness of service n How would you rate each of the following: n n n July, 2001 Help you received (not at all helpful–extremely helpful) Information about occupations (not useful – extremely useful) Training in job skills (not related to my job—very closely related to my job) Availability of staff (always available to never available) How would you rate staff follow-up (no follow-up to very timely follow-up) The Charter Oak Group, LLC 24

Why? n n n Asking the questions about general satisfaction and about service quality in the same way, with the same response set (1 -10 satisfaction), seems to convey that the same questions is being repeated Using distinctive scales for each makes the helps the customer make distinctions about different aspects of service Reduces the value of asking additional questions about service quality to illuminate the source of general satisfaction July, 2001 The Charter Oak Group, LLC 25

How different approaches interact n n n State-wide telephone survey provides an unbiased, consistent anchor for customer satisfaction accountability and benchmarking, focused on outcomes Point-of-service survey focuses on process Point of service can be used to determine key drivers of customer satisfaction (guide selection of questions for state-wide survey Focus groups identify details of areas needing improvement Customer comment cards identify potential major problems July, 2001 The Charter Oak Group, LLC 26

What do you need? For a strategy to be integrated it must have something that ties the pieces together Local Point-of Service July, 2001 ACSI Statewide Survey Service-specific Surveys (e. g. , workshop evaluations The Charter Oak Group, LLC 27

Using a One-Stop Perspective July, 2001 The Charter Oak Group, LLC

Wagner-Peyser (W-P) and other One-Stop partners n n Guidance for W-P participants is less clear First, there is less overlap between participants in the two programs than in the employers in the two programs Second, W-P service delivery and timing for seeking outcomes such as entered employment differ significantly between W-P and WIA Guidance is not out yet to help develop a coordinated strategy July, 2001 The Charter Oak Group, LLC 29

Other Partners n n Adult Education Vocational Rehabilitation Older Workers TANF These have all been major players in some One-Stop systems July, 2001 The Charter Oak Group, LLC 30

Value of a Multi-Partner Approach for Customer Satisfaction Surveying n n n Some customers receive services from multiple programs Many customers and most stakeholders don’t recognize program differences Comparison of different customer segments’ perception of services (e. g. , a W-P customer and WIA customer may take the same workshop and have very different perceptions) can provide new insights July, 2001 The Charter Oak Group, LLC 31

Value of a Multi-Partner Approach for Customer Satisfaction Surveying, cont. n n Focuses on service coordination among partners to achieve a common goal— their common customers’ satisfaction Provides a common language among partners July, 2001 The Charter Oak Group, LLC 32

Analysis and Reporting July, 2001 The Charter Oak Group, LLC

Analysis of Results n n n What were the results? What are some effective ways of displaying and looking at the information? What are some analyses that make the ACSI or any general index of satisfaction more useful? July, 2001 The Charter Oak Group, LLC 34

Descriptive Statistics Participant ACSISAT Questions and Index Region # of Responses Overall Satisfaction Compared to expectations Compared to ideal ACSISAT A 405 7. 86 (2. 71)* 7. 45 (2. 96) 7. 51 (2. 80) 73. 21 (29. 38) B 454 8. 03 (2. 46) 7. 30 (2. 77) 73. 23 (26. 73) C 417 7. 70 (2. 54) 7. 43 (2. 77) 7. 27 (2. 70) 71. 88 (27. 47) D 444 7. 41 (2. 54) 6. 94 (2. 9) 6. 76 (2. 79) 67. 0 (28. 42) E 456 7. 69 (2. 54) 7. 41 (2. 79) 7. 33 (2. 74) 72. 08 (27. 5) F 257 8. 08 (2. 12) 7. 68 (2. 53) 7. 30 (2. 53) 74. 37 (24. 0) 7. 44 (2. 69) *Mean with standard deviation in parenthesis July, 2001 The Charter Oak Group, LLC 35

Basic Principles for Presenting Data n n n Include the number of customers in any analysis since percentages and averages do not give a complete picture without knowing how many customers’ responses were involved Display the individual questions and the index since the responses to the individual questions tend to differ Make table titles explanatory Identify any special features, such as numbers in parenthesis, on the table Use graphics and numbers together, whenever possible July, 2001 The Charter Oak Group, LLC 36

Another Set of Distributions Participants July, 2001 Employers The Charter Oak Group, LLC 37

What do the comparisons tell us? n n n Comparisons are the core analysis for understanding Participants are generally more satisfied than employers There are many fewer highly satisfied employer customers than participant customers Both participants and employers have small but significant, extremely dissatisfied groups The most common experience for participants in high satisfaction (90’s range); the most common for employers is moderate satisfaction (70’s range) July, 2001 The Charter Oak Group, LLC 38

Other comparisons n n By program By geographic area Compared to services Compared to outcomes July, 2001 The Charter Oak Group, LLC 39

Program Comparisons Table 3: State C Participant Satisfaction Questions and ACSISAT Index by Program Title State C (N=340) Customer Group Overall Satisfaction Compared to Expectations Compared to Ideal ACSISAT State Average 7. 70 7. 43 7. 27 71. 9 Title IIA 7. 56 7. 15 7. 11 69. 9 Title III 7. 88 7. 82 7. 36 74. 1 July, 2001 The Charter Oak Group, LLC 40

Services and Satisfaction n n Full service records were not attached to either participant or employer satisfaction data so the actual quantity or specifics of service could not be determined in hours or dollars A measure was constructed to determine how many different services an individual participant and employer had received Those participants with the most services were the most satisfied Those employers with the most services were also the most satisfied July, 2001 The Charter Oak Group, LLC 41

Participant Satisfaction with Mix of Training and Job Search Assistance Region A Region B Received Job Training Received Job Search Assistance Received Job Training YES 83. 6 61. 1 YES 78. 8 66. 9 NO 76. 4 55. 7 NO 60. 4 51. 3 YES Region C Received Job Search Assistance NO Received Job Training YES July, 2001 YES NO YES 78. 6 62. 3 NO 78. 4 48. 1 The Charter Oak Group, LLC 42

Types of Service and Satisfaction: State D Employers July, 2001 The Charter Oak Group, LLC 43

Differences in Quality of Service: Key Drivers of ACSISAT for State F Employers Key Drivers Correlations Understanding needs . 385** Being responsive . 369** Knowledgeable staff . 344** Works as a partner July, 2001 . 357** The Charter Oak Group, LLC 44

Lessons from Key Driver Analysis, continued n n Ask questions to distinguish them from overall satisfaction questions Look to your goals and the focus of your service strategy n n n Fast and on-time? Comprehensive? Total solution? Highly coordinated? One-Stop? Convenient? Customized to meet your needs? Ask the questions about the services and qualities that are the focus of your program design July, 2001 The Charter Oak Group, LLC 45

Numbers, pictures and words July, 2001 The Charter Oak Group, LLC 46

Accessible formats n n Information in graphic and numeric forms Quick access to details (drill-down capability) Integration of numbers, graphics, and text Design formats to suit the primary purpose to which the audience will put the information July, 2001 The Charter Oak Group, LLC 47

Customer Satisfaction for Continuous Improvement July, 2001 The Charter Oak Group, LLC

Using CS to Improve Services n n Develop arrays of measures for different audiences Use CS is part of a comprehensive picture of service quality Develop multi-channel, accessible formats for managers/staff and for planners/policy makers Develop the capacity to use information and the procedures and staff structures to support a performance management system July, 2001 The Charter Oak Group, LLC 49

WIA Strategic and Operational Goals Cost Inputs/ resources Processes & Outputs Outcomes Impact “Intermediate Outcomes” Workload (e. g. , people served) Customer Feedback Environmental Measures (context) July, 2001 The Charter Oak Group, LLC 50

Using the Measures in a Performance Management System n n n Periodic reporting to local, One-Stop managers and staff for managing to results (e. g. , quarterly report cards) Periodic reports to program managers and policy makers for strategic decision making (e. g. , annual program, policy review) Guidance to customers for informed choice and maintaining customer focus (e. g. , setting accurate customer expectations) July, 2001 The Charter Oak Group, LLC 51

Implementation n n Develop baseline implementation Develop the performance system with partners--an integrative, teambuilding activity Build a system that supports various levels of comparison (e. g. , negotiated levels, over time, best-practices, national benchmarks) Make the system a driver of continuous improvement Build capacity and train, train July, 2001 The Charter Oak Group, LLC 52

No one way to monitor, assess, and improve customer satisfaction Staff Training Performance Measurement Customer Surveys Needs Assessment Other customer feedback July, 2001 Customer Satisfaction Malcolm Baldrige Voice of the Customer, Service by Design The Charter Oak Group, LLC 53