Automatically Generated Quality Control Tables and Quality Improvement

Automatically Generated Quality Control Tables and Quality Improvement Programs Justin D. Nguyen Carma R. Hogue 06/2018

Outline § Overview of the Current Quality Audit Program (QAP) § Challenges § Goals and Achievement of Quality Processing § Procedures for Improving Quality and Controlling Costs § Future Steps

Overview of the Current Quality Audit Programs (QAP) – Results of Current QAP Five-Year Cycle § Over 87% of all audited programs found to be largely compliant § Only one program with no compliance issues and no recommendations for improvement § 38 standards found noncompliant across all 31 audited programs

Challenges What caused these issues? § Program areas are not realizing certain activities should be documented § Lack of accurate and current program documentation led to managers having difficulty finding copies of documents they believed existed § Document management system software by outside vendor

Ultimate Goal of Quality Improvement § Assure Data Users that Our Products Are of the Highest Quality • the public • sponsors • joint partners • other customers § Continually Improve All Processes • automate quality control systems • conduct research and program evaluation § Develop Quality Control Procedures to Monitor and Evaluate All Processes

Continuous Quality Improvement § Automate Quality Control Tables for Improvement: Examples: • nonresponse bias analysis • total quantity response rate • unit response rate § Monitor Progress Results Examples: • monitor survey response rates • monitor complete, incomplete, and follow-up interview cases § Apply Tableplots and Big Data Methods

Continuous Quality Improvement (cont’d) § Example of monitoring progress results

Continuous Quality Improvement (cont’d) § Applying Quality Statistical Standards throughout the Survey Life Cycle • Planning and Development • Collecting and Acquiring Data • Capturing and Processing Data • Producing Estimates and Measure • Preparing Report • Releasing Information Products

Continuous Quality Improvement (cont’d)

Procedures for Improving Quality

Procedures for Improving Quality (cont’d) § Apply Tableplots and Big Data Methods: • build more “adaptive” editing process • reduce biases during the edit process • reduce cost and improve timeliness

Procedures for Improving Quality (cont’d) § Tableplot of Services Annual Survey Data through March 31, 2017

Procedures for Improving Quality (cont’d) § Tableplot of Services Annual Survey Data through June 30, 2017

Future Steps § Automate quality control procedures and implementing quality processes in the Survey Life Cycle § Build self-documenting systems to ensure documentation matches what system and programs are doing § Build quality control measures for emerging methodologies § Automate editing as much as possible to ensure that all editing processes are repeatable § Make use of Big Data and tableplots to selectively edit large datasets § Make use of paradata and key performance indicators to improve data quality

Questions and Contact Information § Questions § Contact Information Justin Nguyen, Staff Chief Justin. D. Nguyen@Census. Gov Carma Hogue, Director of Research & Methodology for Econ. Programs Carma. Ray. Hogue@Census. Gov