U S Bureau of Labor Statistics Why Should
U. S. Bureau of Labor Statistics Why Should the Federal Government Care? Clyde Tucker Bureau of Labor Statistics
Reasons Not to Care • Getting the numbers out on time is how we’re judged • Quality is fine but it’s of secondary importance • Nobody can know the truth anyway
Destroying Myths • Obviously, a lot of people are interested in quality • Actually, they want everything—speed and accuracy • Maybe nobody knows the right answer, but everyone thinks they do
Problems to Overcome • Skill level of the Federal workforce • Multiple measures with different kinds of errors • Skepticism of program managers • Practical concerns – Time lag – Method of reporting
Skills of the Workforce • New field and rare skill • Hard to get and keep talent • Focus of much of research on production issues (response rates, question wording, imputation, variance, etc. ) • Advances in field likely to come from outside
Multiple Measures With Different Errors • Programs have multiple outputs • There are multiple programs • Broad range of error properties and methods of analysis • Agreement on where to start can be difficult
Skepticism of Program Managers • Haven’t given them much up to now • Placing faith in a somewhat incoherent field—nonsampling error research • Don’t you think they would report the right answers if they had them • Will we do more harm than good—subjecting programs to greater criticism • Can the measures of nonsampling error be trusted (What are their error properties? ) • What’s it going to cost and is it worth it—cost benefit analysis needed • Even if cost-effective , there still may not be the money • Even if we had good measures of these errors, can we use them to improve methods (e. g. , continuous improvement)
Practical Concerns • Time lag – Unless the measures of nonsampling errors, like variances, are produced at the same time as the estimates they will not be very useful • Method of reporting – Will MSE now be used? – What about the asymmetrical nature of bias? – What exactly are we talking about, anyway—if we knew the right answer we’d report it and not a measure of error – We’re likely to have different ways of measuring different errors, not just one, like variance – Do these become our performance measures?
So Is the Federal Government Doing Something Already • Agencies now have information quality guidelines specifying procedures for addressing complaints about the quality of estimates • OMB shortly will be releasing new survey standards, including guidance on nonresponse bias • Some agencies already have their own standards • Some surveys have a discussion of nonsampling errors in their documentation (e. g. , Chapter 16 of CPS Technical Paper 63) • The FCSM interagency subcommittees on both household and establishment survey nonresponse have workgroups on nonresponse bias
What the Federal Government Is Doing Continued • Research – Hansen, et al. (1951, 1964, 1967) – Bailar (1983) – Linacre and Trewin (1989, 1993) – Groves and Couper (1998) – Other Census match studies – IRS and SSA studies
Research on BLS Surveys – BLS Consumer Expenditure Survey --Silberstein (1989); Tucker (1992); Kojetin and Jerstad (1997); Schober and Conrad (1997); Rips, et al. (2001); Tucker, et al. (2003) – Current Population Survey– Martin, Campanelli, and Fay (1991); Cohany, Polivka, and Rothgeb (1994); Polivka and Miller (1998); Kojetin and Mullin (1995); Biemer and Bushery (2000); Dixon (2004); section of Survey Methodology (December 2004) – Current Employment Statistics Program—Phipps and Tupek (1991); Copeland (2004); Tucker (2005) – American Time Use Survey—Fricker (2005? )
Looking to the Future • Development of expertise in the field • Long-term commitment of resources to developing measures of nonsampling error • Collaborating with those outside government • Figure out how to report errors and use them • Evaluate the quality of the measures of nonsampling error • Conduct cost-benefit analyses of their value
- Slides: 12