Assessing Nonresponse Bias and Measurement Error in Estimates

Assessing Nonresponse Bias and Measurement Error in Estimates of Employment John Dixon Clyde Tucker Polly Phipps Bureau of Labor Statistics any opinions expressed in this paper are those of the authors and do not constitute policy of the Bureau of Labor Statistics

Types of Nonresponse • Ignorable: Conditioning on known auxiliary variables some bias may be eliminated from estimates—accomplished by making weighting adjustments. • Nonignorable: Bias that cannot be eliminated by conditioning on auxiliary variables—remains in estimates and contributes to mean squared error • Weighting adjustments may be of only limited utility

Previous Research • Household surveys – Haraldsen, et al. 1999; Keeter, et al. 2000; Curtin, et al. 2000 found little nonresponse bias – Brick and Bose (2001) found weighting was effective – Most results confined to response rates between 40% and 70%

Previous Research • Establishment Surveys – Nonresponding units can have more effect than in household surveys – Tomaskovic-Devey, et al. (1994) found differential nonresponse by size, industry, and profitability of firm – Most research focused on characterizing nonrespondents (Hidiroglou, et al. 1993 and Phipps et al. 2007, adjusting for nonresponse (Sommers, et al. 2004) or understanding why establishments don’t respond (Mc. Carthy, et al. 1999 and Mc. Carthy and Beckler 2000) – Copeland (2003), in contrast, looked at the bias in early estimates that resulted from late responders in the Current Employment Statistics Program (CES) – Tucker, et al. (2005) began an examination of bias in CES using data from the QCEW (State UI files). This paper is a more thorough report on that work.

The CES • Collects employment, hours and earnings monthly from a current sample of over 300, 000 establishments • Tracks the gains and losses in jobs in various sectors of the economy • In this paper, nonresponse bias work on this survey focuses on estimating bias for establishment subpopulations with different patterns of nonresponse using data from the 2003 CES and QCEW (over 400, 000 respondents)

Nonresponse Error for Sample Mean In simplest terms OR Respondent Mean = Full Sample Mean + (Nonresponse Rate)*(Respondent Mean – Nonrespondent Mean)

Theory • Levels of bias will differ by subpopulations • The difference between respondent and nonrespondent estimates will be greatest on either end of the nonresponse continuum, but potential bias greatest when response rates are low • Bias in business surveys may be greatest in the Services sector

The Current Study of Nonresponse Bias in the CES • Because this work is theoretically driven, the paper just examines the difference in employment of respondents and nonrespondents but not the ultimate effect on estimates after accounting for response rate. • Furthermore, the CES benchmarks the estimates to the QCEW (the data used in this study) using estimation cells based on industry and region. • Thus, the eventual aggregate bias should be small.

CES Response Rates by NAICS Categories

Measurement error • Administrative data for employment counts is thought to be less error-prone than survey data. – Errors in the unemployment insurance counts have consequences for taxes or fines. – Surveys are often handled by less experienced staff. • Differences between the survey and the administrative data could be due to – definitional differences (e. g. : students usually aren’t counted in the unemployment insurance counts, but may be in the payroll counts) – reporting period

Measurement error (continued) Both the administrative and the survey data vary from month to month, but if the survey has more measurement error, then the variability of the survey data relative to the administrative data could be used as a rough indicator of measurement error.

Distribution of relative variance

Measurement error measures and closing missing mix no All lrvdces mlpces 0. 0938 0. 1807 0. 2877 0. 1695 0. 1709 0. 6552 0. 6702 0. 4665 0. 1476 0. 4911 0. 4325 0. 3617 close 1 2 3 All -2. 006 -1. 873 -1. 798 -1. 903 -2. 585 -2. 237 -2. 118 -2. 374 -2. 417 -2. 114 -1. 909 -2. 211

Estimate of Biases (nonresponse and measurement error) • Using the most recent employment reports in the QCEW (not CES) for both responders and nonresponders • Compare the employment reports for respondents to that for nonrespondents • Compare the variability of estimates within companies for the CES survey relative to the variability of the QCEW as an estimate of measurement bias. • Results presented are not weighted by probability of selection, but weighted results show similar patterns

Nonresponse and Measurement Bias Survey Administrative Median (N) Missingness Always missing Mix No missing 08. 84 (294410) 12. 92 (144203) 12. 84 (144512) 15. 65 (222228) 15. 47 (221121)

Quantile Regression • Bias analysis performed at the establishment level on subpopulations defined by size and industry • Testing for the difference in employment between CES responders and nonresponders. Y=a+Bx+e where x is an indicator of nonresponse (essentially a t-test). • Since size of firm is theorized to relate to nonresponse, the coefficients relating nonresponse to employment is likely to be different for different size firms. • Quantile regression examines the coefficients for different quantiles of the distribution of the sizes of firms. • Since industries can be expected to have different patterns, the quantile regressions are done by industry group.

Interpretation of Quantile Regression of size on bias • At the micro level, the quantile regression shows the coefficients relating nonresponse (coded 0) to the size of firm. Each point on the curve is a regression relating nonresponse to size conditional on the rest of the distribution. The skewness affects the standard errors, so a stabilizing transformation is needed. This can be seen in the box & whisker plot to the right.

Distribution of size and the quantile regression curve

Employment and Measurement error

Measures of Measurement error

Measures of Nonresponse

Interpretation of a Log Transformation • Using the log of size, the proportionate effect is greater for smaller firms. Since the coefficients are based on linear models, the transformation makes the distribution of responders and nonresponders more reasonable, as seen in the box & whisker plot to the right. • The quantile regression curve shows smaller firms have proportionately more bias than larger firms.

Quantile regression using the log of size.

Industry patterns • In almost all cases, there is some significant positive bias. • The most common pattern of the quantile curves is rapidly accelerating coefficients with increasing establishment size for almost half the industries. • Another pattern is flat coefficients until past the middle of the size distribution followed by accelerating coefficients. • A third pattern is a relatively gradual increase in the coefficients to an asymptote.

Retail trade

Accelerating patterns • The Agriculture, Forestry, Fishing and Hunting industries typify the rapidly accelerating pattern. The logged values show decreasing coefficients. The statistical significance shown by the confidence intervals varied.

Accelerating pattern industries • Agriculture; mining; metal manufacturing; transportation and warehousing; information; finance and insurance; professional, scientific and technical services; administrative services and waste management; and accommodation and food services.

Late accelerating pattern • The Health Care and Social Assistance industries typified the late accelerating pattern. While health care had positive coefficients, The real estate industry started negative and only became positive at the 80 th quantile. • While most other patterns of the logged values showed decline, these showed an increase.

Late Accelerating Industries • Health Care and social assistance; retail trade; real estate, rental and leasing; education services; and other services

Asymptote pattern • The construction industry typifies the asymptote pattern of gradually increasing coefficients until the highest quantiles. • The logged values showed an early decline to an asymptote.

Asymptote industries • The construction; food manufacturing; wood and mineral manufacturing; wholesale trade; and arts, entertainment and recreation showed the asymptote pattern. • They may have a critical mass, where once the size reaches the 90 th quantile, there is a more uniform participation rate.

Utilities • The utilities industry shows a relatively flat curve of coefficients. • The logged values were higher for small firms.

Management of Companies and Enterprises • The coefficients is virtually flat until the highest quantiles and then turns decidedly. • The coefficients from the logged values linearly declined.

Measurement error and nonresponse

Flat patterns • Measurement error stayed mostly constant over the likelihood of nonresponse for Agriculture, Arts, Information, and Transportation

Declining pattern • Only Real-estate had a declining pattern of measurement error over the likelihood of nonresponse.

Summary • Quantile regression is a useful technique for studying nonresponse bias • The patterns of differences between industries varied • The most common pattern indicated efforts to adjust for nonresponse should be spent on larger firms to better estimate overall changes in employment • Yet, proportionate differences were higher for smaller firms • The patterns weren’t related to the nonresponse rate for the industry • Measurement error relates to smaller differences than nonresponse bias. Mixed responders had lower measurement error, contrary to expectations. • Most industries had an increase in measurement error as the probability of nonresponse increased.

Assessing Nonresponse Bias and Measurement Error in Estimates of Employment John Dixon dixon. j@bls. gov Clyde Tucker tucker_p@bls. gov Polly Phipps phipps_p@bls. gov Bureau of Labor Statistics