Preliminary work Do not cite without permission The

  • Slides: 15
Download presentation
Preliminary work. Do not cite without permission. The Implications of Misreporting for Longitudinal Studies

Preliminary work. Do not cite without permission. The Implications of Misreporting for Longitudinal Studies of SNAP Brian Stacy Erik Scherpf The findings and conclusions in this preliminary publication have not been formally disseminated by the USDA and should not be construed to represent any agency determination or policy. This research was supported in part by the intramural research program of the USDA, Economic Research Service. 1

Introduction • Researchers often rely on survey datasets to examine SNAP • Several studies

Introduction • Researchers often rely on survey datasets to examine SNAP • Several studies have noted misreporting of SNAP participation • Many researchers use longitudinal estimators of SNAP – To study dynamics of SNAP participation – To possibly help with selection bias • However, longitudinal methods can exacerbate bias from misreporting 2

Objectives & Findings • This paper examines the misreporting of SNAP on causal effects

Objectives & Findings • This paper examines the misreporting of SNAP on causal effects – Particularly focused on longitudinal estimators • Link SNAP Administrative Data to CPS ASEC survey records – Create panel dataset for individuals • Three main findings: 1. Find evidence of severe errors in changes in reported SNAP participation • Of those switching onto SNAP from one period to next based on surveys, 86% are false 2. Application: Effects of SNAP on work effort • Find severe bias in longitudinal estimators 3. Find evidence that some methods to correct estimators for misreporting perform well • Conditional distribution method (CDM) works well for first-difference OLS estimator • CDM can be applied by researchers without access to admin records 3

Background • Meyer, Mittag, & Goerge (2018) find substantial under-reporting of SNAP in CPS

Background • Meyer, Mittag, & Goerge (2018) find substantial under-reporting of SNAP in CPS ASEC survey in cross-section – Close to 50% of SNAP participants don’t report receipt in CPS ASEC • Little literature on errors in changes in reported SNAP participation over time, which identify many longitudinal estimators of SNAP effects • Paper is similar to Bound & Krueger (1991); Bound, Brown, Duncan, Rogers (1994); Freeman (1984) – These papers examine effects on longitudinal estimators of misreporting of earnings and union participation 4

Data • Combine SNAP administrative records on SNAP receipt to CPS ASEC survey –

Data • Combine SNAP administrative records on SNAP receipt to CPS ASEC survey – We assume admin reports are “true”, mismatch with survey is false report – Links based on PIK created by Census Bureau staff • Can create two-period panel datasets using longitudinal structure of CPS ASEC – Can directly assess errors in changes in reported SNAP participation • Create linkages for nine anonymous states from 2005 -2015 • Unfortunately, not all states have data available for all years. – From 2010 -2013, states make up 32% of individual SNAP caseload – In 2005, when fewest states were available, made up 5% of caseload 5

Results on Misreporting in Cross-Section • Using our linked SNAP admin records and CPS

Results on Misreporting in Cross-Section • Using our linked SNAP admin records and CPS ASEC records, examine misreporting • Cross-sectional Misreporting: – Find under-reporting rate of 43. 9% – Find over-reporting rate of 2. 9% – Find evidence that misreporting related to individual characteristics • • Those who work are more likely to under-report SNAP Females more likely to under-report Those with more education more likely to under-report Meyer et al. (2018) found higher income individuals more likely to under-report 6

Results on Misreporting Longitudinally • Using linked data, also examine changes in reported SNAP

Results on Misreporting Longitudinally • Using linked data, also examine changes in reported SNAP participation • Errors in changes in reported SNAP participation: – Four groups: 1. Switch off SNAP: 86% of survey based changes are false • Most of these people were on SNAP in both periods or off SNAP both periods 2. On SNAP – Both Periods: 27. 5% of survey reports false • Even though over-reporting rare, off SNAP group makes up 88. 5% of records 3. Off SNAP – Both Periods: 4. 7% of survey reports false 4. Switch on SNAP: 87% of survey based changes are false • Again, most were either off SNAP or on SNAP both periods • Find errors in changes related to characteristics – Employment, age, race 7

Empirical Application: Effects of SNAP on Work • 8

Empirical Application: Effects of SNAP on Work • 8

Table 4. Regression Estimates of Effects of SNAP on Work and Hours Worked using

Table 4. Regression Estimates of Effects of SNAP on Work and Hours Worked using Reported SNAP Measure from CPS ASEC versus Admin Verified SNAP Measure. Results from 16 Separate Regressions. Estimated Effects Cross Sectional w/ OLS: Reported SNAP Admin Verified SNAP First Differences w/ OLS: Reported SNAP Admin Verified SNAP Covariates (1) (2) Worked Past Year -0. 291*** (0. 007) -0. 286*** (0. 006) -0. 040*** (0. 010) 0. 050*** (0. 015) NO -0. 222*** (0. 007) -0. 221*** (0. 006) -0. 039*** (0. 010) 0. 048*** (0. 015) YES (3) Hours Worked (4) Hours Worked -13. 82*** (0. 257) -13. 60*** (0. 238) -2. 373*** (0. 413) 1. 769*** (0. 607) NO -9. 912*** (0. 267) -9. 556*** (0. 251) -2. 330*** (0. 419) 1. 710*** (0. 612) YES Covariates include: age, gender, race, education, marital status, unemployment rate. Robust standard errors in parenthesis. *** p<0. 01, ** p<0. 05, * p<0. 1 9

Solutions for Researchers • Find Conditional Distribution Methods work well (Mittag, 2018) • Can

Solutions for Researchers • Find Conditional Distribution Methods work well (Mittag, 2018) • Can be used by researchers using only public use data – No admin data link required • Two-Step approach: 1. Researchers with access to admin data publish estimates of conditional distribution of “true” SNAP participation conditional on covariates in public use data 2. Researchers using public use data then impute SNAP participation based on these covariates – Approach similar to multiple imputation • Key is that imputed SNAP participation has same covariance structure as “true” SNAP participation measure • Assumptions: 1. 2. Conditional distribution must be same for linked data and public use data Parametric model used to estimate conditional distribution must be properly specified 10

Table 5. Regression Estimates of First Differences in Work and Hours Worked on First

Table 5. Regression Estimates of First Differences in Work and Hours Worked on First Differences in SNAP on using OLS with Reported SNAP Measure, OLS with Admin Verified Measure, and the Mittag (2018) Conditional Distribution Method (1) (2) (3) (4) Estimated Effects Worked Past Hours Worked Year Worked First Differences w/ OLS: Reported SNAP -0. 040*** -0. 039*** -2. 373*** -2. 330*** (0. 010) (0. 413) (0. 419) Admin Verified SNAP 0. 050*** 0. 048*** 1. 769*** 1. 710*** (0. 015) (0. 607) (0. 612) Conditional Distribution Method 0. 054 *** 0. 047 *** 1. 947*** 1. 698*** (0. 006) (0. 003) (0. 288) (0. 126) Covariates NO YES Observations 36, 000 Bootstrap standard errors in parenthesis from 100 bootstrap replications. In each bootstrap replication, both the multinomial logit estimation and regression estimation steps are performed. Covariates include controls for age, education, race, marital status, and the county unemployment rate. *** p<0. 01, ** p<0. 05, * p<0. 1 11

Conclusions • • Many researcher make use of survey data and apply longitudinal methods

Conclusions • • Many researcher make use of survey data and apply longitudinal methods CPS ASEC supplement frequently used and has substantial misreporting Find errors in changes in SNAP participation worse than cross-section We characterize bias for cross-sectional and longitudinal estimators – Find severe bias in longitudinal estimator of effects of SNAP on Work • Solutions: – Conditional Distribution Methods seem promising • Researchers with access to admin linkages may wish to publish conditional distribution estimates 12

Questions? 13

Questions? 13

Misreporting and Bias • 14

Misreporting and Bias • 14

Non-random Misreporting and Bias • 15

Non-random Misreporting and Bias • 15