Using System Paradata to Target and Evaluate Data

  • Slides: 32
Download presentation
Using System Paradata to Target and Evaluate Data Collection Operations Stephanie Coffey WSS Seminar

Using System Paradata to Target and Evaluate Data Collection Operations Stephanie Coffey WSS Seminar 4/18/2016 U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU

Overview • National Survey of College Graduates (NSCG) Survey Background • System Paradata Collected

Overview • National Survey of College Graduates (NSCG) Survey Background • System Paradata Collected • 2 Examples of Paradata Use – Web instrument performance – Intelligent Mail Barcoding • Wrap-Up U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 2

National Survey of College Graduates • Sponsored by NCSES at NSF • U. S.

National Survey of College Graduates • Sponsored by NCSES at NSF • U. S. college-educated science & engineering population – Education, demographic and employment info for S&E degrees and occupations • Recent college graduates • Women, minorities, persons with disabilities • Longitudinal survey with four rotating panels – New cohort is selected out of ACS for 1 st interview – Three follow-up interviews (every 2 -3 years) – Total sample size is approximately 135, 000 cases U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 3

National Survey of College Graduates Standard Data Collection Path (New Cohort) Weeks 0 –

National Survey of College Graduates Standard Data Collection Path (New Cohort) Weeks 0 – 6 Initial Invite Phase Weeks 7 - 11 1 st Reminder Phase Weeks 12 - 17 Nonresp. FU Phase Prenotice Web Invites #1&2 1 st Paper Q’naire Web Invite #3 CATI NRFU Begins Web Invite #4 Weeks 23 - end Final Reminder Phase Weeks 18 - 22 2 nd Reminder Phase Web Invite #6 2 nd Paper Q’naire Web Invite #5 U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU

National Survey of College Graduates Types of Paradata Collected from Major Operations • Outgoing

National Survey of College Graduates Types of Paradata Collected from Major Operations • Outgoing Mailings: Mailed Web Invites, Reminders, Questionnaires – Census – National Processing Center (NPC) • Mail-Out Date • Undeliverable-as-Addressed (UAA) Status/Reason Upon Check-In at NPC – USPS – Integrated Postal Tracking Service (IPTS) – New in 2015 • First Scan into Postal System / Last Scan out of Postal System (out for delivery) • UAA Status/Reason Upon USPS Determination • Incoming Mailings: Return Paper Questionnaires – Census – NPC • Return Date Upon Receipt and Check-In at NPC – USPS – IPTS • First Scan Back into Postal System / Last Scan out of Postal System U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 5

National Survey of College Graduates Types of Paradata Collected from Major Operations • Web

National Survey of College Graduates Types of Paradata Collected from Major Operations • Web Access from Respondent – Server-side Paradata • Log-in Attempt Date/Time • Log-in Success Status • Time-in-Instrument • Log-out Date/Time • Submission Date/Time • Outgoing Telephone Contact Attempt – Centralized Call Center Case Management • Call Attempt Date/Time • Time in Instrument • Outcome of Call Attempt • Other Operations Not Included (Email Reminders, Incoming Telephone) U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 6

Web Instrument Performance Event U. S. Department of Commerce Economics and Statistics Administration U.

Web Instrument Performance Event U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU

2013 Data Collection • Web is initial mode offered in NSCG new cohort –

2013 Data Collection • Web is initial mode offered in NSCG new cohort – Web first used as production mode in 2013 – Prior to 2013, web experiments, mode preference experiments • NSCG achieves about 30% response rate in the first month – NSCG has 6 month data collection • First invites with usernames and passwords sent on 2/20 – Arrival expected between 2/22 -2/24 (Sat or Mon) U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU

Web Interruption • Afternoon of 2/25 – Morning 2/26 – External DHS simulated attack

Web Interruption • Afternoon of 2/25 – Morning 2/26 – External DHS simulated attack to check data security (unscheduled, unknown by Census) – IT teams at Census: maintenance & applying updates • Resulted in performance issues in the web instrument – No advance warning of external simulations – NSCG was informed by respondents calling TQA line: • trouble logging in, delays in navigating through survey • Question: Can we figure out who was affected? U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU

Evidence in Server-Side Paradata service type id time (seconds) type 2 NSCG paradata 10800084

Evidence in Server-Side Paradata service type id time (seconds) type 2 NSCG paradata 10800084 1361842005 login NSCG paradata 10800084 1361845429 submitted NSCG paradata 10800084 1361845466 logout NSCG paradata 10800159 1361843035 login NSCG paradata 10800159 1361846593 submitted NSCG paradata 10800159 1361846627 logout NSCG paradata 10800258 1361847610 login NSCG paradata 10800258 1361847625 login NSCG paradata 10800258 1361847959 login NSCG paradata 10802031 1361840282 failed_login NSCG paradata 10802031 1361840295 failed_login NSCG paradata 10802031 1361840349 failed_login U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 57 min 60 min 15 sec 5 min 13 sec <1 min

Evidence in Server-Side Paradata service type id time (seconds) type 2 NSCG paradata 10800084

Evidence in Server-Side Paradata service type id time (seconds) type 2 NSCG paradata 10800084 1361842005 login NSCG paradata 10800084 1361845429 submitted NSCG paradata 10800084 1361845466 logout NSCG paradata 10800159 1361843035 login NSCG paradata 10800159 1361846593 submitted NSCG paradata 10800159 1361846627 logout NSCG paradata 10800258 1361847610 login NSCG paradata 10800258 1361847625 login NSCG paradata 10800258 1361847959 login NSCG paradata 10802031 1361840282 failed_login NSCG paradata 10802031 1361840295 failed_login NSCG paradata 10802031 1361840349 failed_login U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 57 min Patient respondents! 60 min 15 sec 5 min 13 sec <1 min

Evidence in Server-Side Paradata service type id time (seconds) type 2 NSCG paradata 10800084

Evidence in Server-Side Paradata service type id time (seconds) type 2 NSCG paradata 10800084 1361842005 login NSCG paradata 10800084 1361845429 submitted NSCG paradata 10800084 1361845466 logout NSCG paradata 10800159 1361843035 login NSCG paradata 10800159 1361846593 submitted NSCG paradata 10800159 1361846627 logout NSCG paradata 10800258 1361847610 login NSCG paradata 10800258 1361847625 login NSCG paradata 10800258 1361847959 login NSCG paradata 10802031 1361840282 failed_login NSCG paradata 10802031 1361840295 failed_login NSCG paradata 10802031 1361840349 failed_login U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 57 min 60 min 2+ Logins in Short Time 15 sec 5 min 13 sec <1 min

Evidence in Server-Side Paradata service type id time (seconds) type 2 NSCG paradata 10800084

Evidence in Server-Side Paradata service type id time (seconds) type 2 NSCG paradata 10800084 1361842005 login NSCG paradata 10800084 1361845429 submitted NSCG paradata 10800084 1361845466 logout NSCG paradata 10800159 1361843035 login NSCG paradata 10800159 1361846593 submitted NSCG paradata 10800159 1361846627 logout NSCG paradata 10800258 1361847610 login NSCG paradata 10800258 1361847625 login NSCG paradata 10800258 1361847959 login NSCG paradata 10802031 1361840282 failed_login NSCG paradata 10802031 1361840295 failed_login NSCG paradata 10802031 1361840349 failed_login U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 57 min 60 min 15 sec 5 min Can’t Successfully Log In 13 sec <1 min

Operational Questions • Major Concern: Burden / Respondent Frustration – Time in instrument •

Operational Questions • Major Concern: Burden / Respondent Frustration – Time in instrument • How long did individuals spend logged in? – Server breakoffs • Will sample persons come back to respond? • Examine Login Events – – Login/Submit = 1 event Login/Submit/Logout = 1 event Login/Logout = 1 event If there was no Submission or Logout: • Login alone = Server Breakoff • Failed login = Failed Login U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU

Increase in the mean time-in-instrument U. S. Department of Commerce Economics and Statistics Administration

Increase in the mean time-in-instrument U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU

Increase in the mean time-in-instrument More server breakoffs than submissions! U. S. Department of

Increase in the mean time-in-instrument More server breakoffs than submissions! U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU

Mean time in instrument nearly back to average Server breakoffs still higher than expected

Mean time in instrument nearly back to average Server breakoffs still higher than expected U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 17

Action & Results • Determined time period of web issue: 6 pm – 2

Action & Results • Determined time period of web issue: 6 pm – 2 am • Obtained approval to send special letter to affected cases – Apology and reminder of sample person’s importance to data quality – Turnaround time: 3 business days from analysis to mailing • Sent letter to all breakoff cases: – Not an experiment • Response rate of affected cases same as unaffected cases a month after mailing – Would this be the same if this issue occurred late in data collection? (Early respondents = higher engagement? ) • Encouraged that we could be responsive to a new issue U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU

Intelligent Mail Barcoding U. S. Department of Commerce Economics and Statistics Administration U. S.

Intelligent Mail Barcoding U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU

Mailing Operation in NSCG • NSCG heavily relies upon mailing operations – – –

Mailing Operation in NSCG • NSCG heavily relies upon mailing operations – – – Pre-notice Letter Web invites Paper Questionnaires Reminder Letters & Postcards Prepaid Incentives • A respondent receiving all mailings in sequence: 11 mailings (plus any requested remails) • Historically a low-information operation – Was the address mailable? – Did we receive something in return? – Only have date of check-in at NPC U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 20

Integrated Postal System Tracking • USPS offers IPTS Data • Tracks data through each

Integrated Postal System Tracking • USPS offers IPTS Data • Tracks data through each step of postal delivery – First scan at first post office – Every transfer scan – Last scan before delivery – Scan to reroute due to UAAs – Reasons for UAAs U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 21

IPTS for NSCG • Two points of interest in the NSCG – How much

IPTS for NSCG • Two points of interest in the NSCG – How much earlier can we know about UAAs? • Can send sample persons to locating earlier • Can save cost through not mailing future mailings – How much earlier can we know about return questionnaires? • Can reduce burden by not contacting likely respondents • Can reduce cost by not contacting likely respondents – Discuss UAA timeliness today U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 22

Implementation of IPTS • Add barcodes to mailing packages – Need outgoing barcodes to

Implementation of IPTS • Add barcodes to mailing packages – Need outgoing barcodes to track progress and UAAs (letters and questionnaires) – Need return barcodes to track progress of return questionnaires (questionnaires only) • Redesign envelope to accommodate barcodes on printed letters and questionnaires – Ensure barcode can be scanned by USPS • Provide messaging on questionnaire return envelope to insert questionnaire correctly – Ensure barcode is displayed U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 23

Lag in UAA Identification • UAAs can be identified at various points – Delivery

Lag in UAA Identification • UAAs can be identified at various points – Delivery Point Verification before mailing (not mailed) – During any scan through postal system – When post office attempts to deliver mail • After UAA is identified – Mail package is rerouted through USPS back to NPC – Upon arrival at NPC, UAAs are checked in as resources allow • Report in Unified Tracking System (UTS) to track – Each mail piece and its mail date – When the UAA was reported by USPS – When the UAA was checked in at NPC U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 24

Lag in UAA Identification List of web invite letters and reminder mailings sent during

Lag in UAA Identification List of web invite letters and reminder mailings sent during Weeks 1 - 13 U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 25

Lag in UAA Identification Number of UAAs recorded by the USPS For each data

Lag in UAA Identification Number of UAAs recorded by the USPS For each data collection operation. U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 26

Lag in UAA Identification Number & percentage of USPS records that also have a

Lag in UAA Identification Number & percentage of USPS records that also have a check-in record at NPC. U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 27

Lag in UAA Identification Question: How long is the lag from when we could

Lag in UAA Identification Question: How long is the lag from when we could see this information in IPTS to when we see it in checkin records at NPC? Number & percentage of USPS records that also have a check-in record at NPC. U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 28

Lag in UAA Identification Average, median, and max lag between USPS and NPC U.

Lag in UAA Identification Average, median, and max lag between USPS and NPC U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 29

Action & Results • Clear that data from the USPS scanning – More timely

Action & Results • Clear that data from the USPS scanning – More timely – Cost neutral, if not cheaper • Still some questions about UAA data – Understand lag between mailing and USPS UAA – Better understand NPC check-in process • Records we cannot currently identify • Monitor and use USPS UAA data in 2017 • Implement return questionnaire scan data in 2017 – Monitor and evaluate – Place potential respondents on hold in other modes until we determine validity of complete U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU

Summary • High volume of system-generated paradata – – Need to know what is

Summary • High volume of system-generated paradata – – Need to know what is available Need to know who provides it so you can ask Possible operational changes to collect or use (envelopes) May need new systems to manage or track paradata • Can be monitored to improve data collection – Efficiency and timeliness • Might be able to help answer research questions – May need to combine data across modes to answer research questions – Important to keep in mind limitations of system paradata U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 31

Contact Information stephanie. coffey@census. gov U. S. Department of Commerce Economics and Statistics Administration

Contact Information stephanie. coffey@census. gov U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU