Using System Paradata to Target and Evaluate Data
- Slides: 32
Using System Paradata to Target and Evaluate Data Collection Operations Stephanie Coffey WSS Seminar 4/18/2016 U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU
Overview • National Survey of College Graduates (NSCG) Survey Background • System Paradata Collected • 2 Examples of Paradata Use – Web instrument performance – Intelligent Mail Barcoding • Wrap-Up U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 2
National Survey of College Graduates • Sponsored by NCSES at NSF • U. S. college-educated science & engineering population – Education, demographic and employment info for S&E degrees and occupations • Recent college graduates • Women, minorities, persons with disabilities • Longitudinal survey with four rotating panels – New cohort is selected out of ACS for 1 st interview – Three follow-up interviews (every 2 -3 years) – Total sample size is approximately 135, 000 cases U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 3
National Survey of College Graduates Standard Data Collection Path (New Cohort) Weeks 0 – 6 Initial Invite Phase Weeks 7 - 11 1 st Reminder Phase Weeks 12 - 17 Nonresp. FU Phase Prenotice Web Invites #1&2 1 st Paper Q’naire Web Invite #3 CATI NRFU Begins Web Invite #4 Weeks 23 - end Final Reminder Phase Weeks 18 - 22 2 nd Reminder Phase Web Invite #6 2 nd Paper Q’naire Web Invite #5 U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU
National Survey of College Graduates Types of Paradata Collected from Major Operations • Outgoing Mailings: Mailed Web Invites, Reminders, Questionnaires – Census – National Processing Center (NPC) • Mail-Out Date • Undeliverable-as-Addressed (UAA) Status/Reason Upon Check-In at NPC – USPS – Integrated Postal Tracking Service (IPTS) – New in 2015 • First Scan into Postal System / Last Scan out of Postal System (out for delivery) • UAA Status/Reason Upon USPS Determination • Incoming Mailings: Return Paper Questionnaires – Census – NPC • Return Date Upon Receipt and Check-In at NPC – USPS – IPTS • First Scan Back into Postal System / Last Scan out of Postal System U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 5
National Survey of College Graduates Types of Paradata Collected from Major Operations • Web Access from Respondent – Server-side Paradata • Log-in Attempt Date/Time • Log-in Success Status • Time-in-Instrument • Log-out Date/Time • Submission Date/Time • Outgoing Telephone Contact Attempt – Centralized Call Center Case Management • Call Attempt Date/Time • Time in Instrument • Outcome of Call Attempt • Other Operations Not Included (Email Reminders, Incoming Telephone) U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 6
Web Instrument Performance Event U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU
2013 Data Collection • Web is initial mode offered in NSCG new cohort – Web first used as production mode in 2013 – Prior to 2013, web experiments, mode preference experiments • NSCG achieves about 30% response rate in the first month – NSCG has 6 month data collection • First invites with usernames and passwords sent on 2/20 – Arrival expected between 2/22 -2/24 (Sat or Mon) U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU
Web Interruption • Afternoon of 2/25 – Morning 2/26 – External DHS simulated attack to check data security (unscheduled, unknown by Census) – IT teams at Census: maintenance & applying updates • Resulted in performance issues in the web instrument – No advance warning of external simulations – NSCG was informed by respondents calling TQA line: • trouble logging in, delays in navigating through survey • Question: Can we figure out who was affected? U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU
Evidence in Server-Side Paradata service type id time (seconds) type 2 NSCG paradata 10800084 1361842005 login NSCG paradata 10800084 1361845429 submitted NSCG paradata 10800084 1361845466 logout NSCG paradata 10800159 1361843035 login NSCG paradata 10800159 1361846593 submitted NSCG paradata 10800159 1361846627 logout NSCG paradata 10800258 1361847610 login NSCG paradata 10800258 1361847625 login NSCG paradata 10800258 1361847959 login NSCG paradata 10802031 1361840282 failed_login NSCG paradata 10802031 1361840295 failed_login NSCG paradata 10802031 1361840349 failed_login U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 57 min 60 min 15 sec 5 min 13 sec <1 min
Evidence in Server-Side Paradata service type id time (seconds) type 2 NSCG paradata 10800084 1361842005 login NSCG paradata 10800084 1361845429 submitted NSCG paradata 10800084 1361845466 logout NSCG paradata 10800159 1361843035 login NSCG paradata 10800159 1361846593 submitted NSCG paradata 10800159 1361846627 logout NSCG paradata 10800258 1361847610 login NSCG paradata 10800258 1361847625 login NSCG paradata 10800258 1361847959 login NSCG paradata 10802031 1361840282 failed_login NSCG paradata 10802031 1361840295 failed_login NSCG paradata 10802031 1361840349 failed_login U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 57 min Patient respondents! 60 min 15 sec 5 min 13 sec <1 min
Evidence in Server-Side Paradata service type id time (seconds) type 2 NSCG paradata 10800084 1361842005 login NSCG paradata 10800084 1361845429 submitted NSCG paradata 10800084 1361845466 logout NSCG paradata 10800159 1361843035 login NSCG paradata 10800159 1361846593 submitted NSCG paradata 10800159 1361846627 logout NSCG paradata 10800258 1361847610 login NSCG paradata 10800258 1361847625 login NSCG paradata 10800258 1361847959 login NSCG paradata 10802031 1361840282 failed_login NSCG paradata 10802031 1361840295 failed_login NSCG paradata 10802031 1361840349 failed_login U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 57 min 60 min 2+ Logins in Short Time 15 sec 5 min 13 sec <1 min
Evidence in Server-Side Paradata service type id time (seconds) type 2 NSCG paradata 10800084 1361842005 login NSCG paradata 10800084 1361845429 submitted NSCG paradata 10800084 1361845466 logout NSCG paradata 10800159 1361843035 login NSCG paradata 10800159 1361846593 submitted NSCG paradata 10800159 1361846627 logout NSCG paradata 10800258 1361847610 login NSCG paradata 10800258 1361847625 login NSCG paradata 10800258 1361847959 login NSCG paradata 10802031 1361840282 failed_login NSCG paradata 10802031 1361840295 failed_login NSCG paradata 10802031 1361840349 failed_login U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 57 min 60 min 15 sec 5 min Can’t Successfully Log In 13 sec <1 min
Operational Questions • Major Concern: Burden / Respondent Frustration – Time in instrument • How long did individuals spend logged in? – Server breakoffs • Will sample persons come back to respond? • Examine Login Events – – Login/Submit = 1 event Login/Submit/Logout = 1 event Login/Logout = 1 event If there was no Submission or Logout: • Login alone = Server Breakoff • Failed login = Failed Login U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU
Increase in the mean time-in-instrument U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU
Increase in the mean time-in-instrument More server breakoffs than submissions! U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU
Mean time in instrument nearly back to average Server breakoffs still higher than expected U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 17
Action & Results • Determined time period of web issue: 6 pm – 2 am • Obtained approval to send special letter to affected cases – Apology and reminder of sample person’s importance to data quality – Turnaround time: 3 business days from analysis to mailing • Sent letter to all breakoff cases: – Not an experiment • Response rate of affected cases same as unaffected cases a month after mailing – Would this be the same if this issue occurred late in data collection? (Early respondents = higher engagement? ) • Encouraged that we could be responsive to a new issue U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU
Intelligent Mail Barcoding U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU
Mailing Operation in NSCG • NSCG heavily relies upon mailing operations – – – Pre-notice Letter Web invites Paper Questionnaires Reminder Letters & Postcards Prepaid Incentives • A respondent receiving all mailings in sequence: 11 mailings (plus any requested remails) • Historically a low-information operation – Was the address mailable? – Did we receive something in return? – Only have date of check-in at NPC U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 20
Integrated Postal System Tracking • USPS offers IPTS Data • Tracks data through each step of postal delivery – First scan at first post office – Every transfer scan – Last scan before delivery – Scan to reroute due to UAAs – Reasons for UAAs U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 21
IPTS for NSCG • Two points of interest in the NSCG – How much earlier can we know about UAAs? • Can send sample persons to locating earlier • Can save cost through not mailing future mailings – How much earlier can we know about return questionnaires? • Can reduce burden by not contacting likely respondents • Can reduce cost by not contacting likely respondents – Discuss UAA timeliness today U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 22
Implementation of IPTS • Add barcodes to mailing packages – Need outgoing barcodes to track progress and UAAs (letters and questionnaires) – Need return barcodes to track progress of return questionnaires (questionnaires only) • Redesign envelope to accommodate barcodes on printed letters and questionnaires – Ensure barcode can be scanned by USPS • Provide messaging on questionnaire return envelope to insert questionnaire correctly – Ensure barcode is displayed U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 23
Lag in UAA Identification • UAAs can be identified at various points – Delivery Point Verification before mailing (not mailed) – During any scan through postal system – When post office attempts to deliver mail • After UAA is identified – Mail package is rerouted through USPS back to NPC – Upon arrival at NPC, UAAs are checked in as resources allow • Report in Unified Tracking System (UTS) to track – Each mail piece and its mail date – When the UAA was reported by USPS – When the UAA was checked in at NPC U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 24
Lag in UAA Identification List of web invite letters and reminder mailings sent during Weeks 1 - 13 U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 25
Lag in UAA Identification Number of UAAs recorded by the USPS For each data collection operation. U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 26
Lag in UAA Identification Number & percentage of USPS records that also have a check-in record at NPC. U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 27
Lag in UAA Identification Question: How long is the lag from when we could see this information in IPTS to when we see it in checkin records at NPC? Number & percentage of USPS records that also have a check-in record at NPC. U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 28
Lag in UAA Identification Average, median, and max lag between USPS and NPC U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 29
Action & Results • Clear that data from the USPS scanning – More timely – Cost neutral, if not cheaper • Still some questions about UAA data – Understand lag between mailing and USPS UAA – Better understand NPC check-in process • Records we cannot currently identify • Monitor and use USPS UAA data in 2017 • Implement return questionnaire scan data in 2017 – Monitor and evaluate – Place potential respondents on hold in other modes until we determine validity of complete U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU
Summary • High volume of system-generated paradata – – Need to know what is available Need to know who provides it so you can ask Possible operational changes to collect or use (envelopes) May need new systems to manage or track paradata • Can be monitored to improve data collection – Efficiency and timeliness • Might be able to help answer research questions – May need to combine data across modes to answer research questions – Important to keep in mind limitations of system paradata U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU 31
Contact Information stephanie. coffey@census. gov U. S. Department of Commerce Economics and Statistics Administration U. S. CENSUS BUREAU
- Paradata vs metadata
- Using system.collections.generic
- Primary target market and secondary target market
- Substitution division
- Evaluate trig function using period as aid
- What is synthetic substitution
- Unit circle explained
- Simplifying expressions using properties of exponents
- Min data
- Cost of target breach
- Cognitive psychology essay
- Assure lesson plan
- Productdesigners do not conceptualize and evaluate ideas
- Evaluate permutations and combinations
- Clive wearing tutor2u
- What is media information
- Evaluate own knowledge and performance
- Simplify and evaluate algebraic expressions
- Learning theory attachment
- Zam is laying out patio stones in a triangular pattern
- Evaluate and graph polynomial functions
- Outline and evaluate bowlby's monotropic theory 8 marks
- Target system
- An embedded software primer
- Target board in embedded system
- Dtfd switch
- Is earth a closed system or open system
- Respiratory system circulatory system digestive system
- Using big data to solve economic and social problems
- My nasa data earth system data explorer
- Radar system analysis and design using matlab
- Zero and negative exponents examples
- Why is imprisonment an external form of social control