Electronic Questionnaire Collection at Statistics Canada Milana Karaganis

Electronic Questionnaire Collection at Statistics Canada Milana Karaganis Marc St-Denis

Outline § Electronic Questionnaire project § Electronic Questionnaire for business surveys 2 Statistics Canada • Statistique Canada 2020 -11 -23

Electronic Questionnaire Project § Build generic EQ service to be used by all surveys • EQ platform • Standard approach to EQ § Create a critical mass of EQ service users • 160+ surveys • Business surveys and some social surveys § Reduce collection and operations costs § Timeline: April 1, 2010 -> March 31, 2016 3 Statistics Canada • Statistique Canada 2020 -11 -23

Electronic Questionnaire Project § E-questionnaire solution • Support all EQ applications for all surveys • Use Statistics Canada infrastructure • Comply with security and confidentiality standards • Automate EQ survey application development -> build an engine to render EQ applications based on provided inputs, E-questionnaire Generation System (EQGS) 4 Statistics Canada • Statistique Canada 2020 -11 -23

Electronic Questionnaire Project 5 Statistics Canada • Statistique Canada 2020 -11 -23

Electronic Questionnaire Project § Standard approach to EQ – corporate solution • • Develop EQ applications according to EQ standards Apply standard look and feel Build functionalities required for all surveys Use standard collection strategies EQ as initial collection mode in sequential mode collection • Use standard development process Shorten development cycle 6 Statistics Canada • Statistique Canada 2020 -11 -23

7 Statistics Canada • Statistique Canada 2020 -11 -23

8 Statistics Canada • Statistique Canada 2020 -11 -23

Progress To-Date § EQ solution implemented • • Change Management process First set of EQ standards released Survey development process created Initial collection strategies established § Deployed 39 EQ applications and realized efficiencies § Overall success: • Lessons learned • Big cultural shift 9 Statistics Canada • Statistique Canada 2020 -11 -23

EQ Among Business Surveys § Outline • • 10 Evolution of EQ collection strategies Results observed to date Survey collection strategy guidelines Results of tests Statistics Canada • Statistique Canada 2020 -11 -23

Collection Strategies Among Business Surveys § Moving from conventional collection modes (PAPI & CATI) to EQ (over 100 surveys in 5 yrs) § Evolution of multi-modal collection with EQ as primary mode § Measuring results and improving • Respecting data quality, timeliness, budgets § Reviewing collection strategies 11 Statistics Canada • Statistique Canada 2020 -11 -23

Reviewing Collection Strategy Quarterly Services Indicators (QSI) Survey Services Indicator Survey (quarterly ) (QSI) 2011 Q 4 2011 Mail out / E-invitation e-reminder / fax Phone reminder (all) e-reminder / fax Fax to EQ End of collection 12 Q 1 2012 Oct 3, 2011 Oct 17 Oct 24 Nov 7 Nov 14 Nov 28 Dec 5 Dec 9, 2011 Statistics Canada • Statistique Canada Mail out / E-invitation e-reminder Phone reminder (all) e-reminder Phone and fax (all) End of collection Jan 4, 2012 Jan 18 Jan 25 Feb 8 Feb 15 Feb 29 Mar 7 Mar 9, 2012 2020 -11 -23

Reviewing Collection Strategy Quarterly Services Indicators (QSI) Survey EQ Q 4 2011 Final Sample 1, 525 Response Rate # of cases in NRFU 98. 3% 905 59. 3% 1, 351 13 96. 4% 22 50. 6% Statistics Canada • Statistique Canada 1. 7% 81. 5% Mail 50 65. 6% 683 Total 50. 0% EQ Response Rate # of cases in NRFU 27 58. 5% Q 1 2012 Final Sample Mail 3. 6% 46. 2% 41 82. 0% 1, 552 58. 0% 927 Total 1, 401 65. 6% 724 2020 -11 -23

EQ Collection Strategy Guidelines § Developing tools to ensure optimal collection strategies (monthly; quarterly; annual) • Follow-up schedule geared to multi-mode collection § Standardise approach towards efficiencies § Considering respondent burden, data quality, response rates, costs 14 Statistics Canada • Statistique Canada 2020 -11 -23

Date Pre-contact Mail-out / E-invitation send date: Collection Start date: Bounce-backs follow up 1 st Follow up attempt date: Phone EQ Fax Mail 2 nd Follow up attempt date: Phone EQ Fax Mail 3 rd Follow up attempt date: Phone 4 th Follow up attempt date: Phone EQ Fax Mail 5 th Follow up attempt date: Phone 6 th Follow up attempt date: Phone EQ Days after Mail out n/a Type of FU 0 1 week max to complete then switch to mail for those unable 2 to get correct email 30 30 X X only for those without e-mail address (mail units) 1 st reminder email no fax unless requested no mail unless requested 60 only for those without e-mail address (mail units) 2 nd reminder email X no fax unless requested X no mail unless requested Every 2 nd week 90 only for those without e-mail address (mail units) 2 nd reminder email X no fax unless requested X no mail unless requested Every 2 nd week 120 only for those without e-mail address (mail units) 3 rd reminder email fax reminder + copy of questionnaire to those without email 120 address 120 mail only to those without email address or fax number Every 2 nd week Fax Mail 140 all outstanding 5 th reminder email fax reminder + copy of questionnaire to those without email 140 address 140 mail only to those without email address or fax number EQ Fax Mail 160 105 X X Every 2 nd week all outstanding units 6 th and final reminder no fax no mail

EQ Experimental Design for the UES Mailout Mar 18 T 1 E-invitation Mar 28 (Standard) T 2 (E-mails only) T 3 (phone attempt) Reminders E-mail only Reminder schedule 1 st April 26 2 nd May 26 3 rd June 23 4 th July 25 Yes 1 st April 26 2 nd May 9 3 rd May 26 4 th June 7 No (firm appt only) June 20 *July 11 1 st attempt – phone; 3 next – E-mail 1 st April 26 -29 2 nd May 26 3 rd June 23 4 th July 25 Yes Phone follow-up Experiment end date Sample size allocated to the treatment 16 August 1 1125 1102 Statistics Canada • Statistique Canada August 1 1148 T 4 (phone attempt + e-mails) 1 st attempt – phone; 3 next – E-mail 1 st April 26 -29 2 nd May 9 3 rd May 26 4 th June 7 No (firm appt only) June 20 *July 11 1114 2020 -11 -23

Results of the experimental design Progressive unweighted return rate (%) for each treatment (2011) April 26 May 9 May 26 June 7 June 20 August 1 Sept 12 Oct 14 T 1 %u 13% 24% 36% 44% 51% 66% 77% 82% T 2 %u 12. 5% 22% 41% 43% 46% 54% 76% 81% T 3 %u 15% 24% 37% 46% 53% 68% 78% 82% T 4 %u 12% 22% 45% 49% 52% 58% 75% 81% Number of call attempts by survey treatment 17 T 1 T 2 T 3 T 4 3. 8 2. 8 4. 3 3. 9 Statistics Canada • Statistique Canada 2020 -11 -23

Highlights § Treatments consisting of 4 e-reminders only resulted in 46+% as of June 20 th § At end of collection (Oct 14) the 4 treatments had very similar response rates § T 2 showed the lowest attempts and lowest TPU § Limited worth conducting telephone NRFU for first few months; still reach similar final response rates with fewer phone attempts 18 Statistics Canada • Statistique Canada 2020 -11 -23

Access Code Letter § Piloted for UES 2012 collection § Conducted pre-contact for birth cases in sample § Unable to contact cases = mailed an access code letter providing hyperlink to an EQ § More difficult cases since unable to reach during pre-contact § After 30 days, telephone NRFU 19 Statistics Canada • Statistique Canada 2020 -11 -23

Access Code Letters: Results Collection status on July 16, 2012 for units that received an access code letter # units (total) Total 20 Resp. (EQ) 765 150 (20%) Statistics Canada • Statistique Canada Status Access code letter Resp. Non-Resp. (Mail) 39 (5%) 52 (7%) Out of Scope 74 (9%) In Progress 450 (60%) 2020 -11 -23

The Future § § § § § Successful in achieving its objectives Mission critical monthly surveys UES (60 surveys) + LFS Add new functionalities to EQ platform Respondent EQ = Interviewer EQ Assess respondent’s level of tolerance to edits Streamline processes and operations Monitor quality, response rates, costs Research 21 Statistics Canada • Statistique Canada 2020 -11 -23

milana. karaganis@statcan. gc. ca marc. st-denis@statcan. gc. ca 22 Statistics Canada • Statistique Canada 2020 -11 -23