Adjust Survey Response Distributions Using Multiple Imputation A

Adjust Survey Response Distributions Using Multiple Imputation: A Simulation with External Validation Frank Liu & Yu-Sung Su

We Want a Better Guess of Respondents' Preferences.

How to Utilize Multiple Imputation to make better guesses about the respondent’s preferences?

• • Calculating proportions based on raw data and omitting the non-response data result in biased proportion of interested variables (Bernhagen & Marsh, 2007). Multiple Imputation (MI) is a cost-efficient and methodological sound approach for better use of raw survey and poll data (Barzi, 2004).

Picture Source: Kyle F. Edwards, Christopher A. Klausmeier, and Elena Litchman (2011)

MI Modeling vs. MI for DV

While most studies use MI for modeling, there is room to examine to which MI can be applied to electoral forecast (King et al. , 2001; Snijders & Bosker, 2011).

Joint MI (Amelia II) vs. Conditional MI (mi & MICE)

Joint MI assumes a multivariate normal distribution. vs. Conditional MI does NOT.

mi takes advantage of existing regression models to handle various kinds of variables types (Su, Gelman, Hill, & Yajima, 2011): • using a logistical regression model to predict a binary outcome, • Using an ordered logit regression model to predict an ordinal outcome, • Using a multinomial logit regression model to predict an unordered categorical outcome.

mi results can be very biased when data include extreme values (He and Raghunathan, 2009)

Data RDD telephone survey data about Taiwanese voters’ national identification (N=1, 078), collected from Jan. 23 to Feb. 4, 2013. (AAPOR #3: 21. 56%)

Sensitive Question (DV) and Item Non-Response Variables camp (v 33) Question Wording Distribution Missing (%) Do you support for any Blue camp: 215 658 political party? Green camp: 204 (61. 04)

camp v 6 v 7 v 9 v 10 v 17 v 18 v 20 v 22 v 27 v 28 v 29 v 32 v 37 v 38 age edu sex Missing % 70 60 50 40 30 Missing % 20 10 0

Auxiliary Variables (1) v Wording v 6 Do you agree that we can influence the government with our votes? v 7 Do you agree that we have little influence on what the government plans to do? v 9 Do you agree that we should use Taiwanese as the major language in Taiwan? v 10 Do you agree that Taiwanese children perform better than those in mainland China? v 17 Do you agree that those identifying with Taiwan can be titled as Taiwanese? v 18 Do you agree that Chinese from mainland China have more money than sense? v 20 Do you agree that those people should not be called Taiwanese if they don't know Matsu(name of a sea goddess widely worshipped on the SE China coast and in SE Asia)?

v 22 Do you agree that our government should give more restrict policy on mainland China tourists? v 27 Some call themselves Taiwanese, some Chinese, and some Both, how about you? v 28 Do you agree that “Taiwan” is the formal name of our country? v 29 Do you agree that Taiwanese people and those in mainland China belongs to a same nation? v 32 Do you agree that we should seek for unification with mainland China if it becomes a democracy? v 37 Do you agree that the two sides of Taiwan Straight will eventually becomes one country? v 38 Do you ever go to mainland China in the past two years? age Which year were you born? (re-coded to real age) edu What is your education level? sex (coded by interviewer)

Study 1: Check Missingness Patterns • Using conditional MI algorithm to check its missingness patterns and to conduct MI. • the data should be at least missing at random (MAR), i. e. , the missingness mechanism does not depend on the unobserved data. • MCAR > MNAR

Check the Missingness Patterns 1: Specify a conditional MI model • include many valid auxiliary variables (17); 2: Contrast Simulated Data with MI Data • • • Set the imputed datasets as the baseline for comparison create three copies of the MI data randomly remove values from the completely imputed data according to the original missing rate of the data, i. e. , 61%.

• We create simulated datasets by modeling the missingness of each variable conditional on a linear combination of the rest of variables with logistic regressions. • Then we use the predictive missingness to create missing values on the three imputed data. • In short, we compare simulated datasets with the original MI dataset.

Results of the Check for MCAR (1)

Results of the Check for MCAR (2) Mean SE Blue Green Original Data 0. 488 0. 500 51. 1% 48. 8% Imputed Original Data 0. 474+ 0. 500+ 52. 6% 47. 4% #1 Imputed MCAR Data 0. 464+ 0. 499+ 53. 5% 46. 5% #2 Imputed MCAR Data 0. 464+ 0. 500+ 53. 6% 46. 4% #3 Imputed MCAR Data 0. 453+ 0. 500+ 54. 7% 45. 3% Table 2: Summary of the camp variable between the original data and imputed MCAR datasets. Note: + The mean’s and SE’s reported here are pooled mean and SE’s for three chains of MI.

Results of the Check for MCAR (3) Figure 3: Plots of Camp Variable against Other Variable Using the Imputed MCAR data.

Results of the Check for MAR (1)

Results of the Check for MAR (2) Mean* SE* Blue Green Original Data 0. 488 0. 500 51. 1% 48. 8% Imputed Original Data 0. 474+ 0. 500+ 52. 6% 47. 4% #1 Imputed MAR Data 0. 477+ 0. 500+ 50. 1% 49. 9% #2 Imputed MAR Data 0. 497+ 0. 500+ 50. 3% 49. 7% #3 Imputed MAR Data 0. 499+ 0. 500+ 50. 1% 49. 9% Table 3: Summary of the camp variable between the original data and imputed MAR datasets. Note: + The mean’s and SE’s reported here are pooled mean and SE’s for three chains of MI.

Results of the Check for MAR (3)

Summary: the assumption of MAR holds

Study 2: External Validation 1: compare respondents' answers with MI guesses • figure out how well the MI prediction works 2: understand why prediction performs not so well, if this is the case. • • follow-up telephone surveys with identity check Face-to-face one-by-one interviews for explanations

Follow-up telephone surveys 1. Called out • • • for the 658 respondents’ camp identification whose political camp id is NA. (April 13 -15, 2013) Forced to choose between Green and Blue. N=143 2. Identity check • Answers must be consistent with the first survey regarding two questions: (1) whether ever going to mainland China in the past two years; (2) the frequency of watching political news

3. In-depth Interview: • • 45 out of 143 respondents were contacted for face-toface one-by-one Interview. (incentive: cash $70) 5 out of 45 respondents were interviewed between April 20 to May 6, 2013.

Findings ID 5 8 9 15 24 25 36 44 58 59 69 MI 0. 161 0. 400 0. 804 0. 728 0. 348 0. 259 0. 300 0. 574 0. 587 0. 730 0. 382 Camp B B G G B B B G B the total number of the 145 respondents whose values fall between. 45 and. 55: 19 pred. correct 1 1 1 1 0 1 1

Rate of Successful Prediction using MI: 70% (100/145).

Case #1 ID Sex Age Edu Date-Time 905 F 34 Place 3 2013. 4. 27 Taipei Main 10: 00 Station, Taipei City MI Camp ID 0. 42 G (B) Causes of Inconsistency Disappointed by the KMT's reform on domestic policies. Feeling lost some reasons to keep supporting KMT.

Case #2 ID Sex Age Edu 206 M 43 2 Date. Time 13. 4. 30 10: 00 Place MI Camp ID NSYSU campus, 0. 50 Kaohsiung City (B) B Causes of Inconsistency Disappointed by both political camps but felt more concerned about DPP than KMT regarding DPP's ideology of seeking Taiwan independence.

Case #3 ID Sex Age Edu Date. Time 140 M 29 3 Place MI Camp ID 13. 4. 30 Kaohsiung City 0. 54 14: 00 (G) B Causes of Inconsistency Growing up with KMT supporter mother and have been ok with KMT. Turning to like DPP for a growing Taiwanese national identification.

Case #4 ID Sex Age Edu 384 M 25 4 Date. Time Place 13. 5. 4 14: 00 Taipei Main Station, Taipei City MI Camp ID 0. 47 G (B) Causes of Inconsistency Feeling crosspressured because parents support for KMT but friends support for DPP. The first vote is for DPP in 2008 presidential election. Feeling disappointed by KMT's performance but not aware the core ideology of DPP.

Case #5 ID Sex Age Edu 286 M 37 3 Date. Time Place 2013. 5. 4 16: 00 Taipei Main Station, Taipei City MI Camp ID 0. 40 G (B) Causes of Inconsistency Feeling crosspressured because his family have been supporting for KMT but wife's family support for DPP. Feeling disappoint about KMT's leadership.

Summary 1. These “danglers” are politically aware. 2. Explanations for the ambivalence: 1. potential blue camp supporters (ID 905, 206, 384, and 286) have become unsatisfied with the incumbent's performance and public policy. 2. potential green camp supporters (ID 905, 206, 384, and 286) are affected by nationalism. 3. Cross-pressured (ID 140, 384, and 286).

Conclusion 1. MI scores reflect respondents’ partisan orientation, including the level of their ambivalence. It seems reasonable to adopt this method to reconstruct the distribution of partisan orientation of the electorate. 2. By face-to-face interviews inconsistency of their answers can be explained.

Implications & Suggestions • Encourage respondents to give up answering those they could not answer • Explore auxiliary questions for partisanship (e. g. , policy and performance ones) and encourage respondents to answer sincerely that seem less sensitive to them. • Use these easier auxiliary variables for MI.

• Continue to test the reliability of this method using panel data. • Be aware of the limit of forcing respondents to choose an answer. Alternative methods or question wording of probing true answers about partisanship are still needed.

Thank you. request for the paper, data, and/or R code frankcsliu@gmail. com