Does the polarity of rating scales matter How

Does the polarity of rating scales matter? How unipolar, bipolar and mixed rating scales affect measurements with latent variables Natalja Menold 5 th Conference of the European Survey Research Association (ESRA), 15 -19 July, 2013

Content • • • Introduction: Polarity and mixed scales Research questions Study design Results: Concept 1 and Concept 2 Conclusions

3 Scale Polarity Krosnick & Fabrigar , 1997 Bipolar: Scales reflecting two opposing dimensions • agree-disagree, like-dislike, -5…+5 Unipolar: reflect varying levels of the same dimension with a zero point on the one end • do not agree at all -fully agree; not important – very important, 0 -10 Meaning of the middle category • in bipolar scales: neither nor, partly-partly • in unipolar scales: a moderate position

4 Mixed scales in Surveys ESS blue print ISSP 2007 B 3: bipolar Very difficult Difficult Neither difficult nor easy Easy Very easy moderately? D 20 a: Mixed Not at all important Not important Neither important nor unimportant Important Very important 12 a-e: DE Mixed stimme voll und ganz zu stimme zu weder noch stimme nicht zu stimme überhaupt nicht zu 12 a-e: AU Mixed stimme voll und ganz zu stimme zu teils / teils stimme nicht zu stimme überhaupt nicht zu

Response Sets Attraction to middle category (Korman 1971) as Middle Response Set (MRS): respondents tend to use the middle category more often than other categories Positivity bias: Reluctance to give negative evaluations resulting in tendency to overuse the positive end of the scale (e. g. Landy & Farr, 1980; Lau, Sears, & Centers, 1979) Response set: tendency (affected by situation) to respond to questionnaire items on some basis other than the item content (Paulhus, 1991)

Factors Associated with Response Sets Need for accuracy (Kruglanski, 1989) • personality variable, extent to which individuals are motivated to be as accurate as possible Need for cognition (NFC, Cacioppo & Petty, 1982) • personality variable, extent to which individuals are inclined towards effortful cognitive activities Attitude strength (AS, Petty & Krosnick, 1995; Bassili & Krosnick, 2000) • Strong attitudes: highly pronounced, with a high impact on cognition and behavior

Research Questions • How does variation in polarity using verbal labels and mixing middle category affect • measurement quality of latent variables? • attitude values measured with latent variables? • Which is the role of… • response sets (MRS and positivity bias)? • tendency to be accurate, need for cognition and attitude strength?

Methods • A randomized 2 x 2 between-subject experimental design, varying polarity and middle category • Analysis of measurement quality of latent variables: – Factorial validity (CFAs, MLR with Mplus) • R 2 of items as items‘ reliability • Loadings (min. λ =. 30) as items validities (e. g. Andrews, 1984) – Internal consistencies (Cronbach‘s α; min. α =. 60) • SEM with structural and measurement components and covariates (MLR with Mplus) • Web Survey: GESIS Online Access Panel • A probability sample of German residents: N = 552, 53% males, age M = 42. 63, SD = 14. 77

Items and Constructs Women’s labour force participation (ALLBUS 2008) 2 Factors: CFA, MLR: CMIN (8) = 56. 52, p <. 001; RMSEA =. 04; CFI =. 99 r = -. 81; loadings between. 54 and. 88 • F 1: Employed mothers: 3 items, e. g. a little child would suffer when the mother is employed • F 2: Women as wife: 3 items, e. g. a married women should better support the husband in his career than make her own career Economic values (ALLBUS 2004) 2 Factors: CFA, MLR: CMIN (13) = 66. 52, p <. 001; RMSEA =. 037; CFI =. 93; r =. 24; loadings between. 38 and. 89 • F 1: Government intervention: 2 items, e. g. Government should ensure that everybody has a job and that prises are stable. • F 2: Inequality: 5 items, e. g. economic rewards are fairly shared in Germany / even though one would want, social differences in Germany could not be lower than those currenly are.

Realization of rating scales Women’s labour force participation: 7 categories Economic Values: 5 categories Bipolar (Bip) lehne völlig ab / lehne eher ab /teils-teils /stimme eher zu / stimme zu/ stimme voll und ganz zu fully disagree……partly-partly ……fully agree Bipolar Mixed (Bip. Mix) lehne völlig ab / lehne eher ab / in der Mitte / stimme eher zu/stimme zu / stimme voll und ganz zu fully disagree ……. . in the Middle…. . fully agree Unipolar Mixed (Unip. Mix) stimmt überhaupt nicht / stimmt wenig/ teils-teils / stimmt ziemlich/stimmt voll und ganz does not apply at all ……partly-partly…. fully applies Unipolar (Uni) stimmt überhaupt nicht / stimmt wenig / stimmt mittelmäßig /stimmt / stimmt ziemlich / stimmt voll und ganz does not apply at all…. . moderately applies…. fully applies

Covariates • Conscientiousness (accuracy and thoroughness): 5 items, a subscale of BFI-K (Rammstedt & John, 2005) • Need for cognition (NFC): 16 items of a German NFCscale (Bless et al. , 2006) • Attitude Strength Topic importance * certainty when answering the items related to the topic.

Women’s labour force participation Model fit of CFAs Bip. Mix Unip 23. 26** 15. 36* 8. 56 10. 62 a CFI . 91 . 96 1. 00 . 99 RMSEA . 12 . 08 . 02 . 04 . 07 -. 18 . 00 -. 14 . 00 -. 11 . 00 -. 011 132 138 134 130 CMIN (df = 8) 90% C. I. of RMSEA N - pure model fit in the case of bipolar scale - close fit in the case of unipolar scales adf = 9

Women’s labour force participation Item and Factor Statistics R 2 of items 1 2 3 4 5 6 λ of items f 1 1 3 5 f 2 2 4 6 r(f 1, f 2) Bip . 24. 09. 42. 44. 50. 86 . 49. 65 -. 71. 30 -. 66 -. 93 Bip. Mix . 47. 46. 70. 58. 47. 44 . 69 -. 83. 68. 67. 76. 67 Unip. Mix . 45. 28. 53. 99. 42. 41 . 67 -. 73. 65. 53 1. 00. 64 Unip . 45. 47. 56. 86. 39. 59 . 67 -. 75. 63. 68. 93. 77 . 86 -. 66 -. 75 -. 71 Bip and Unip. Mix: Low reliabilities of several items Bip: The highest factor intercorrelation; sign differs from other scales

Women’s labour force participation Cronbach‘s Alpha f 1 f 2 Bip . 76 . 83 Bip. Mix Unip . 77 . 71 . 72 . 74 . 77 . 84 Mixed scales: lower internal consistencies of factor 2

Women’s labour force participation SEM NFC C AS a 1 f 1 MID MRS a 3 a 5 NEG a 2 POL f 2 a 4 a 6 NFC Need for cognition C Conscientiousness AS Attitude Strength MID: non-mixed = 1 mixed = 2 POL: bipolar = 1 unipolar = 2 MRS: middle responding NEG: Negativity: usage of the negative part of the scale Indirekt effects of POL, MID and interaction on f 1 and f 2 are not presented

-. 18*** -. 16*** NFC . 14** C AS. 33*** -. 09** -. 38*** MID . 10* MRS f 1 -. 56*** -. 75*** a 3 -. 30*** -. 17** -. 37* a 5 . 5 -. 63*** . 21*. 34** a 1 1 -. 24** -. 70*** NEG -. 88*** POL -. 16***. 33***. 06* NFC C -. 20*** f 2 -. 25*** AS a 2 a 4 a 6 Covariates NFC, C: no or small effects Effect of AS - the higher AS the lower MRS - the higher AS the higher NEG 1 . 5 CMIN (49) = 137. 18*** RMSEA =. 06 90% C. I. of RMSEA. 05 -. 07 CFI =. 96 SRMR =. 04

-. 18*** -. 16*** NFC . 14** C No sign. direct effects of MID, POL and interaction term on f 1/f 2 AS. 33*** -. 09** -. 38*** MID -. 37* MRS -. 17** -. 56*** -. 63*** . 21*. 34** POL -. 30*** . 10* f 1 NEG -. 20*** 1 a 1 Sign. direct effects of MRS and NEG on f 1 and f 2 -. 75*** a 3 Sign. direct effect of MID on MRS: higher MRS for mixed . 5 a 5 Sign. direct effect of POL on MRS and NEG: - higher MRS for unipolar -. 70*** - lower NEG for unipolar 1 a 2 Sign. direct effects of interaction term -. 88*** -. 24** -. 16***. 33***. 06* NFC C f 2 -. 25*** AS a 4 a 6 . 5

Women’s labour force participation MRS Interactions NEG partly-partly

Indirect Effects MID f 1 -. 02*. 11**. 08** -. 07** -. 19* -. 10** POL MRS NEG . 21** f 2 - Sign. indirect effects of polarity and interaction term on factors

Economic Values: Results of CFAs Bip. Mix Unip 28. 86* 21. 49 16. 14 33. 67** CFI . 93 . 95 . 98 . 80 RMSEA . 07 . 04 . 11 . 02 -. 12 . 00 -. 10 . 07 -. 17 132 138 134 130 CMIN (df = 13) 90% C. I. of RMSEA N Acceptable results for both mixed scales; best model fit for unipolar mixed scale

Economic Values Item and scale statistics R 2 of items 2 3 4 5 6 7 8 λ of items f 1 3 4 f 2 2 5 6 7 8 r(f 1, f 2) Bip . 33. 36. 75. 05. 22. 58. 34 . 60. 87. 58. 23. 47. 76. 59 Bip. Mix . 30. 68. 30. 17. 20. 53. 64 . 83. 55. 41. 45. 73. 80 Unip. Mix . 45. 56. 66. 11. 20. 54. 66 . 75. 81. 67. 33. 44. 73. 81 Unip . 31. 63. 35. 13. 14. 52. 35 . 79. 55. 36. 37. 72. 58 -. 46 -. 62 -. 25 -. 31 Biplar scale: Low reliability and validity of item 5 Higher (negative) factor intercorrelations in bipolar scales

Economic Values Cronbach‘s Alpha f 1 f 2 Bip . 68. 64 Bip. Mix Unip . 62. 71 . 75. 71 . 63. 60 Factor 1: a higher value in unipolar mixed Factor 2: low values in bipolar and unipolar

Economic Values Summary of SEM results - Bipolar and unipolar mixed scale: higher MRS and positivity bias - No, little effects of Need for cognition and Conscientiousness; An effect of Attitude Strength - MRS and positivity bias are mediators of rating scale effects on factor measurements

Conclusions • • • Polarity does matter! Bipolar (non mixed) scale: poor measurement quality Effect of rating scales on factor values Attitude Strength is an important covariate variable MRS and NEG (=positivity bias) are mediators of effects on the factor measurements • Mixed scales and measurement quality – > more research is needed – > number of categories should be considered

Thank you! contact: natalja. menold@gesis. org