Chapter Thirteen Measurement Winston Jackson and Norine Verberg

A. Theoretical, Conceptual, and Operational Levels Measurement is the “process of linking abstract concepts to empirical referents”(Carmines & Zeller) n Hence, one moves from the general (theoretical level) to the specific (empirical level) n i. e. , For each concept, an indicator is identified e. g. , What is the best way to measure, or indicate, a person’s social prestige? The concepts we measure are called variables n See Figure 13. 1 (next slide) n n © 2007 Pearson Education Canada 2

Figure 13. 1 Levels in Research Design n FPO Figure 13. 1 Levels in

Figure 13. 1 n Shows movement from the general to the specific – from theoretical level to the operational level n referred to as operationalization n At theoretical level, concepts (e. g. , socioeconomic status, alienation, job satisfaction, conformity, age, gender, poverty, political efficacy) are conceptualized n At the operational level, the researcher must create measures (or indicators) for the concept n Indicators should reflect the variable’s conceptual definition © 2007 Pearson Education Canada 4

Assessing Indicators We assess the link between the concepts and the indicators by evaluating the validity and reliability of the indicators 1. Validity n the extent to which a measure reflects a concept, reflecting neither more nor less than what is implied by the conceptual definition 2. Reliability n the extent to which, on repeated measures, an indicator yields similar readings © 2007 Pearson Education Canada 5

1. Validity (in Quantitative Research) n Illustration: concept - socioeconomic status n Conceptual definition: a “hierarchical continuum of respect and prestige” n Operational definition: annual salary n Assessment: Low validity (salary might not capture prestige – widows, ministers, nuns – prestige and respect would be higher than income suggests n Measure should be congruent with conceptual definition © 2007 Pearson Education Canada 6

Types of validity n Face validity… on the face of it. . . n Content validity…reflects the dimension(s) implied by the concept n Criterion validity: two types Concurrent validity…correlation of one measure with another n Predictive validity. . . predict accurately n n Construct validity…distinguishes participants who differ on the construct © 2007 Pearson Education Canada 7

Validity in experimental design n Internal validity- the extent to which you can demonstrate that the treatment produces changes in dependent variable n External validity - the extent to which one can extrapolate from study to the general pop’n n In qualitative research… “credibility” is the issue. . n Degree to which the description “rings true” to the subjects of the study, to other readers, or to other researchers © 2007 Pearson Education Canada 8

2. Reliability (in ( Quantitative Methods) Reliability refers to the extent to which, on repeated measures, an indicator yields similar readings. n Assessing internal reliability of items used to construct an index (an index combines several items into a single score) Split-half method: randomly split the items in two, construct index, do results correlate highly? n Internal consistency: statistical procedure done in SPSS (described later in chapter) n © 2007 Pearson Education Canada 9

B. Measurement Error n Researchers assume that the object being measured has two or more values (i. e. , is not a constant) and that it has a “true value” n true value – the underlying exact quantity of a variable at any given time n Researchers also assume that measurement errors will always occur because instruments are imperfect n Measurement error is any deviation from “true value” © 2007 Pearson Education Canada 10

Measurement Error (Cont’d) n Measures are made up of the following components: n MEASURE= true value +/- (SE+/-RE) n SE ~ Systematic error is non-random error that systematically over- or under-estimates a value (eg. , systematically assigning the lowest value when a respondent does not answer) n RE ~ Random error is random fluctuations around the true value n Not a problematic…should average out © 2007 Pearson Education Canada 11

1. Tips for Reducing Random and Systematic Error 1. Take average of several measures 2. Use several different indicators 3. Use random sampling procedures 4. Use sensitive measures 5. Avoid confusion in wording of questions or instructions 6. Error-check data carefully 7. Reduce subject/experimenter expectations © 2007 Pearson Education Canada 12

C. Levels of Measurement n Introduced in Chapter 8; This chapter stresses the importance of level of measurement for measuring concepts n Type of level of measurement influences which statistical procedures one can use n Three levels of measurement 1. Nominal 2. Ordinal 3. Ratio © 2007 Pearson Education Canada 13

D. The Effects of Reduced Levels of Measurement n Best to achieve most precise, and highest, level of measurements possible n When lower levels are used, the results underestimate the relative importance of a variable The greater the reduction in measurement precision, the greater the drop in correlations between variable n Precisely measured variables will appear to be more important than poorly measured ones n © 2007 Pearson Education Canada 15

E. Indexes, Scales, and Special Measurement Procedures n Combining several indicators into one score results in an index or scale n While used interchangeably, an index refers to the combination of two or more indicators; a scale refers to a more complex combination of indicators where the pattern of responses is taken into account n Indexes are routinely constructed to reflect complex variables n Socioeconomic status, job satisfaction, group dynamics, social attitudes toward an issue © 2007 Pearson Education Canada 16

1. Item Analysis n Items in an index should discriminate well n Example of test item development n Test graded, students divided into upper and lower quartile n Examine performance on each question n Select those questions that discriminate best © 2007 Pearson Education Canada 17

Discrimination of Items n FPO Table 13. 1 Discrimination Ability of 100 Items: Percentage

2. Selecting Index Items 1. Review conceptual definition 1. Does the concept have ranges or dimensions 2. Develop measures for each dimension 1. Developed items for each dimension of the concept 3. Pre-test index 1. Complete the index yourself, then pre-test it with target-group members 4. Pilot test index 1. Use SPSS to assess internal consistency © 2007 Pearson Education Canada 19

3. The Rationale for Using Several Items in an Index © 2007 Pearson Education

4. Likert-based Indexes © 2007 Pearson Education Canada 21

A. Tips for Constructing Likertbased Index n The “and” alert: avoid multiple dimensions n Strongly Agree on right hand side 9 -points n Response set issue n Avoid negatives like “not” simply use negative wording. n Vary strength of wording to produce variation in response n Exercise…. items for a euthanasia index © 2007 Pearson Education Canada 22

A. Tips for Constructing a Likertbased Index © 2007 Pearson Education Canada 23

B. Evaluation of Likert-based Indexes © 2007 Pearson Education Canada 24

C. Using the Internal Consistency Approach to Selecting Index Items © 2007 Pearson Education

5. Semantic Differential Procedures n A variety of anchors are used and people place themselves or others on a continuum: shy/outgoing; bookworm/social butterfly Continued… © 2007 Pearson Education Canada 26

5. Box 13. 3 © 2007 Pearson Education Canada 27

6. Magnitude Estimation Procedures n subjects use numbers or line lengths to indicate perceptions. Very good for comparisons: yields ratio level measures. Comparing liking of teachers; seriousness of crimes; liking of one community compared to another one, etc. © 2007 Pearson Education Canada 28

Tips for Using Magnitude Estimation Procedures 1. Only use ME when a researcher is present to explain the method to respondents 2. Use ME when comparative judgments sought 3. Use a stimulus category somewhere near the middle of the range you intend to use as a standard (avoid a standard too high or low) 4. After the standard established, © 2007 Pearson Education Canada 29