Measurements and Validity Julia Braverman Ph D Division

Measurements and Validity Julia Braverman, Ph. D Division on Addictions

Types of measures ANXIETY? ? Michael John

Types of measurement 1. 2. 3. Objective/Physiological measures • • Bodily activity, nervous system. Response time Observational measures • Direct observing participants. Self-report • Participants provide information about themselves.

Converging operations Using several measurement approaches to measure a particular variable

Basics of psychometrics: How to build a trait/state assessment measure? l Concept • Affect • E. g. I feel sad • Behavior • E. g. I cannot sleep, I cry a lot • Cognition • E. g. I think about suicide. l Question format (Likert scale, yes/no, reverse scale)

Measure quality 1. 2. Reliability Validity

Reliability l The degree of consistency between observations made by the same measurement tool.

Measurement Error. No measure is perfect. l Observed score = True score + Measurement error. l True score – is the score that the participant would have obtained if our measure were perfect.

Sources of measurement errors 1. 2. 3. 4. 5. Transient states • Mood, health, anxiety Stable attributes • Suspicious participant may distort their answers Situational factors • Weather outside, baseball game. Characteristics of the measure • E. g. instruction ambiguity Actual mistakes

Assessing reliability 1. Test-retest reliability • • • Measuring the same thing twice. Reliability = correlation ( r) between results of the first and the second measurements. High reliability >. 70

Assessing reliability 1. Test-retest reliability • Problems • Memory • Experience

Assessing reliability l Interitem Reliability - Measure of consistency among the items on a scale. 1. Item-total correlation q. For each item how it is correlated with the sum of other items. >. 30 2. Split-item reliability q. Divide the items on the scale into 2 sets and test the correlation (instead of test-retest). 3. Cronbach’s alpha coefficient q. Average of all possible split-half reliabilities.

Benevolent sexism scale: 1 (disagree) – 7 (agree) 1. 2. 3. 4. 5. 6. Women should be cherished and protected by men. Women, compared to men, tend to have a superior moral sensibility. Men should be willing to sacrifice their own well-being in order to provide financially for the women in their lives. Many women have a quality of purity that few men possess. A good woman should be set on a pedestal by her man. Men are complete without women.

Made-up table of item-total correlations Item # r 1 . 7 2 . 5 3 . 9 4 . 6 5 . 7 6 . 4

Made-up table of item-total correlations Chronbach α =. 85 Item # r Chronbach α (without the item) 1 . 7 . 8 2 . 5 . 8 3 . 9 . 7 4 . 6 . 8 5 . 7 . 8 6 . 4 . 9

Assessing reliability l Interrater reliability – consistency between two or more raters or judges who observe the same behavior. • High reliability >. 70

Increasing the Reliability Measures 1. 2. 3. 4. Standardize administration of the measure • Same test conditions Clarify instructions and questions. • • To reduce ambiguity and misinterpretations. Pretest questionnaires if possible. Train observers. • To increase interrater reliability. Minimize error in coding data.

Validity l If the measurement actually measures what it is supposed to measure • Different from reliability • Same measure maybe valid for one purpose and invalid for another one.

Assessing validity 1. Face validity – if a measure appears to be valid. • • • Does not mean actual validity. E. g. SAT reading comprehension test • Does it measure reading comprehension or common sense? (Katz et al. , 1990) Affect motivation to participate?

Assessing validity 2. Construct validity n Relation to other measures. § n Convergent validity § High correlation with conceptually relevant measures. Discriminate validity • Low correlation with conceptually unrelated constructs

Assessing validity 3. Criterion-Related validity – the correlation between the measure and some current behavior. • • E. g. IQ and GPA Doctor’s productivity • Peer evaluation • Patient evaluation

Assessing validity 3. Predictive validity – the ability of a measure to predict a certain behavior/situation in a future. • • • E. g. SAT and GPA or GPA and after-college salary. Doctor’s productivity ?

Reliability and Validity l If reliable l If not reliable • May be valid or not. • Not valid

Threats to measurement validity l Using non-validated measures l Solution • Validate the measure • Use pre-validated measures

Threats to measurement validity l Loose connection between theory and method. • Disagreement between conceptional and operational definitions. • E. g. putting more pepper as a measurement of aggression? • Solution • Validate your measure with previous measurements

Threats to measurement validity l Social desirability (evaluation apprehension) – Desire to look “normal” or to be judged favorably by another person (including the experimenter). l Solutions • • Anonymity Ask indirect questions • “How many drinks an average college student have during a party? ”

Threats to measurement validity l Yes-bias Extreme-score bias l Solution l • Reverse score. • Z-transformation within an individual.

Threats to measurement validity l l Testing effects • Most participants perform better on a test of personality/behavior/IQ measure the second time they take it. Reasons • • Learning (e. g. IQ test) Practice (e. g. physical skills) Learn the test goal (e. g. personality test) Attitude polarization • Thinking about their attitudes

Threats to measurement validity l l Testing effects Solutions • Control group • No pretest • Long waiting period

Validity of experiment l Internal validity • Extent to which a study provides evidence of a cause-effect relationship between the variables. l External validity • The ability to generalize results of the experiment.

Internal validity l 3 conditions to determine causality l Low internal validity – the conclusion that A affects B is wrong. • Covariation • Temporal sequence • No confounds

Threats to internal validity l Role demands – participants’ expectations to what an experiment requires them to do • Good-subject tendency • E. g. hypnosis and antisocial acts • Participants reactance • E. g. What is the weather today?

Threats to internal validity l Role demands l Solution • Cover story • E. g. Independent studies • Add non-relevant tasks, items (For measurements)

Threats to internal validity l Experimenter bias l Solution • E. g. Gratitude study • Double-blind

Threats to internal validity l Hawthorne effect – Increases in productivity that occur when participants know they are being studied. • Workers responded to any change in working conditions by working harder than usual. l Solution • Control group

Common Threats to Internal Validity of Quasi-experiments l l l l History • Something occurred between the pretest and posttest. • Normal time changes • If extreme scored Ss. were selected. • Pretest affects the posttest results • Comparison groups differed from the beginning • • Only most motivated participants stay Only participants who experience less adverse effects of treatment stay Maturation Regression to the mean Pretest sensitization Selection bias Local history Contemporary history Attrition/mortality

External validity l How well the findings of an experiment generalize to other situations or populations.

Threats to external validity l Other subjects l Other times Other settings l • Sampling/selection bias

Threats to external validity l Sampling bias • Motivated volunteers • Those available (at home, have phone)

Threats to external validity l Other setting • Artificial experimental environment

External validity l l External validity - the ability to generalize results of the experiment. Tight control - highly specific and artificial situation -> less external validity. Internal validity External validity

l You are a researcher. In your experiment, you assign the first 20 people in your study to the experimental condition and the second 20 people to your control condition. This could pose a threat to: • • Internal validity Reliability External validity Construct validity

l Saying that some measure is ____ definitely means it is also _____. • valid, reliable • reliable, valid • nominal, numerical • observational, self-report • none of the above

l An experimenter wants to examine if a new behavioral intervention program increases compliance among hypertension patients. For this purpose she recruits hypertension patients with low medication compliance and tests their compliance before and after the intervention. What are the potential threats to internal validity: • • • Regression to the mean Maturation History Pretest sensitization All of the above

Find a threat/threats to internal validity l The Alzheimers Center wants to evaluate the effectiveness of their support groups for caregivers of individuals with Alzheimers Disease. The caregivers are given the choice when they first come to the center as to whether they want to join these support groups. The center gives a stress measure to the caregivers that attend these weekly meetings, once they have attended meetings for three months. They also administer the same stress measure to the caregivers who have not attended the support groups, as a control group. Both groups of caregivers are married to the person with Alzheimers disease and both groups have been involved with the center for the same length of time

Any questions?