Reliability Validity Levels of Measurement Scales How we

Reliability Validity Levels of Measurement Scales

How we figure out what to measure • Conceptualization – Process of taking a construct and refining it by giving it a conceptual or theoretical definition – Research focusing on college students • In Ohio? What region? Age? Major? • Operationalization – Links a conceptual definition to a specific set of measurement techniques

Coming up with a measure • • • Remember the conceptual definition Keep an open mind Borrow from others Anticipate difficulties Don’t forget units of analysis

Empirical Hypothesis • The degree of association • How well operationalized variables are associated (or not) with the concept construct determines the hypothesis

Reliability • Reliability means dependability or consistency • Same thing occurs over and over under same conditions • A scale, for example • How dependable is the study? • Is the study consistent, or does it yield wide varying results? • Can the study be replicated?

Reliability • Measurement directly affects the quality of conclusions. • Care is needed to make sure that results are not corrupted by improper measurement. • The operational definition of a concept should have a precise meaning: – The terms by which you measure a concept should be explicit.

Reliability • Reliability and validity are the biggest threats to proper measurement. • Reliability is the extent to which an experiment, test, or any measuring procedure yields the same results on repeated trials. • Do you get the same result every time?

Reliability • Three tests of reliability: – Test-retest method • Applying the same test to the same observations after a period of time and then comparing the results of the different measurements – Alternative form method • Two different measures of the same concept administered to the same respondents at different times before the scores are compared – Split-halves method • Divide a multi-item measure into two measures with both of the new measures applied at the same time

Improving Reliability • Clearly conceptualize constructs • Increase level of measurement • Use multiple indicators of a variable – Triangulation • Use pretests, pilot studies, and replication

Validity • A valid measure is one that measures what it is supposed to measure, in other words, the degree of correspondence between the measure and the concept it is thought to measure. • Four tests of validity

Validity • Truthfulness • Refers to the match between a construct and a measure • Want it to be valid for a particular purpose and definition • How good is the measure? • Is the data measured correctly? • Is the data analyzed correctly (statistical)?

Internal Validity • Are there errors as a result of the internal design of the study? • Are there errors as a result of the controls? • Internal validity problems can occur from a flawed survey along with a multitude of other factors

External Validity • Can your experiment’s findings be generalized? • External Validity questions are evident in every study; however, methods exist to keep external validity high and the number of external flaws low

Types of Validity • Face validity – Judgment that the indicator really measures the construct • Content validity – Does your measure represent the full content of a defintion? • Criterion validity – Use some standard or criteria to indicate a construct accurately • Concurrent validity – Indicator must be associated with a preexisting indicator judged to be valid • Predictive validity – Indicator predicts future events that are logically related to a construct

Types of Validity • • • 1) If I create a new test of mathematical ability for high school students and test it by having high school math teachers look at it and tell me if it seems appropriate, I am measuring for __________ validity. 2) If I am examining an individuals ability to cope with stress and have three attributes I am particularly interested in and I am checking to see if my construct hits on all three attributes, I am measuring for ____________ validity. 3) If I create a new test for cognitive recognition and students that score high on it also score high on previously existing tests for cognitive recognition, I have demonstrated __________ validity. 4) If I compare my measure for testing the potential to suffer from childhood diabetes with a previously used test, I am looking for _____________validity. 5) If I create a new test of intelligence and students that score high on it also do better in college than those who score lowly, I have shown __________ validity.

Validity • Tests of validity are not as good as tests of reliability. • Reliability is easy to demonstrate through some form of repeated trials. • Validity is more difficult because we can never be sure about the true value of a concept: – Especially true with abstract concepts

Validity • Whereas a valid measure is reliable (because if truly valid, it will measure the concept correctly every time), a reliable measure is not necessarily valid. • The measure could be measuring the concept incorrectly in a consistent way.

Relationship between reliability and validity

Levels of Measurement • The level of measurement of a variable describes – The amount of precision associated with a variable – The mathematical properties of the variable • Both precision and mathematical properties increase as you increase the level of measurement from nominal to ratio.

Levels of Measurement • Continuous v. discrete variables – Continuous • Have an infinite number of values or attributes that flow directly along a continuum • Temperature, age, income, crime rate – Discrete • Relatively fixed set of separate values or attributes • Gender, religion, marital status

Nominal Level • Only reports a difference • Candidate preference, religious preference, Yes/No, etc. • Discrete Variables

Levels of Measurement • The level of measurement of a variable describes – The amount of precision associated with a variable – The mathematical properties of the variable • Both precision and mathematical properties increase as you increase the level of measurement from nominal to ratio.

Ordinal Level • Rank ordered • Grades, opinion • • Strongly Agree Disagree Strongly Disagree

Levels of Measurement • At the ordinal level, categories may be ranked in order in addition to indicating a difference between categories. • Example: Please indicate the highest level of education you reached (elem. , high, college, more). • Precision: A little more precision and can be used with more statistical tools

Interval/Ratio Level • A specified distance • Interval does not contain a true zero point (ratio does) • Interval: IQ, SAT • Ratio: years of school, income

Levels of Measurement • The interval level includes all of the information of the preceding levels and adds meaningful intervals between values of the variable but does not use a meaningful zero. • Example: What did you score on the SAT? • Precision: More precision and can be used with most statistical tools

Levels of Measurement • The ratio level adds a meaningful zero to the interval level. • Example: How many years of education? • Precision: Most precision and can be used with most statistical tools

Scales • Some concepts can be captured with a single question. • More complex concepts may require a multi-item measure consisting of several questions that capture different components of the concept and increase validity.

Scales • Summation index: – Combines the scores on multiple questions to create one single measure of a concept • Likert scale: – Uses only select questions from an index that differentiate between different respondents to create a single score for each respondent

Scales • Guttman scale: – Has answer choices arranged in an ordinal manner; respondents will agree with each of the lower-ranked answers if they agree with a higher-ranked answer • Factor analysis: – Allows researchers to uncover patterns across related measures to create summary variables that represent different dimensions of the same concept

Mutually Exclusive • “One and only. ” • One may only fit the criteria of one category • Ex: Religion: Christian, non-Christian, Jewish, Buddhist NOT MUTUALLY EXCLUSIVE

Exhaustive • All cases fit into one category • Ex: If the Election were today, would you support Sherrod Brown, the Democrat, or Mike De. Wine, the Republican or neither? • NOT EXHAUSTIVE

Missing Data • No survey is perfect, and certain questions will be left unanswered or completely skipped • Remedy? • A “catch all” category and/or a way to factor out missing data • Yet, missing data can still mislead a study

One Last Thing • Feeling Thermometers • Likert Scales • Response set problem – How do we fix this? ? ?