UNDERSTANDING RELIABILITY AND VALIDITY How do you know

UNDERSTANDING RELIABILITY AND VALIDITY How do you know if you should trust your research?

DEFINITIONS • Validity • is the degree to which it measures what it is supposed to measure. • Reliability • is the extent to which a measurement gives results that are consistent.

TYPES OF VALIDITY • Construct validity • refers to the extent to which operationalizations of a construct (i. e. , practical tests developed from a theory) do actually measure what theory says they do. • Convergent validity • refers to the degree to which a measure is correlated with other measures that it is theoretically predicted to correlate with.

TYPES OF VALIDITY (CONT’D) • Discriminant validity • tests whether concepts or measurements that are supposed to be unrelated are, in fact, unrelated. • Content validity • is a non-statistical type of validity that involves "the systematic examination of the test content to determine whether it covers a representative sample of the behavior domain to be measured" (Anastasi & Urbina, 1997 p. 114). (For example, does an IQ questionnaire have items covering all areas of intelligence discussed in the scientific literature? )

TYPES OF VALIDITY (CONT’D) • Representation validity • also known as translation validity, is about the extent to which an abstract theoretical construct can be turned into a specific practical test • Face validity • is an estimate of whether a test appears to measure a certain criterion; it does not guarantee that the test actually measures phenomena in that domain. Measures may have high validity, but when the test does not appear to be measuring what it is, it has low face validity.

TYPES OF VALIDITY (CONT’D) • Concurrent validity • refers to the degree to which the operationalization correlates with other measures of the same construct that are measured at the same time. ( eg. this would mean that the tests are administered to current employees should also be correlated with their scores on performance reviews).

TYPES OF VALIDITY (CONT’D) • Predictive validity • refers to the degree to which the operationalization can predict (or correlate with) other measures of the same construct that are measured at some time in the future. (e. g. , hiring test correlated to employee performance when hired.

TYPES OF VALIDITY (CONT’D) • External validity • concerns the extent to which the (internally valid) results of a study can be held to be true for other cases, for example to different people, places or times. In other words, it is about whether findings can be validly generalized. If the same research study was conducted in those other cases, would it get the same results?

CLASSES OF RELIABILITY • Inter-rater reliability • assesses the degree of agreement between two or more raters in their appraisals. • Test-retest reliability • assesses the degree to which test scores are consistent from one test administration to the next. Measurements are gathered from a single rater who uses the same methods or instruments and the same testing conditions. [2] This includes intra-rater reliability.

RELIABILITY (CONT’D) • Inter-method reliability • assesses the degree to which test scores are consistent when there is a variation in the methods or instruments used. This allows inter-rater reliability to be ruled out. When dealing with forms, it may be termed parallel-forms reliability. • Internal consistency reliability • assesses the consistency of results across items within a test.