Validity 1 Agenda Types of validity Approaches to

Validity 1

Agenda Types of validity Approaches to establishing/assessing validity Discussion Practice with predictive validity 2

Types of Validity Content validity (content relevance) Ø Construct validity (construct representation) Ø The content of an assessment should represent the full range of knowledge or abilities that the assessment is intended to measure. The degree to which an assessment can capture the construct of interest that it claims to measure. Predictive validity (criterion-related evidence) Ø The extent to which a score on an assessment predicts scores on other criterion measures. 3

Types of Validity Scoring validity (consistency among measures) Ø The consistency and stability with which an assessment reflects test-takers’ true scores (e. g. , test-retest reliability or parallel forms reliability) Consequential validity (appropriate consequences) Ø Effects of an assessment on institutional system (e. g. , schools or universities) and activities of stake-holders and test-takers (e. g. , teachers or student). 4

Approaches to Validity Unitary Theory of Validity All types of validity subsumed under construct validity (Messick, 1989) Construct validity: evidence that shows that the construct is related to the assessment) Argument-Based Validation Program evaluation background Toulmin's model to evaluate evidence 5

Concerns 1. It is difficult for test developers to afford the cost of hiring scholars to conduct rigorous test validation research. 2. It is challenging to consecutively incorporate results from validation research into refining established language tests because they may change their conventional uses. 3. Teachers often do not learn how to conduct test validation research; their work conditions do not allow them to do so. 4. There is little agreement between language acquisition theories for assessment purposes. *This underscores the importance of validity being contextdependent and applicable only to the purposes for which the assessment is designed* 6

Discussion If you were tasked with creating and maintaining a large-scale language assessment, which types of validity would you prioritize and why? What steps would you take to ensure those types of validity were evaluated and refined? Several scholars (Chapelle, 2012; Kane, 2012) point to the need for third party research and evaluation of language assessment validity. . . How can this feasibly be accomplished? What suggestions do you have for scholars or test developers to improve efforts to objectively validate assessments? 7

Practice 1 Assume that you work in the College of Education at the University of Iowa. You have recently admitted 40 students into a special program that will allow them to obtain a K-12 teaching certificate in 18 months by attending class on weekends. Your charge is to better understand if the traditional GRE exam is the appropriate admissions test for this type of program. Using the GRE dataset (Practice 1), define the following: Predictor variable Criterion variable Relationship between the 2 variables Write a statement to the Dean summarizing your evaluation of the predictive validity of using the GRE for this special program.

Practice 2 Assume that you work in the College of Business at Kirkwood Community College. You have recently admitted 40 students into the MBA program based on their scores on the GMAT and TOEFL tests. Your charge is to better understand which test is a better admissions test for your program. Using the GMAT dataset (Practice 2), define the following: Predictor variable Criterion variable Relationships between the variables Write a statement to the Dean summarizing your evaluation of the predictive validity of using the GMAT or TOEFL for this program. 9