Introduction to the Validation Phase Rel Ex Relating

  • Slides: 14
Download presentation
Introduction to the Validation Phase Rel. Ex Relating language examinations to the Common European

Introduction to the Validation Phase Rel. Ex Relating language examinations to the Common European Framework of Reference for Languages José Noijons APEOICVA/ECML Valencia, 27 -28 March 2009

Rel. Ex Suggested Linking Procedures in the Manual • Familiarisation with the CEFR •

Rel. Ex Suggested Linking Procedures in the Manual • Familiarisation with the CEFR • Linking on the basis of specification of examination content • Standardisation and Benchmarking • Standard setting • Validation: checking that exam results relate to CEFR levels as intended

What is validity? Rel. Ex Validity refers to the degree to which evidence and

What is validity? Rel. Ex Validity refers to the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests. Although classical models divided the concept into various "validities, " such as content validity, criterion validity, and construct validity, the modern view is that validity is a single unitary construct. (Wikipedia) In simpler terms: does the test measure what it intends to measure?

Rel. Ex Aspects of validity • • • Content Validity Operational validity: pilots and

Rel. Ex Aspects of validity • • • Content Validity Operational validity: pilots and pretests Psychometric aspects Procedural validity of standardization Internal validity of standard setting External validation

Rel. Ex Content validity • Does the test accurately reflect the syllabus on which

Rel. Ex Content validity • Does the test accurately reflect the syllabus on which it is based AND reflect the descriptors in the CEFR? • Does the content specification reflect all areas to be assessed in suitable proportions?

Quality criteria for items Rel. Ex An item must be • • • relevant

Quality criteria for items Rel. Ex An item must be • • • relevant at intended level specific objective acceptable transparent efficient in correct language in a clear lay-out These criteria will contribute to the validity and reliability of a test.

Rel. Ex Quality criteria for items Relevance • Is the item addressing the intended

Rel. Ex Quality criteria for items Relevance • Is the item addressing the intended knowledge or skill at the intended CEFR level? • Does the item not test other knowledge and abilities than the intended ones (e. g. reading skill, general intelligence, knowledge of grammar)? How to realize: • Refer to specific CEFR descriptors for each skill • Use test matrix (or test grid) • Relate questions with purpose of the test • Make items that are recognizable for the student

Quality criteria for items Rel. Ex At intended CEFR level • Is the question

Quality criteria for items Rel. Ex At intended CEFR level • Is the question appropriate for the CEFR level ? • Does the question make a correct selection between those who know and those who do not know? How to realize: • Make experts work in a team • Assign screeners • Avoid manipulation of wording to influence level of difficulty • Avoid unnecessary information • Use data analysis (through pretesting or afterwards)

Quality criteria for items Rel. Ex Transparent • Does the student know how many

Quality criteria for items Rel. Ex Transparent • Does the student know how many answers, details or arguments are expected? • Does the item relate to what students expect (syllabus, preparatory work)? • Do students know the maximum score for the item? • Is the item format known to the students? How to realize: • Use clear instructions for student • Use clear terminology in line with syllabus and related tests • Indicate maximum score for an item • Use item formats or types students have been acquainted with

Validation of Standard Setting • Rel. Ex • • Has the procedure of standard

Validation of Standard Setting • Rel. Ex • • Has the procedure of standard setting had the effects as intended: was the training effective, did the judges feel free to follow their own insights? These are questions of procedural validity. Are the judgments of the judges to be trusted: – Are judges consistent within themselves? – Are judges consistent with each other? – Is the aggregated standard to be considered as the definite standard? These questions and their answer constitute the internal validity of the standard setting. Are the results of the standard setting – allocating students to a CEFR level on the basis of their test score – trustworthy? The basic answer to this question comes from independent evidence which corroborates the results of a particular standard setting: empirical validity.

Equivalence & Equating Rel. Ex How to make sure that exams are equal across

Equivalence & Equating Rel. Ex How to make sure that exams are equal across • sittings • years • languages regarding • content • CEFR level

Maintaining equivalence (1) Rel. Ex How to make sure that exams are equal across

Maintaining equivalence (1) Rel. Ex How to make sure that exams are equal across • sittings • years • languages? Starting point: • Syllabus approved by the Generalitiat – global description per subject of the content of the exam

Maintaining equivalence (2) Regarding • content • CEFR level Rel. Ex Through content specification

Maintaining equivalence (2) Regarding • content • CEFR level Rel. Ex Through content specification in an exam model • • • Number of items and examination time Item format: MC, open items knowledge, skills etc. Domains / topics Test grid

Maintaining equivalence (3) How to realise equivalence in difficulty? Rel. Ex Assuming that the

Maintaining equivalence (3) How to realise equivalence in difficulty? Rel. Ex Assuming that the overall achievements of the group of candidates in subsequent years are comparable: • adaptation of standards • equalizing the mean score • equalizing the percentage of pass/fail