TSL 3123 LANGUAGE ASSESSMENT SCORING GRADING AND ASSESSMENT

TSL 3123 LANGUAGE ASSESSMENT SCORING, GRADING AND ASSESSMENT CRITERIA (3 hours) 1

LECTURE’S OBJECTIVES • Identify and differentiate the different approaches used in scoring ü Objective ü Holistic ü Analytic • Use the different approaches used in scoring in assessing language 2

OBJECTIVE APPROACH 3

OBJECTIVE SCORING • relies on quantified methods of evaluating students’ writing • analyzing occurrences of linguistic features. • The objective scoring used were categorized as word and sentence counts, syntactic complexity with no attention to error, and syntactic complexity with attention to error. • objective scoring that took error into account were able to discriminate between three essay quality levels. 4

OBJECTIVE SCORING A sample of how objective scoring is conducted is given by Bailey (1999) • Establish standardization by limiting the length of the assessment: Count the first 250 words of the essay. • Identify the elements to be assessed: Go through the essay up to the 250 th word underlining every mistake – from spelling and mechanics through verb tenses, morphology, vocabulary, etc. Include every error that a literate reader might note. • Operationalise the assessment: Assign a weight score to each error, from 3 to 1. A score of 3 is a severe distortion of readability or flow of ideas; 2 is a moderate distortion; and 1 is a minor error that does not affect readability in any significant way. • Quantify the assessment: Calculate the essay Correctness Score by using 250 words as the numerator of a fraction, and the sum of error scores as the denominator: The denominator is the sum 5 of all the error scores.

HOLISTIC SCORING 6

HOLISTIC SCORING • A rater makes an overall judgement about the quality of the performance (to rate the overall proficiency level) • A single score is awarded to the writing • Normally this score is on a scale of 1 to 4, or 1 to 6, or even 1 to 10. (Bailey, 1998 : 187). • Each score on the scale will be accompanied with general descriptors of ability. • The example below is of a holistic scoring scheme based on a 6 point scale. • It includes broad descriptors of what a student’s 7 essay reflects for each band.

HOLISTIC SCORING METHOD 8

HOLISTIC SCORING • Another type of scoring related to the holistic approach is primary trait scoring (Bailey, 1998). • A particular functional focus is selected based on the purpose of the writing and grading is based on how well the student is able to express that function. • For example, if the function is to persuade, scoring would be on how well the author has been able to persuade the grader rather than how well organised the ideas were, or how grammatical the structures in the essay were. • This technique emphasises functional and communicative ability rather than discrete linguistic ability and accuracy. 9

HOLISTIC SCORING • It is usually used for large scale assessment because raters can be trained and scores can be computed more quickly (Bauer, 1981). • The major disadvantage emerges from the limitations of the single score, which gives no details. • Does not allow raters to distinguish between various aspects of writing such as control of syntax, depth of vocabulary, organization, and so on. • Problematic for second-language writers since different aspects of writing ability may develop at different rates for different L 2 learner. 10

ANALYTIC SCORING 11

ANALYTIC SCORING • A rater assigns a score to each of the component for the task. • Example of analytical scoring approach: Components Weight Content 30 points Organisation 20 points Vocabulary 20 points Language Used 25 points Mechanics 5 points • The points assigned to each component reflect the importance of each of the components. • Analytic scoring is useful in the classroom because the result can help teachers diagnose students strengths and learning 12 needs.

ANALYTICAL SCORING METHOD 13

ANALYTIC SCORING • it provides more useful diagnostic information about students’ writing abilities. • it is easier to train raters to use analytic scoring schemes • The major disadvantage of analytic scoring is that it takes longer than holistic scoring 14

COMPARING THE THREE APPROACHES Scoring Approach Holistic Analytical Objective Advantages Disadvantages ü Quickly graded ü Provide a public standard that is understood by the teachers and students alike ü Relatively higher degree of rater reliability ü Applicable to the assessment of many different topics ü Emphasise the students’ strengths rather than their weaknesses. ü It provides clear guidelines in grading in the form of the various components. ü Allows the graders to consciously address important aspects of writing. ü The single score may actually mask differences ü across individual compositions. ü Does not provide a lot of diagnostic feedback ü Emphasises the students’ strengths rather than their weaknesses. ü Still some degree of subjectivity involved. ü Accentuates negative aspects of the learner’s 15 ü writing without giving credit for what ü Writing ability is unnaturally split up into components.

DESIGNING SCORING RUBRICS FOR YOUR CLASSROOM • Rubrics are rating scales - as opposed to checklists-that are used with performance assessments. • They are formally defined as scoring guides, consisting of specific pre-established performance criteria, used in evaluating student work on performance assessments. 16

DESIGNING SCORING RUBRICS FOR YOUR CLASSROOM • Prior to designing a specific rubric, a teacher must decide whether the performance or product will be scored holistically or analytically (Airasian, 2000 & 2001). • specific performance criteria and observable indicators must be identified as an initial step to development. • the use of a holistic or analytic approach to scoring has several possible implications. ü teachers must consider first how they intend to use the results. If an overall, summative score is desired, a holistic scoring approach would be more desirable. In contrast, if formative feedback is the goal, an analytic scoring rubric should be used 2001). ü Other implications include the time requirements, the nature of the task itself, and the specific performance criteria being 17 observed.

HOLISTIC RUBRICS • Holistic rubrics are customarily utilized when errors in some part of the process can be tolerated provided the overall quality is high (Chase, 1999). • Nitko (2001) states that use of holistic rubrics is probably more appropriate when performance tasks require students to create some sort of response and where there is no definitive correct answer. • Holistic rubrics are also used when the purpose of the performance assessment is summative in nature. • A template for holistic scoring rubrics is presented in Table 1. 18

HOLISTIC RUBRICS 19

ANALYTIC RUBRICS • Analytic rubrics are usually preferred when a fairly focused type of response is required (Nitko, 2001) • For performance tasks in which there may be one or two acceptable responses and creativity is not an essential feature of the students' responses. • A general rule of thumb is that an individual's work should be examined a separate time for each of the specific performance tasks or scoring criteria (Mertler, 2001) 20

ANALYTIC RUBRICS 21

RUBRICS TO GRADES • It is not a good idea to think of rubrics in terms of percentages (Trice, 2000). • For example, if a rubric has six levels (or “points”), a score of 3 should not be equated to 50% (an “F” in most letter grading systems). • The converting process is more a process of logic than it is a mathematical one. • Trice (2000) suggests that in a rubric scoring system, there are typically more scores at the average and above average categories (i. e. , equating to grades of “C” or better) than there are below average categories. 22

RUBRICS TO GRADES • if a rubric consisted of nine score categories, the equivalent grades and categories might look like this: 23

STEPS IN THE DESIGN OF SCORING RUBRICS 24

TUTORIAL TASKS • Based on a stimulus, construct the scoring rubrics in the marking of essay questions to enhance test objectivity. ISL TASK Do some background reading on scoring rubrics. 25

REFERENCES • Brown, H. D. , & Abeywickrama, P. (2010). Language Assessment: Principles and Classroom Practices. New York, NY: Pearson Education. • Brown, H. D. (2007). Teaching by principles: An interactive approach to language pedagogy. Englewood Cliffs, NJ: Prentice Hall Regents. • Chitravelu, Nesamalar, 2005. ELT Methodology: Principles and Practice. Penerbit Fajar Bakti, Sdn, Bhd. • Malaysia Education Blueprint 2013 -2025. 26