Understanding the Work of the HKEAA Mark Processing

Understanding the Work of the HKEAA: Mark Processing and Grading 1

Mark processing Some important steps after completion of marking: – Equating of marks of optional parts / papers – Moderation of SBA submitted by schools marks 2

Graded Approach for English Language Papers 1 & 3 3

Why do we need a graded approach? • Adopted in Paper 1 (Reading) and Paper 3 (Listening and Integrated Skills) • Compulsory Part A and either Part B 1 (easier) or Part B 2 (more difficult) • Cater to candidates with a wide range of ability 4

What are the advantages? • No labeling effect • No need to extend examination time or reduce coverage of the curriculum • Candidates need not indicate choice at registration so possible to catch up before the examination • Statistical equating to ensure that candidates receive comparable scores on the same scale whether they opt for the easier or more difficult part 5

Why is there a capping policy? • Candidates attempting Parts A and B 2 will be able to attain the highest level of performance (Levels 5 and 5* / 5**) • Candidates who attempt Parts A and B 1 will only be able to attain up to Level 4 in the relevant component • Reason for capping policy is that items in B 1 do not assess skills covered by Level 5 descriptors 6

Why is equating necessary? • Easier to get high marks in Part B 1 than in Part B 2 because the items are easier • 1 mark in B 1 is not equal to 1 mark in B 2 • Need to convert the marks to the same scale • Use the marks of Part A as the mediator because all candidates do Part A 7

Example • 100 candidates got 36 marks in Part A • 50 of them did B 1 and 50 did B 2 • The 50 B 1 candidates got 40 marks in B 1 • The 50 B 2 candidates got 32 marks in B 2 • 40 marks in B 1 = 32 marks in B 2 8

What is equipercentile equating? • Large candidature for English Language (e. g. ~ 80, 000 in 2013) • All candidates do Part A and can be rank ordered based on their marks in Part A • B 1 marks are statistically adjusted for difficulty to the same level as B 2 marks using the percentile ranks of candidates’ performance in Part A and Part B 1/B 2 9

Equipercentile method of equating B 2 candidates’ ranking in Part A B 1 candidates’ ranking in Part B 1 50 th% 40 th% 40 50 th% 36 36 32 10

Equipercentile equating http: //www. hkeaa. edu. hk/Doc. Library/Media/Le aflets/HKDSE_SRR_A 4_Booklet_Jun 2011. p df 11

Is there a fixed conversion factor? • The examination papers for each year will be different from the previous year’s • Therefore no fixed conversion factor • A conversion table will be produced every year for that particular examination paper based on the performance of the candidates in Parts A, B 1 and B 2 12

How to choose between B 1 and B 2? Levels 4 -5 Levels 3 -4 Levels 1 -3 Parts A & B 2 (more difficult part) Parts A & B 1 or Parts A and B 2 Parts A and B 1 (easier part) 13

How is the paper mark calculated? • For Part B 1 candidates: Part A marks + Part B 2 equivalent marks • For Part B 2 candidates: Part A marks + Part B 2 marks • Therefore, choosing B 1 or B 2 makes no difference for mid-range candidates 14

Moderation of SBA Marks 15

Some queries/concerns • Why moderate schools’ SBA marks? • Is it fair to moderate students’ SBA marks based on exam results? 16

Why moderation? • Teachers know their students well and thus are best placed to judge their relative performance within a school • However, they are not necessarily aware of the standards of performance across all schools • Moderation ensures the consistency of assessment standards across schools in order to achieve across-school comparability 17

What are the principles of SBA moderation? • Accept teacher judgments of relative student performance within a school (internal standardisation, student queries) • Moderation is done on school basis (not on individual basis) to standardise across schools 18

Methods of SBA moderation A. Statistical – The distribution of a school’s SBA marks for a subject is re-shaped taking into consideration school’s pattern of mark distribution for that subject’s examination papers B. Expert judgment – A team of experienced, subject experts reviews samples of schools’ SBA work for a subject 19

Moderation methods Expert Judgment Moderation Statistical Moderation for subjects with a small candidature or that involve outcomes that are very different from those assessed through the public examination for subjects in which the SBA and the public examination share a substantial portion of common assessment objectives Design and Applied Technology English Language, Chinese Language Liberal Studies, History, Chinese History Biology, Chemistry, Physics Science (Combined Science/Integrated Science) Information and Communication Technology Visual Arts 20

Features of statistical moderation • Internal changed ranking would not be • The statistical moderation method would consider the empirical correlation between examination results and SBA results • Eliminate the effect of strictness and leniency 21

Features of statistical moderation • Sample review to help identify outlier schools • For outlier schools, adjustment will be carried out, if necessary, by considering a number of factors, such as moderated marks based on examination results and sample review findings so as to reflect the actual SBA performance 22

Expert judgment moderation • Review of samples of students’ work • Determination of school performance level on SBA and individual students’ moderated scores • Detection of cases with extreme adjustment and follow-up actions 23

Review of samples of students’ work • Stratified random sampling within each school • District Coordinators / External Assessors review samples with reference to the stipulated assessment criteria to determine the group performance level • More samples from a school may be collected if necessary 24

Result of moderation in 2012 HKDSE 70. 0% 60. 0% 50. 0% 40. 0% 30. 0% 2009 CE 2010 CE 2012 DSE 10. 0% much lower slightly lower as expected slightly higher much higher Mean SBA Adjustment 25

Result of moderation in 2012 HKDSE 70. 0% 60. 0% 50. 0% 40. 0% 30. 0% 2009 CE 2010 CE 2012 DSE 10. 0% much lower slightly lower as expected slightly higher much higher Mean SBA Adjustment 26

Result of moderation in 2012 HKDSE In general, most teachers are aware of the standards and their SBA marks need very little adjustment • Small deviation from the standard is expected due to normal variation of experience and subjective judgment: no systemic bias observed • Moderation provides assurance of fairness to students, parents, and • schools 27

Reference • HKDSE Examination: Moderation of School-based Assessment Scores http: //www. hkeaa. edu. hk/Doc. Library/SBA/H KDSE/HKDSE-SBA-Moderation. Booklet_r. pdf 28

Standards-referenced Reporting 29

Some queries/concerns • Why the grade awards are different across subjects? • Is this unfair to my subject? 30

科目 Subject 資訊及通訊科技 Information and Communication Technology 通識教育 Liberal Studies 英語文學 Literature in English 必修部分 Compulsory Part 出席 人數 No. Sat 考生考獲各等級的百分率 Percentage of levels awarded 中文作答 Chinese Version % 5** 5*+ 5+ 4+ 3+ 2+ 1+ U 7 918 61. 6 0. 7 2. 9 7. 2 25. 2 50. 9 77. 5 93. 0 7. 0 69 721 88. 0 0. 8 3. 0 7. 7 35. 7 72. 5 90. 8 98. 9 1. 1 2. 7 10. 4 25. 9 57. 6 80. 5 92. 5 96. 9 3. 1 451 - 69 299 49. 5 1. 1 4. 8 12. 1 34. 1 57. 1 79. 7 91. 7 8. 3 延伸部分（微積分與統計） Extended Part (Calculus and Statistics) 7 347 27. 1 1. 4 5. 9 15. 1 43. 4 69. 7 86. 9 95. 1 4. 9 延伸部分（代數與微積分） Extended Part (Algebra and Calculus) 8 091 18. 1 3. 0 12. 0 31. 4 56. 1 79. 8 92. 5 98. 0 2. 0 音樂 Music 232 8. 6 1. 7 7. 8 19. 0 60. 8 85. 8 97. 8 99. 6 0. 4 體育 Physical Education 681 93. 5 0. 1 0. 4 1. 6 9. 3 37. 7 63. 9 88. 7 11. 3 15 233 31. 4 2. 4 9. 6 23. 8 48. 8 73. 0 88. 8 97. 0 3. 0 數學 Mathematics 物理 Physics 31

Standards-referenced reporting system 24 NSS Subjects Ø Ø Descriptors and exemplars to indicate standards required for each level Facilitate users, including tertiary institutions and employers, to set appropriate entrance/ job requirements 32 32

Standards-referenced reporting system 24 NSS Subjects Ø Adopted to report candidates' assessment results in all 24 HKDSE Category A subjects Ø SRR adopted for HKCE English Language and Chinese Language since 2007 33

Grading in the HKDSE Cut scores and grade levels determined so that each level represents a fixed standard of achievement from year to year – Methodology combines psychometric measurement and expert judgment – Psychometric measurements: i) Monitoring tests for the core subjects – ii) Group ability index (GAI) for elective subjects 34

Grading in the HKDSE Expert judgments made independently by experienced examiners and experts based on the actual performance of candidates on the exams, using the level descriptors and library scripts for reference – Psychometric cut score recommendations are derived for the reference of the expert panel – No pre-set quota for each level /subject – 35

Standards maintenance in 2013 Four core subjects (CEML) Additional reference made to: – statistical data from the monitoring tests which were administered to a representative sample of 2012 and 2013 DSE school candidates 36

Using monitoring tests to maintain standards Statistically-generated cut scores for reference of Expert Panel 37

Group Ability Index (GAI) for elective subjects • Different Category A elective subjects may attract students of different abilities • Compute an indicator (index) to reflect overall performance in core subjects (ability) of all candidates taking the subject (group), taking into account how the performance in the elective correlates with each of the core subjects – Provided as supplementary statistics, in addition to the level descriptors, library scripts and live scripts, to the subject experts for their expert judgment. Final decision based on performance as reflected in the candidates’ scripts 38

Group Ability Index The GAI of Subject X for a certain level (e. g. Level 3) or above is defined as: b n/N Coefficients obtained by Proportion of candidates in a regressing scores of Subject X on scores of CEML group attained a particular level (e. g. Level 3) for a core subject 39

Grading Methodology and Procedures

Who are involved in grading • Each subject sets up its expert panel which comprises: the chief examiner(s) of different papers, – assistant examiners/markers, if necessary; – SBA supervisors/chief project examiner, if applicable, and – subject manager – 41

Overview of grading procedures 1. Familiarisation with standards 2. Sample script selection 3. Standardisation 4. Post-marking exercise 5. Expert Panel Meeting 6. Internal Meeting 7. Public Examinations Board Meeting 42

4. Post-marking exercise 1. Determine preliminary cut scores ranges for all levels for each exam paper (incl. SBA) with reference to: – – – the difficulty of live question paper and requirements of marking scheme inter-paper correlations, mean and standard deviation of current year’s papers in comparison to previous years item statistics, if available impression of candidates’ performance when marking live scripts or reviewing students’ SBA work level descriptors and borderline performance samples in the previous exams 43

5. Expert panel meeting Ø Objective: Ø Apply expert judgment to arrive at a set of recommended cut scores for different levels for the subject and its components, if applicable Ø Make reference to marked live scripts falling within / around the recommended cut score ranges and exam statistics 44

Expert panel meeting (cont’d) • Also refer to cut scores suggested by statistical means: for core subjects, the suggested cut scores are derived from the results of the monitoring tests (2013 data in comparison with the 2012 data) – for elective subjects, the suggested cut scores are derived from GAI – 45

科目 Subject 資訊及通訊科技 Information and Communication Technology 通識教育 Liberal Studies 英語文學 Literature in English 必修部分 Compulsory Part 出席 人數 No. Sat 考生考獲各等級的百分率 Percentage of levels awarded 中文作答 Chinese Version % 5** 5*+ 5+ 4+ 3+ 2+ 1+ U 7 918 61. 6 0. 7 2. 9 7. 2 25. 2 50. 9 77. 5 93. 0 7. 0 69 721 88. 0 0. 8 3. 0 7. 7 35. 7 72. 5 90. 8 98. 9 1. 1 2. 7 10. 4 25. 9 57. 6 80. 5 92. 5 96. 9 3. 1 451 - 69 299 49. 5 1. 1 4. 8 12. 1 34. 1 57. 1 79. 7 91. 7 8. 3 延伸部分（微積分與統計） Extended Part (Calculus and Statistics) 7 347 27. 1 1. 4 5. 9 15. 1 43. 4 69. 7 86. 9 95. 1 4. 9 延伸部分（代數與微積分） Extended Part (Algebra and Calculus) 8 091 18. 1 3. 0 12. 0 31. 4 56. 1 79. 8 92. 5 98. 0 2. 0 音樂 Music 232 8. 6 1. 7 7. 8 19. 0 60. 8 85. 8 97. 8 99. 6 0. 4 體育 Physical Education 681 93. 5 0. 1 0. 4 1. 6 9. 3 37. 7 63. 9 88. 7 11. 3 15 233 31. 4 2. 4 9. 6 23. 8 48. 8 73. 0 88. 8 97. 0 3. 0 數學 Mathematics 物理 Physics 46

Reference • Grading Procedures and Standards- reference Reporting in the HKDSE Examination http: //www. hkeaa. edu. hk/Doc. Library/Media/Leaflets/HKD SE_SRR_A 4_Booklet_Jun 2011. pdf 47

Reporting of Results

Reporting of HKDSE results • For Category A subjects, results reported mostly on subject basis • Component results reported for: Chinese Language – English Language – Combined Science – 49

Reasons for component reporting • Allow end users to understand the achievement of candidates in different areas of the subject e. g. reading, writing, listening and speaking skills in language subjects – different half subjects in Combined Science – • For selection for studies or employment 50

FAQ on component result and subject result • Why is my subject level lower than the average of individual component levels? 51

The facts • Subject level is based on the overall subject mark which is the combination of all component marks • NOT a simple averaging of component levels achieved 52

The facts • Each level covers a range of marks; a candidate may have a score close to the lower/higher end of the range • Components of a subject carry different weightings; the marks for each component go through a weighting procedure before combining them to arrive at the subject mark 53

Example: HKDSE English Language • Comprises four components: – Component 1: Reading (20%) – Component 2: Writing (25%) – Component 3: Listening and Integrated Skills (30%) – Component 4: Speaking + SBA (25%) 54

Level achieved Student A Student B 5 4 3 * Reading (20%) Writing (25%) Listening (30%) Speaking (25%) Subject 5 5 4 5 3 5 4 5 * * * 2 1 55

Thank You 56