Moderation of Schoolbased Assessment SBA Dr George Pook

Moderation of School-based Assessment (SBA) Dr George Pook Deputy Secretary General, HKEAA School Leaders’ Briefing Session 5 March 2012

1. The Crucial Importance of SBA to the Education Reform Broader range of student skills must be developed (e. g. : inquiry, problem solving, project planning, practical activities, in-depth analysis, reflection, personal interaction) These skills must be reflected in the HKDSE assessment structures (curriculum, pedagogy and assessment closely inter-linked) These skills cannot be properly addressed in a restricted examination environment Teachers best placed to assess student performance on these skills 2

2. Critical Requirement for Manageability of SBA Must be built into ordinary classroom teaching Sufficient control and data collection measures, but without undue administrative burden Holistic view of impact on students and schools 3

3. Critical Requirement for Fairness of SBA Authenticity of student work Moderation of school assessments 4

4. Principles of SBA Moderation Accept teacher judgments of relative student performance within a school (internal standardization, student queries) Moderation done per school standardize across schools to 5

5. Basic Techniques of SBA Moderation A. Statistical The distribution of a school’s SBA marks for a subject is re-shaped to reflect that school’s pattern of mark distribution for that subject’s examination papers Done on a school group basis, not on individual basis B. Expert judgment An experienced, external professional reviews a sample of a school’s SBA work for a subject Based on a comparison of the original marks and this external review, an average mark adjustment trend is calculated and applied to the school’s SBA marks for that subject 6

6. Statistical and Expert Judgment Techniques for Moderation Points to note (1) Both techniques retain the original rank order of the students within a school, unless the school shows exceptional anomalies (2) Neither technique is perfect 7

7. Statistical technique Characteristics: Objective and precise, but does not take proper account of the standard of SBA work done in a school Reasonable statistical basis for large entry subjects, but not reliable for small entry subjects Can be conducted largely through automatic processing, but this cannot happen until after examination paper marking has been completed 8

8. Expert judgment technique Characteristics: Focused on the standard of student work done for SBA, but expert judgment may not be absolutely reliable, even though fully independent Accepts that student SBA performance may not be directly linked to examination paper marks, but depends on a limited sample to construct an adjustment for all candidates Can be conducted well before examination papers are marked, but requires human resource input for the preparation and review of SBA work samples 9

9. Principles Applied to SBA Moderation for HKDSE (1) Use the positive aspects of both techniques combined (2) Rely more on statistical technique when student performance on SBA relates strongly to examination paper performance for a subject; rely more on expert judgment technique when this is not the case 10

10. The Two SBA Moderation Methodologies for HKDSE A. “Statistical Moderation” Apply the normal statistical moderation technique 2) Compare the outcomes per school with expert judgment review of SBA work samples 3) Where there are major discrepancies (“outliers”), strong consideration given to apply a moderation adjustment closer to the expert judgment 4) Currently applied to most HKDSE subjects 1) 11

10. The Two SBA Moderation Methodologies for HKDSE B. “Expert judgment moderation” 1) Experienced professionals re-mark each sample of SBA work 2) These re-marks determine the group performance level of the school for that subject 3) Statistical techniques derived from general trends will be used to improve the reliability of the sampling process, and applied to determine the moderated mark for each student 4) Currently applied to Visual Arts and Design & Applied Technology in HKDSE 12