THE FEASIBILITY OF APPLYING STATISTICAL MODERATION IN PRACTICAL

THE FEASIBILITY OF APPLYING STATISTICAL MODERATION IN PRACTICAL COMPONENTS USING EXTERNAL EXAMINATIONS AS THE ANCHOR Advancing learning, certifying your future

Background • Moderation only applied to practical subjects • Their syllabuses contained practical objectives • Practical subjects differ in nature, e. g. HE, Agriculture, Music, PE, D&T, Art, hence different modes of assessment • Different format of moderation depending on the subject • Advent of Outcome-Based Assessment requires SBA in all subjects Advancing learning, certifying your future

Background Cont’d • e. g BGCSE D&T, employs statistical moderation techniques in its moderation, without necessarily linking it to External exams (Kempa, 1986) • (y = -1. 211 +. 508*Sch Mark). • In some subjects visit schools (JC Agric), in some artifacts centrally marked (BGCSE Agric) • Visiting moderation is expensive, and resources are limited. • Stats moderation is cheap Advancing learning, certifying your future

STATISTICAL MODERATION • Statistical moderation tries to adjust the level and spread of each school’s assessments of its students in a particular study, to match the level and spread of the same students’ scores on a common external examination. • Thus statistical moderation is applied to two different marks measuring the Same Learning Outcomes. Advancing learning, certifying your future

Stats Mod Cont’d • Thus students’ External Score should be a valid predictor of coursework achievement. • The Coursework score must itself be valid • Hence strong correlation between the two!!!!. Advancing learning, certifying your future

Subjects Components JC Subject 1 PP 1: MC – 35% PP 2: SASI – 40% PP 3: SBA – 25% JC Subject 2 PP 1: MC – 30% PP 2: Prac Exam – 45% PP 3: SBA – 25% JC Subject 3 PP 1: Listening – 25% PP 2: SASI – 25% PP 3: Prac Exam – 30% JC Subject 4 PP 1: MC – 20% PP 2: SASI – 40% PP 3: SBA – 40% BGCSE Subject 5 PP 1: Pract Exam – 50% PP 2: Pract Exam – 50% PP 4 or 5: SBA– 50% BGCSE Subject 6 PP 1: Written Exam – 50% PP 3: Indiv Study – 30% PP 2: SBA – 20% Advancing learning, certifying your future PP 4: SBA – 20% PP 3: Pract Exam – 50%

Methodology • A mixed methodology employing Correlational study design as baseline and Design-Based Research design as developmental • Purposive sampling of Subjects (3 from JC and 2 from BGCSE) and schools. • Teacher coursework scores correlated with all other scores Advancing learning, certifying your future

Methodology Cont’d • Agreement (Concordance)/Inter-Rater Reliability between moderator and Teacher scores for coursework scores • National descriptive statistics and Schools descriptive statistics were computed Advancing learning, certifying your future

RESULTS Teacher Coursework score and Moderator Coursework score Means. Level JC Subjects Subject Score type Mean SD 1 Teacher Mark Sample Moderation Mark Teacher Mark Sample Moderation Mark 40. 17 40. 22 51. 88 53. 59 42. 68 36. 52 45. 74 12. 91 59. 01 47. 55 64. 22 61. 57 6. 40 6. 53 19. 49 18. 23 13. 89 14. 34 17. 26 6. 37 17. 22 14. 52 14. 06 15. 17 2 3 4 BGCSE 5 6 Advancing learning, certifying your future

The relationship between Teacher Coursework score and Moderator Coursework score. Level JC BGCSE Subject r r 2 ICC 1 2 . 93. 98 86% 96% . 96. 98 3 4 . 97. 65 94% 42% . 95. 20 5 . 85 72% . 80 6 . 91 83% . 93 Advancing learning, certifying your future

Correlation coefficients between Teacher Coursework score and other components. Component Paper 1 Paper 2 Paper 3 Scaled score JCE 1. 25. 26. 27 2. 27. 43. 45 3. 54. 53. 66 BGCSE 4. 44. 51. 53 6. 50. 85. 50 5. 41. 49 JC Subject 2, JC Subject 3, and BGCSE Subject 5 Teacher’s Coursework scores didn’t have the highest correlation with Practical Exam! Advancing learning, certifying your future

Shared Variance between SBA and External Score with highest r Coefficien t Subject 1 JC BGCSE 2 3 4 5 r . 27 . 45 . 66 . 52 . 41 r 2 7% 20% 44% 27% 17% 24% 72% Paper with highest r Scaled Score Scaled External Score Practical Exam 6 . 49 . 85 Indiv. study NB: JC Subjects 2, 3, and BGCSE Subjects 5, and 6 all have a Practical Exam Component. BGCSE Subject 5 has 3 papers to choose 1 from. Need to investigate such low r 2 for All subjects except BGCSE Subject 6!! Advancing learning, certifying your future

The Statistical Moderation Process: School X JSS – Subject 1 Descriptive Stats. Component N/n Range Min Max Mean S D Sample Moderated Mark 46 31 18 49 36. 89 8. 93 Teacher Cwork mark 154 37 12 49 36. 07 7. 84 Scaled Score 154 49 6 55 30. 69 11. 79 Paper 1 154 25 5 31 18. 21 5. 41 Final Moderated Mark 154 19 6 25 18. 04 3. 92 Paper 2 154 28 0 28 12. 50 6. 88 Advancing learning, certifying your future

Standardise the coursework scores using the Scaled Score – JSS School X Scaled Score’s Z- Teacher Coursework Teacher Course. Work. 27 16 13 52 55 32 12 20 28 49 51 28 53 40 39 35 6 32 Scores -0. 31 -1. 25 -1. 50 1. 81 2. 06 0. 11 -1. 58 -0. 91 -0. 23 1. 55 1. 72 -0. 23 1. 89 0. 70 0. 37 -2. 09 0. 11 Score 38 13 42 43 34 33 40 12 41 34 47 45 49 42 47 35 30 40 Score Z-Score 0. 25 -2. 94 0. 76 0. 88 -0. 26 -0. 39 0. 50 -3. 07 0. 63 -0. 26 1. 40 1. 14 1. 65 0. 76 1. 40 -0. 14 -0. 77 0. 50 Advancing learning, certifying your future

Standardizing the Teacher Coursework score Scaled Score Z- Teacher C/W Mark Z- Stsd Teacher C/W. 27 16 13 52 55 32 12 20 28 49 51 28 53 40 39 35 6 32 30 Scores -0. 31 -1. 25 -1. 50 1. 81 2. 06 0. 11 -1. 58 -0. 91 -0. 23 1. 55 1. 72 -0. 23 1. 89 0. 70 0. 37 -2. 09 0. 11 -0. 06 C/W Mark 38 13 42 43 34 33 40 12 41 34 47 45 49 42 47 35 30 40 45 Score 0. 25 -2. 94 0. 76 0. 88 -0. 26 -0. 39 0. 50 -3. 07 0. 63 -0. 26 1. 40 1. 14 1. 65 0. 76 1. 40 -0. 14 -0. 77 0. 50 1. 14 Advancing learning, certifying your future Score 33. 49793746 -4. 81731461 39. 62837779 41. 16098787 27. 36749713 25. 83488705 36. 56315763 -6. 3499247 38. 09576771 27. 36749713 47. 29142821 44. 22620804 50. 35664837 39. 62837779 47. 29142821 28. 90010721 21. 2370568 36. 56315763 44. 22620804

Applying the VCE correction factor • Then Subjecting the Teacher coursework score to the model: Advancing learning, certifying your future

Final Moderated Ext. Score 27 16 55 32 12 20 28 28 53 40 35 6 32 30 Ext. Score Z- Teacher Scores er Mark ZMark Score -0. 31 38 0. 25 -1. 25 13 -2. 94 2. 06 34 -0. 26 0. 11 33 -0. 39 -1. 58 40 0. 50 -0. 91 12 -3. 07 -0. 23 41 0. 63 -0. 23 45 1. 14 1. 89 49 1. 65 0. 79 42 0. 76 0. 37 35 -0. 14 -2. 09 30 -0. 77 0. 11 40 0. 50 -0. 06 45 1. 14 . Final Moderated Mark (orig) 19 6. 5 17 16. 5 20 6 20. 5 22. 5 24. 5 21 17. 5 15 20 22. 5 Stsd Teacher C/Work Score y t_moder ated 33. 49793746 -4. 81731461 27. 36749713 25. 83488705 36. 56315763 -6. 3499247 38. 09576771 44. 22620804 50. 35664837 39. 62837779 28. 90010721 21. 2370568 36. 56315763 44. 22620804 32. 20372 6. 288486 25. 64027 24. 14328 35. 83073 6 37. 73056 45. 90535 55. 00091 39. 68794 27. 19481 19. 99759 35. 83073 45. 90535 16. 10186 3. 144243 12. 82014 12. 07164 17. 91537 3 18. 86528 22. 95268 27. 50045 19. 84397 13. 59741 9. 998794 17. 91537 22. 95268 Advancing learning, certifying your future

Agric stats moderation outcome. Domain Raw Teacher Coursew ork 2 3 4 Schools and Pass Rate 5 6 7 Low Low High Average High 29 0 12 36 34 32 50 22 49 50 49 43. 51 3. 81 4 53 24. 19 11. 06 -18. 00 13. 94 4. 76 6 49 25. 01 10. 23 -4. 95 36. 07 7. 84 6 55 30. 69 11. 79 -6. 35 46. 99 1. 89 20 65 42. 29 8. 96 -9. 83 43. 06 24. 19 11. 06 42. 33 25. 02 10. 23 50. 36 30. 54 12. 01 1. 63 53. 12 24. 12 11. 71 6 49. 01 25. 00 10. 55 6. 00 55. 00 30. 68 11. 60 Statistics 1 Min Mark Max Mark Mean SD External min Score max mean SD Standardi min sed max Teacher Mean Coursew SD ork Score Stats min moderate max d score Mean SD 8 9 10 Low 30 Average ge 26 36 47 48 50 48 49 44. 18 3. 32 9 55 35. 18 11. 14 1 43. 57 2. 63 10 60 34. 03 10. 27 -11. 12 42. 18 3. 652 7 59 35. 96 12. 40 -107 43. 39 4. 074 0 59 30. 20 12. 81 -24. 5 44. 81 2. 33 12 62 30. 57 10. 80 -10. 20 41. 09 5. 142 6 54 27. 54 11. 40 6 56. 56 42. 28 8. 96 52 35. 32 11. 19 47. 41 34. 02 10. 27 69. 83 35. 96 12. 33 50. 98 30. 20 12. 81 45. 33 30. 57 10. 80 54 27. 54 11. 40 16. 21 65. 01 42. 28 11. 08 9. 59 54. 32 34. 16 11. 45 3. 82 60. 01 34. 02 15. 03 7. 00 54. 00 35. 95 11. 93. 4. 15 59. 00 30. 19 13. 11 2. 89 62. 00 30. 50 17. 14 6 54 27. 54 11. 40 Advancing learning, certifying your future 21

Development of a Statistical Model for BEC • Employed Design-Based Research (DBR) methodology. • DBR develops product with practitioners interactively and iteractively • To ensures practicability and utility Advancing learning, certifying your future

Development of the model Intervention (R Q 2) Baseline (RQ 1) Prototype 3 Design Specifications Prototype 2 Baseline survey findings Prototyp e 1 Expert Review Define Measure Baseline survey Final Prototyp e 4 Analyse Design Pilot Try-out Develop Intervention Advancing learning, certifying your future Implement Field test

Developmental work. • System-wide Approach (pathways/OBE) • Moderation Observation and interviews • Need to understand why some subjects teacher coursework and moderator coursework are high and others’ moderate • What is it that the teachers or moderators or students or schools or the system does/doesn’t do Advancing learning, certifying your future

Conclusion • Subjects showed high correlation and agreement between Teacher Score and Moderator Score suggesting that Teachers and Moderators applied the scoring guide consistently the same. Thus the teachers scoring could be relied upon. • Correlations between the Teacher score and the External scores was low to moderate for all the subjects • Correlations between Teacher score and Practical test scores was found to be low to moderate Advancing learning, certifying your future

Conclusion • Development of the model is a system-wide endeavour hence takes long Interventionist: developing an intervention in a real world setting. • Interactive: Involves active participation of practitioners & professionals in the various stages and activities • Iterative: incoporates cycles of analysis, design & development, evaluation & revision. Advancing learning, certifying your future

THANK YOU!! Advancing learning, certifying your future

JC Subject 1 Corr Matrix Components. PP 1 PP 2 1. 82**. 94** 1. 97** 1 Teacher. 25** C/W Mark Sample. 23** Mod Mark . 26** . 27** 1 . 23** . 24** . 93** 1 Final Mod Mark . 25** . 26** . 99** . 93** PP 1 PP 2 Ext. Score . 25** Ext. Score Teacher Sample Final C/W Mark Mod. Mark Advancing learning, certifying your future 1

Corr matrix for JC Subject 2 Components Component PP 1 PP 2 Ext. Scor e PP 1 1 . 35** 1 PP 2 Ext. Score Teacher C/work Mark Sample Mod Mark Final Mod Mark . 73** Teacher C/work Mark. 27** Sample Mod Mark. 26** Final Mod Mark. 28** . 89** . 44** . 42** . 44** 1 . 45** . 44** . 45** 1 . 98** 1. 0** 1 . 99** 1 Advancing learning, certifying your future

Corr matrix for JC Subject 3 Components Component PP 1 PP 2 PP 3 External Teacher Sample Score Mark Mod Mark PP 1 -Listening PP 2 (SASI) PP 3 (PRACT Exam) External Score Teacher Mark (SBA) Sample Mod Mark 1. 77**. 57** 1. 60** 1 . 83**. 54** . 87**. 53** . 90**. 63** 1. 66** 1 . 62** . 73** . 75** . 96** 1 Final Mod Mark . 56** . 57** . 67** . 70** . 94** 1. 0** Advancing learning, certifying your future Final Mod Mark 1

Correlation matrix for JC Subject 4 Component Paper 1 Paper 2 External Teacher Sampl Final Score C/Work e Mod Mark 1 Paper 2 . 769** 1 External Score . 872** . 982** 1 Teacher C/work. 443 ** Mark Sample Mod. 447** Mark Final Mod Mark. 417** . 511** . 516** 1 . 509** . 512** . 647** 1 . 476** . 482** . 711** . 658** 1 Advancing learning, certifying your future

Corr matrix for BGCSE Subject 6 Components Component PP 2 PP 1 PP 2 PP 3 External Teacher Sample Final Score Mark Mod Mark 1. 46** 1 PP 3 . 48**. 39** 1 External Score . 46**. 48** 1** Teacher Mark . 50**. 34** . 85**. 50** 1 Sample Mod Mark . 46**. 35** . 98**. 46** . 85** 1 Final Mod Mark . 48**. 39** 1** . 85** . 98** PP 1 1 . 48** Advancing learning, certifying your future 1

Corr matrix for BGCSE Subject 5 Component PP 1 PP 2 PP 3 Final Scaled Mark Teacher Mark PP 1 1 . a 1. 000** PP 2 . a 1 . a PP 3 . a 1 Final Scaled Mark Teacher Mark Sampled Mod Mark Sample Mod Mark Final Mod Mark . 414** . 495** . 512** 1. 000** . 487** . 570** . 537** . 999** . 492** . 537** . 573** 1 . 482** . 553** . 559** 1 . 906** . 959** 1 . 907** Final Mod Mark 1 Advancing learning, certifying your future