Datadriven School Improvement through the VCE Data Service

Data-driven School Improvement through the VCE Data Service Presented by Philip Holmes-Smith School Research Evaluation and Measurement Services

What is the VCE Data Service? VCEDS • is a confidential online service provided by the VCAA to all VCE schools through the VASS system; • includes an archival database holding a snapshot of your school’s VCE data for each year from 1998 onwards; • allows you to input your own school data to integrate with the data held by the VCAA.

The Origins of VCEDS • VCEDS was developed in 2002 following an extensive R & D trial with 50 schools since the late 1990 s (the VCE Data Project). • VCEDS was fully implemented in January 2003, when the 2002 VCE data were loaded into the system. • Subsequent improvements have been made to the data Service in successive years

What’s available? You can • analyse graded assessment results, GAT scores, and ENTER Subject scores as well as VCAA Study Scores; • generate reports for your “Assessing School” students, as well as your “Home School” students; • look at patterns of results for individual students; • export data for further analyses.

Who has access to VCEDS? • VCEDS was designed to be used by the senior leadership team in each school. • Each VASS coordinator has access to VCEDS through the opening menu of the VASS system (look under “School Administration”). • The VASS coordinator can grant access to other users with the approval of the Principal. • These users have access to VCEDS only; not the rest of the VASS system. • Each user has his/her own security device, which cannot be shared.

Comparisons you can make External to your school: Your school results can be compared with results for • All schools • Like schools. • Schools in your sector. • Like schools in your sector.

Like School Groups (for Government Schools only)

Comparisons you can make Internal to your school: Your school results can be compared • • • by gender, by class grouping, by year level by year (1998 - present), by any subgroups in the school for which you can provide data (e. g. campus, language background, year level).

VCE Data Service: Menu Structure

VCE Data Service: Menu Structure

VCE Data Service: Menu Structure

VCE Data Service: Menu Structure

VCE Data Service: Menu Structure

VCE Data Service: Menu Structure

Report 1

There are 30 students in this Year Level. Therefore: – 50% (or 15 students) are above the median. – 50% (or 15 students are below the median. – 10% (or 3 students) are at or below the 10 th percentile “whisker”. – 10% (or 3 students are at or above the 90 th percentile “whisker”. – 15% (or 4 -5 students) are spread between the 25 th down to the 10 th percentile. – 15% (or 4 -5 students) are spread between the 75 th up to the 90 th percentile. 3 students on or above the 90 th percentile. (Note, all we know is at least one student scored on the 90 th percentile. ) 4 -5 students between the 75 th up to the 90 th percentile. 7 -8 spread above the median. – 50% (or 15 students) are inside the “box”. Half of these (7 -8 students) are above the median and half (7 -8 students) are below the median. 15 students inside the box. 7 -8 bunched below the median. 15 students below median • 15 students above median Interpreting “box and whisker” graphs 4 -5 students between the 25 th down to the 10 th percentile. 3 students on or below the 10 th percentile. (Note, all we know is at least one student scored on the 10 th percentile. )

Report 5

Report 7

Report 8: Example 1

Report 8: Example 2

Report 8: Example 3

Report 8: Example 3 (cont. )

Report 8: Example 3 (cont. )

Report 8: Example 4

Report 9 Mathematical Methods : (VCAA Study Score x Time) z. Test College Home School Data

Report 15: Example 1

Report 15: Example 2

Report 16 Biology: GAT and Study Score Distributions x Time z. Test College Home School Data

Relationship between GAT scores and VCE Study Scores • The last slide shows that for Biology, the higher the ability of the group (as measured by the GAT) the higher the VCE study score • The VCE Data Project established that overall, the GAT is a very good predictor of VCE study scores

Example: Relationship between GAT (Comm) scores and English Study Scores

Predicting VCE Study Scores The VCE Data Project established clearly that, other things being equal, you can expect higher levels of performance from a student who • has greater ability; • is in a class with more able students; • is female.

Adjusted Estimates VCEDS calculates what each of your students could have been expected to get on the basis of • their ability, as indicated by their three GAT component scores; • the average GAT performance of the subject group in their school; and • their gender. It then compares your students’ results to those that would have been expected. The difference is called an adjusted score Aggregated across students, it is called an adjusted estimate for the group.

Example using just the GAT (Comm) Score to predict English study scores Students who scored 25 on the GAT would be predicted to score 26 for their VCE English study score

Example (cont) However, this student actually scored 35 for English which is 9 study score points higher than we would have predicted On the other hand, this student actually scored 18 for English which is 8 study score points lower than we would have predicted

How does VCEDS report Adjusted Estimates? • If students performed precisely as predicted in a given study taking into account their abilities and their gender composition, then the group will have an adjusted estimate of zero. • If the students perform less well than predicted, the group will have a negative adjusted estimate. • If students perform better than predicted, the group will have a positive adjusted estimate.

What sort on numbers do the Adjusted Estimates produce? • The scale on which adjusted estimates are reported is the same as that for Study Scores. • An adjusted estimate of +2. 5 would mean that the student group has achieved, on average, 2. 5 Study Score points higher than expected. • An adjusted estimate of -1. 7 would mean that the student group has achieved, on average, 1. 7 Study Score points lower than expected.

Why the GAT? • The GAT measures the general knowledge and skills that students have built up during their school years. • It provides an indication of how wellequipped students are to undertake VCE studies. • Data show it to be a good predictor of VCE performance and provides a good indicator of where students would be in their VCE studies, “other things being equal”.

Reporting Constraints VCEDS does not report: • Study score statistics and assessment grades where studies have less than 10 students • Ability and Gender Adjusted estimates where there is: – less than 400 total enrolments with complete data – less than 20 schools with enrolments – less than 5 students in the school – insufficient correlation between study and GAT scores

Reporting adjusted estimates All adjusted estimates are associated with a degree of uncertainty. This uncertainty can be quantified and a confidence interval can be constructed around each estimate.

Confidence Intervals Confidence intervals are displayed by means of ‘error bars’ around the estimates. You can be confident that two adjusted estimates differ only if their error bars do not overlap.

Confidence Intervals- Example Estimates Error Bars

Report 10

Report 11

Report 12 Mathematical Methods (Adjusted) x Time z. Test College

VCE study scores for the five Math Method classes

Report 13

GAT (MST) scores vs. study scores for the five Math Method classes

Report 17

Report 18 Student Results across Studies z. Student 14, z. Test Confidential Report 2005 z. Test College Home School Data

A strategy for approaching VCEDS 1. 2. 3. 4. 5. 6. 7. Look at the overall performance for the year. Look at the results, study by study. Drill down within studies, comparing performance of subgroups within your school and trends over time. Look at trends in GAT results within studies and over time. Look at adjusted estimates, study by study. Drill down within studies, comparing adjusted scores of subgroups within your school. Seek a deeper understanding by examining patterns of individual performance.

Exporting data

Exporting data (Cont)

Exporting data (Cont)

Defining school-level variables