Community Updates Gavin BusuttilReynaud Director Alpha Plus Using

Community Updates

Gavin Busuttil-Reynaud Director, Alpha. Plus

Using data for educational benefit: Adaptive assessment in the Wales national assessment system Gavin Reynaud, Alpha. Plus Consultancy Ltd. October 2018 Partnership Quality We work in partnership with our clients. This is more than a cliché for us: we care about the services we provide and the impact they have on learners. Experience has shown us that the best impact our work can have is when it is undertaken alongside our clients so we make partnership a key feature of our project approach and management method. We manage projects effectively and to the highest quality, freeing up experts to concentrate on their specialism, but ensuring that activities are manage to meet expectations. This means only making promises that we know we can keep, and remembering the promises we have made to make sure we deliver. Expertise Development Educationalists We ensure our teams consist of genuine sector experts with understanding in breadth and depth of both theory and the practical complex everyday challenges faced by education providers. We are committed to the improvement of our staff, both to promote the longterm development of our business and as an end in itself: we believe in the value of education for all. We are educationalists with a strong commitment to improving teaching, learning and assessment, based on intellectual integrity, sound evidence and innovative approaches.

Basis of the programme

Overview of national testing in Wales Existing paper tests • Introduced in 2013 • Taken by all learners Year 2 to Year 9 • Subjects are: www. alphaplus. co. uk • • • Numeracy (procedural) Numeracy (reasoning) English Reading Welsh Reading Formative purpose Mapped to Welsh National Literacy and Numeracy Framework (LNF) Sat in a two-week window: April/May Marked by teachers Results as progress scores (PS), and age standardised scores (ASS)

www. alphaplus. co. uk Welsh vs English national assessment

Drivers for change www. alphaplusconsultancy. co. uk Professor Graham Donaldson Successful Futures

2018 onwards: On-screen Adaptive Assessment www. alphaplusconsultancy. co. uk Benefits of Change • Automatic marking • Address ceiling and floor effects • Option of re-test in-year to support formative purpose • Speed of reporting • Change of learner experience

Welsh National Test Consortium • Alpha. Plus prime contractor and English reading content www. alphaplus. co. uk • BTL online assessment platform • WJEC School support, helpdesk Web. UI and Welsh Reading content • CITO Opensource adaptive algorithm • Acumina numeracy content

Progress Adaptive test trials In schools for Procedural numeracy • www. alphaplus. co. uk • • Linear Trial to IRT Calibrate items Integrate algorithm and Surpass Web. UI development Go live with Develop algorithm and Procedural Numeracy Items for reading Adaptive assessments In all Welsh schools By Spring 2018 By summer 2018 Autumn 2018 Go live with English and Welsh Reading Adaptive Assessments in all Welsh schools Sept 2019 Go live with Numerical reasoning Adaptive Assessments in all Welsh schools Sept 2020

Platform updates www. alphaplus. co. uk • • • Simplified test player Updated HTML delivery item type(s) Dual language support (within test) Updated APIs to integrate with Web. UI Interface to stateless adaptive algorithm Custom reporting output • Created Web. UI – a custom user interface

Content creation • Item authoring • Procedural numeracy items all bilingual • Created >3, 000 items (>1, 500 language pairs) www. alphaplus. co. uk • Item calibration • Items adapted for accessibility • Simplification • Alt-text • Braille diagrams

Some findings from pre-tests and trials so far

Some findings from the pre-tests and trials • Linear pre-test • Measurement properties of items and tests very good to excellent. • Internal reliability of 68 linear pre-test forms above 0. 9. • Fit to the Rasch model generally excellent. • Differential item functioning (DIF) by gender very slight, and slightly bigger by language (English vs. Welsh), but still not concerning. • Large-scale adaptive trial www. alphaplus. co. uk • Use with real learners confirmed the test specification values that had been input, and posited in simulations. • Curriculum coverage; test length; item re-use; accuracy of ability estimation • Particular issues in adaptive trials • Spread of learners’ ability: within year vs. between years • Concurrent validity: • Correlations and linear regression model of other measures of ability on estimates from adaptive tests • Check if age had an impact on item difficulty

Feedback reports

www. alphaplus. co. uk Feedback reports • In a formative assessment system, it is essential to feed back to teachers and learners to inform next steps and aid improvement. • Feedback in a computerised testing system can have advantages and challenges. Challenges Advantages • Learners receive different • Possibility of rich and questions from each other. engaging graphics and text. • Curriculum balance is on broad • Tailored content for each categories only. learner. • There may be assessment literacy • IRT model potentially issues. provides richer information on learner abilities than Example previous classical techniques. reports

Reflections

Some reflections on where we are (1) • We have entered an environment where it ought to be possible to roll out a high quality national e-assessment. www. alphaplus. co. uk • The formative purpose should work well – sample reports have been well received • National testing in Wales is not as controversial as in some other jurisdictions. • We have assembled a diverse consortium, and done a large of amount of high quality work. • Results show assessments with high validity and measurement properties. • Sophisticated, informative feedback reports have been developed. • IRT Model fit has been good – a reflection of a clear curriculum and experienced item writers

Some reflections on where we are (2) • Now we move into live implementation phase. www. alphaplus. co. uk • There remain challenging technical developments. • But probably the biggest challenge is one of change management. • Delivering an onscreen test will make demands of schools’ IT infrastructures, arrangements for invigilation, admin data, etc. • Using and understanding adaptive tests, and the reports that are generated will require teachers to update their assessment literacy. • Change is often challenging, but if change is for the better, then it is worth doing!

Thank you. Gavin. Reynaud@alphaplus. c o. uk Alpha. Plus Consultancy Ltd Unit 109 Albert Mill 50 Ellesmere Street Manchester M 1 2 HY +44 (0)161 2499 249 info@alphaplus. co. uk www. alphaplus. co. uk

Figures and tables

www. alphaplus. co. uk Learner ability estimates

Correlations between PPA and paperbased tests Year 2 Ability estimate Raw mark 0. 73 Age standardised score 0. 69 Progress score 0. 72 PPA standardised score 0. 73 0. 69 0. 72 Year 5 Raw mark Age standardised score 0. 76 0. 75 Progress score 0. 76 Ability estimate www. alphaplus. co. uk PPA standardised score Ability estimate PPA standardised score 0. 76 0. 75 0. 76 Year 9 Raw mark Age standardised score 0. 74 0. 73 Progress score 0. 74 0. 73 0. 74

R 2 values from regressing PPA and paper -based test outcome measures Regression formula = [Outcome variable] ~ ability_estimate + age_in_completed_months www. alphaplus. co. uk Outcome variable Year Group Progress score 2 3 4 5 6 7 8 9 0. 523 0. 609 0. 602 0. 574 0. 588 0. 653 0. 637 0. 550 Raw mark 0. 538 0. 627 0. 609 0. 580 0. 600 0. 622 0. 623 0. 540

www. alphaplus. co. uk Visualisation of the relationship between p. Corr, item target year and learner year

www. alphaplus. co. uk