Oregons Online Assessment Tony Alpert Oregon Department of

Oregon’s Online Assessment Tony Alpert Oregon Department of Education CCSSO- NCSA June 21, 2010

Increase in Oregon Online Testing ** *Math general education tests used as an example **461 paper math tests administered in 2008 -09

Oregon’s Contract – Oregon Department of Education • Contracts with Oregon teachers to write all of its test items • Conducts the psychometric work • Develops and documents the system • supplies Tier I help desk services • Maintains a database with “official scores” • Oregon supplies the official student, teacher, and school level reports online

Oregon’s Contract (cont) – Oregon’s Vendor (AIR) • Provides an item bank application that holds Oregon’s items • Supplies the test delivery application including the adaptive algorithm • Supplies an online reporting system that incorporates tiered security • Delivers data to Oregon for Oregon to incorporate into it’s database

Who’s doing what? – Project Management for Vendor and State. If the state doesn’t meet its deadlines, vendor may not meet theirs – Quality Control • Item quality control including item rendering consistency with past format • Unit testing, system testing – Help Desk and Training for State, District and School Personnel

Trust but verify – Application Specifications including use cases and mock-ups – Must include minimum system requirements – Test plans that encompass each element of functionality and on each “flavor” of browser/operating system combination – Test plan reports that describe whether the tests were successful – Plain language is critical – User acceptance testing in school environments is powerful

Browse with Security – Secure browser development should be completed by July of each year to allow districts to install during the summer. Push back development timelines accordingly – Lock down supported browsers and platform versions earlier rather than later – Decide whether to allow auto-update or not

Preventing Problems – Build a system that is designed to work within the technology available to your schools – Limit the use of propriety 3 rd party software – Limit bandwidth use by reducing item size – Include detailed Service Level Agreements in the RFP and resulting contract • • • Include System Latency System down time Item presentation (Correct Item at the Correct Time) Help Desk Response Time/Resolution Data delivery (quality, timeliness) Quality of deliverables

Preventing Problems (cont) – Ensure vendor has a comprehensive approach to change management including • Test, Development and Production environments • Whom has access to change the production environment (Change Review Board) • Version control and documentation – Ensure vendor’s system is reliable • • Fail over requirements Backup requirements Business continuity Security – Limit the volume of work sub-contracted

Fixing Problems – Push development timelines as early as possible – System to track known issues during the development period and during operational testing – Communicate clearly (and early) as to when support for hardware, platforms and operating systems are anticipated to end – Ensure problem resolution process is clearly described – Consider methods for emergent communications

Policy Stuff • Reset vs Restart tests? • Testing in non-traditional environments – Alternative programs – Home school – Students with long term medical illnesses • Online/Virtual schools? • Resolving over multiple testing events • Duration between start and end of the test

System Enhancements – Annual lessons learned sessions – Include specific expectations for enhancements • Federal compliance reviews • Enhancements provided to other clients • Annual improvements to the interface based on school and district requests