TAO Open Source Solution For Successful Assessment Programs

  • Slides: 59
Download presentation
TAO Open Source Solution For Successful Assessment Programs

TAO Open Source Solution For Successful Assessment Programs

What Is Open Source Software? • Software where the copyright holder makes the source

What Is Open Source Software? • Software where the copyright holder makes the source code available to anyone to modify and re-distribute it for any purpose • Some most successful products that you may have heard of…

Security of OSS • Does the Do. D use OSS for security functions? YES.

Security of OSS • Does the Do. D use OSS for security functions? YES. • Doesn't hiding source code automatically make software more secure? NO. • Since open source code is publicly released, anyone can review it, including for the possibility of malicious code. Source: http: //dodcio. defense. gov/Open. Source. Software. FAQ. aspx#OSS_and_Security. 2 FSoftware_Assurance. 2 FSys tem_Assurance. 2 FSupply_Chain_Risk_Management

Bottom Line • Open source software is of highest quality • Open source enables

Bottom Line • Open source software is of highest quality • Open source enables economies of scale that are not available to proprietary products • Assessment professionals benefit by being able to deliver highly innovative tests at a fraction of today’s cost

Brief History • TAO initiated by University of Luxembourg and LIST • Commercial services

Brief History • TAO initiated by University of Luxembourg and LIST • Commercial services since 2012 through spin-off company O. A. T. • Global community of contributors • IMS industry leadership 5

Fast Growing User Base 6

Fast Growing User Base 6

With TAO You Can 1. 2. 3. 4. 5. 6. Author your own content

With TAO You Can 1. 2. 3. 4. 5. 6. Author your own content Ingest 3 rd party content Deliver tests in the cloud …or from your notebook Manage students Create reports 7

Innovation At Its Core • Align your content to any standard • Linear scalability

Innovation At Its Core • Align your content to any standard • Linear scalability thanks to No. SQL delivery engine • Design immersive test experiences that are truly portable

Why Users Select TAO • It’s innovative • It’s completely open • Cuts assessment

Why Users Select TAO • It’s innovative • It’s completely open • Cuts assessment cost by 50% or more 9

Open Source K 12 Assessments A Roadmap for the Future

Open Source K 12 Assessments A Roadmap for the Future

OPEN SOURCE ASSESSMENT ROADMAP 1. 2. 3. 4. 5. 6. 7. Vision What’s been

OPEN SOURCE ASSESSMENT ROADMAP 1. 2. 3. 4. 5. 6. 7. Vision What’s been done so far Review Open Source Assessment Platform Components An overview of the current state of the Open Source Assets What is not part of these platforms? What will happen next? How do we get there from here? 12

VISION The entire world is leveraging open source software in all aspects of business

VISION The entire world is leveraging open source software in all aspects of business and technology. We believe we can core components for assessment construction and delivery, and that we can leverage scoring, reporting, administrative interfaces that may be common within the K 12 industry. We believe this has the potential to fundamentally improve the economics and features of assessment. 13

TRENDS What’s driving us to believe these things? 14

TRENDS What’s driving us to believe these things? 14

TRENDS - STANDARDS IMS – QTI and LTI QTI started slow, but it is

TRENDS - STANDARDS IMS – QTI and LTI QTI started slow, but it is now ubiquitous. LTI has been adopted by Over 130 edtech vendors and platforms! 15

TRENDS – A UBIQUITOUS INTERNET • $5. 5 Billion in federal and private sector

TRENDS – A UBIQUITOUS INTERNET • $5. 5 Billion in federal and private sector investment • Goal is to have 99% of students on broadband in the classroom by 2018 16

TRENDS - REUSE After all how many rostering, reporting, content management and delivery systems

TRENDS - REUSE After all how many rostering, reporting, content management and delivery systems do we need to support large-scale assessments? There are two models of reuse – license a proprietary platform and leverage an open source platform. We are realizing that there are better models of reuse than proprietary software licensing. And that these models have been validated by large scale assessment implementations in the U. S. and internationally. 17

Open Source K 12 Assessments

Open Source K 12 Assessments

SUMMATIVE ASSESSMENT - NCSC CONSORTIUM • 2 pilots in 2014; Operational test in 2015

SUMMATIVE ASSESSMENT - NCSC CONSORTIUM • 2 pilots in 2014; Operational test in 2015 • Participating States: AR, AZ, CT, DC, ID, IN, ME, MT, NM, RI, SC, SD, CNMI, GUAM, VI • States included in Pilots, but not Operational test: CA, MD, FL, MN, ND, PA, TN, VA • 1, 342 unique items • 64 test forms • 2, 619, 290 responses • 74, 040 assigned tests 19

SUMMATIVE ASSESSMENT - NCSC CONSORTIUM What did we learn? • Completed or cancelled tests:

SUMMATIVE ASSESSMENT - NCSC CONSORTIUM What did we learn? • Completed or cancelled tests: 96. 1% • 2, 172 Measured Progress help desk calls • About 10% resulted in a data change or a possible software defect (43) • It was possible to switch testing vendors between pilot and operational testing with minimal disruption • Areas identified for improvement in the platform include: • Audit history, backend queries for CSR, System Reporting, Data changes for CSR 20

NON-SUMMATIVE ASSESSMENT – PARCC CONSORTIUM • Emphasis on diagnostic and lower-stakes assessment delivery •

NON-SUMMATIVE ASSESSMENT – PARCC CONSORTIUM • Emphasis on diagnostic and lower-stakes assessment delivery • Extended with adaptive item selection; integrating Pearson item selection algorithm • Integrating items developed by other vendors • New item types developed against PARCC performance objectives • New test driver designed for disconnected delivery • Student tools, accessibility • PCI Architecture • Still in development – launch in 2015 -2016 school year 21

NON-SUMMATIVE ASSESSMENT – PARCC CONSORTIUM What are we learning? • Systems integration and content

NON-SUMMATIVE ASSESSMENT – PARCC CONSORTIUM What are we learning? • Systems integration and content sharing are challenging to program productivity • SSO is expensive to get right and standardize on across platforms • Different test programs have large differences in platform requirements and impact on user experience • • Interim reports Dynamic test assignment and the ability to personalize tests and align to specific common core standards is essential for non-summative assessments • The ability assess across grade levels is also important 22

K 12 Open Source Assets

K 12 Open Source Assets

US K 12 OPEN SOURCE ASSETS 24

US K 12 OPEN SOURCE ASSETS 24

US K 12 NCSC OPEN SOURCE ASSETS 25

US K 12 NCSC OPEN SOURCE ASSETS 25

ELEMENTS THAT ARE NOT AVAILABLE • Detailed classroom and school reporting: This is being

ELEMENTS THAT ARE NOT AVAILABLE • Detailed classroom and school reporting: This is being developed currently under a separate support contract with another testing vendor; • Human scoring: Allowing teachers or scoring teams to log in and provide grading against rubrics as they review student responses is still in development; • Out-of-the-box Adaptive Delivery: Staged-adaptive section selection was deferred from the year one operational test. R&D was completed and shown that it could be implemented with small extensions based upon features available in QTI. 26

US K 12 PARCC OPEN SOURCE ASSETS 27

US K 12 PARCC OPEN SOURCE ASSETS 27

US K 12 OPEN SOURCE ASSETS 28

US K 12 OPEN SOURCE ASSETS 28

ELEMENTS THAT STILL REQUIRE DEVELOPMENT • • • Detailed classroom and school reporting: This

ELEMENTS THAT STILL REQUIRE DEVELOPMENT • • • Detailed classroom and school reporting: This is being developed currently under a separate RFP with PARCC by Amplify. It may be available to the community in 2017. Human scoring: Allowing teachers or scoring teams to log in and provide grading with rubrics as they review student responses is still in development Out-of-the-box Adaptive Delivery: Adaptive item selection still requires integration with a vendor’s library Smarter Balanced Item Ingestion and adaptive algorithm integration Next Generation Science Item types and tools High Stakes security features and disconnected delivery testing 29

An Open Source Ecosystem for K-12 Assessments

An Open Source Ecosystem for K-12 Assessments

OPEN SOURCE ECOSYSTEM Users – States, districts, publishers Partner Network Holding Company Technology Partner

OPEN SOURCE ECOSYSTEM Users – States, districts, publishers Partner Network Holding Company Technology Partner Free License Community Services Components Customizations

How We Get There: Roadmap

How We Get There: Roadmap

US K-12 ROADMAP – WHAT’S NEXT There a variety of ways that we are

US K-12 ROADMAP – WHAT’S NEXT There a variety of ways that we are going to extend this platform in the future: 1. Seek grant funding to continue development; 2. Implement the platform and extend it with assessment revenues for consortia, states, and districts; 3. Breakthrough and O. A. T. will make investments to continue to lower the cost of deployments and extend the feature set; 4. Partner with O. A. T. to build out features that are critical for the K-12 market; 5. Partner with test publishers edtech developers, and state IT organizations who desire expertise on implementing, integrating, and extending the platform. 33

SPECIFICALLY… 1. Continue to deploy PARCC’s year one nonsummative product; 2. Help all NCSC

SPECIFICALLY… 1. Continue to deploy PARCC’s year one nonsummative product; 2. Help all NCSC member states and/or their test vendors deploy the NCSC platform for their 2015 -2016 alternate assessment; 3. Seek grant funding to build out some missing features and complete the work required to demonstrate the idea a single, unified assessment platform; 4. Demonstrate the platform’s ability to deliver Smarter. Balanced Assessments; 5. Continue to work in and out of the K-12 market to complete missing assessment features and continue the evolution of the TAO platform; 6. Develop certification, training, and services to allow states and districts to utilize the platform and broaden the resource pool who can deploy, integrate and extend it. 34

WHAT WE’RE REALLY DOING We want to fully realize the investment from the RTTA

WHAT WE’RE REALLY DOING We want to fully realize the investment from the RTTA to create an enduring assessment platform ecosystem that the U. S K-12 Assessment community can deploy today, extend tomorrow, and continue to utilize for decades to come. This is bigger than one person, one company, one state, or one consortium’s vision. 35

Thank you! Doug Wilson (847) 864 -0033 x 201 doug. wilson@breaktech. com

Thank you! Doug Wilson (847) 864 -0033 x 201 doug. wilson@breaktech. com

National Center and State Collaborative Open Source Assessment Solution CCSSO National Conference on Student

National Center and State Collaborative Open Source Assessment Solution CCSSO National Conference on Student Assessment June 2015 Chris Domaleski Center for Assessment

Overview • NCSC is a consortium of 24 states along with University and non-profit

Overview • NCSC is a consortium of 24 states along with University and non-profit expert partners, working together over four years to develop coordinated curriculum, instruction, and assessment resources for students with the most significant cognitive disabilities. • In spring 2015, 14 states participated in successful operational testing, which involved over 70, 000 tests administered to over 35, 000 students. 38

NCSC Assessment System • The NCSC Assessment System is a technology platform that facilitates

NCSC Assessment System • The NCSC Assessment System is a technology platform that facilitates a range of functions to include – Assessment creation, management, and delivery – Evidence collection and scoring – Reporting – Training resources – Data management • NCSC partners contributing to the development of the solution include: CTB Mc. Graw Hill, Measured Progress, Open Assessment Technologies, and Breakthough Technologies

NCSC Assessment System 40

NCSC Assessment System 40

Operational Open Source Solution • What does it mean that the NCSC System is

Operational Open Source Solution • What does it mean that the NCSC System is Open Source? – Virtually all of the system is built with software in which the source code is made available and licensed such that the rights to study, change, and distribute the software freely available to anyone. – NCSC will publicly release the code this summer • What benefits does this provide? – Cost Reduction: No licensing fees – Transparency: The software code is completely available for inspection and scrutiny – Flexibility: Because the code is freely available, it can be easily adapted, revised, or expanded as needs warrant. – Collaboration and Support: Everyone benefits from the cumulative and collective improvements efforts.

System Enhancements • The enhancements to the system developed during the project include –

System Enhancements • The enhancements to the system developed during the project include – Creation of a system to manage users and organizations and track progress – Assessment features for user accessibility – Evidence upload functionality to capture and upload student work – Presentation of content on printable PDFs for paper-based administration; – Improvement and expansion of delivery and navigation tools 42

Post Implementation ‘Reflections’ • The system worked without disruption or malfunctions • Feedback on

Post Implementation ‘Reflections’ • The system worked without disruption or malfunctions • Feedback on enhanced assessment features and user interface improvements was positive • ‘Administration’ functions continue to be the most prevalent areas requiring user support • Looking ahead, NCSC seeks to identify efforts/ resources to support the open source community 43

Additional Resources • www. ncscpartners. org • Technology architecture and specifications • Presentations, papers,

Additional Resources • www. ncscpartners. org • Technology architecture and specifications • Presentations, papers, handouts and more for various audiences. Chris Domaleski cdomaleski@nciea. org

The Progress and Promise of Open Source Technology to Support State Assessment Systems June

The Progress and Promise of Open Source Technology to Support State Assessment Systems June 23, 2015

PARCC • The Partnership for Assessment of Readiness for College and Careers (PARCC) is

PARCC • The Partnership for Assessment of Readiness for College and Careers (PARCC) is a consortium of states working together to deliver a set of assessments that measure whether students are on track to be successful in college and their careers. • Summative Administration (2014 -15) – – – Over 15 million tests completed online (PBA and EOY combined) Several days with over 1 million students testing Our busiest day: 1. 4 MILLION students; 204 K students in an hour 100% uptime States reported higher-than-expected online testing rates, some with more than 90% of students test online 46

PARCC • Two pronged assessment approach – Summative assessment – Partnership with Pearson o

PARCC • Two pronged assessment approach – Summative assessment – Partnership with Pearson o o Pearson Access Next – System management Pearson Test. Nav (v 8. 2 first Cloud based solution) – Test administration – Non-summative assessments, instructional tools & professional development – develop PARCC Platform (Open Source) o o o Assessment Delivery System (ADS) Partnership Resource Center (PRC) Data Management & Reporting System (DMR) – Production release – Fall 2015 47

PARCC Platform Solution Diagram Pearson Summative Assessment Platform Individual Summative Test Results Online Reporting

PARCC Platform Solution Diagram Pearson Summative Assessment Platform Individual Summative Test Results Online Reporting States/Districts/Schools End-User Validation Breakthrough Non-Summative Assessment Platform (ADS) Breakthrough Partner Resource Center Platform (PRC) Individual Non-Summative Test Results (Diagnostics) Individual Non-Summative Test Results (Speaking & Listening/K-2) Diagnostics & Instructional Tools End-User Validation Data Management & Reporting (DM&R) Summative & Non-summative Reporting Individual Score Report Batch Generation > Schools 48

Open Source "Linus' Law, " named for Linus Torvalds, the creator of Linux. Given

Open Source "Linus' Law, " named for Linus Torvalds, the creator of Linux. Given enough eyeballs, all bugs are shallow. What that means is that the more people who can see and test a set of code, the more likely flaws will be caught and fixed quickly. The opposite of the security through obscurity argument (secrecy of the design or implementation to provide security) to justify the use of expensive proprietary products. 49

The Benefit of Open Source My first time. 50

The Benefit of Open Source My first time. 50

The Benefit of Open Source • Value proposition for open-source solutions – No license

The Benefit of Open Source • Value proposition for open-source solutions – No license fee o o Commercial costs of software licensing PARCC mission: Drive the costs of assessment as close to zero as possible – Customizable o o o Commercial products with a generalized feature/function set for mass audience Allows for “easier” pathway for integration and enhanced functionality Often much more standards based – “Interoperability” – Deep developer support network – Community benefits 51

The Benefit of Open Source 52

The Benefit of Open Source 52

Issues implementing Open Source • Open Source does not mean free – Development costs

Issues implementing Open Source • Open Source does not mean free – Development costs o o Platform development Content development – Technical operations costs – Operational support costs • Release strategy – Do we encourage States to adopt a common platform or leverage the released code to build out a customized solution 53

Advice • All Open Source is not equal – Make sure you are working

Advice • All Open Source is not equal – Make sure you are working with reputable resources that can support your efforts (BT & OAT) • Be flexible, your plans will change, but be diligent on scope • Collaborate… Talk to others, talk to us Jeffrey Cuff Director of Technology jcuff@parcconline. org 54

Going Open-Source: Learning from the Pioneers Michael Russell Boston College

Going Open-Source: Learning from the Pioneers Michael Russell Boston College

Key Lessons • Open Source can support: – Assessment program control over enhancements on

Key Lessons • Open Source can support: – Assessment program control over enhancements on their timelines – Use of the same platform across different operational vendors – Separation of content development from content delivery

Key Lessons • Open Source assessment solutions may require: – Integration of multiple open

Key Lessons • Open Source assessment solutions may require: – Integration of multiple open source solutions – Connections with in-state systems – Adoption/Refinement of industry-recognized content standards – Clearly articulated strategy for contributing code back to the community

Key Lessons • Open-source can work – But look for: • Mature solutions •

Key Lessons • Open-source can work – But look for: • Mature solutions • Established community • Tested solutions – And be ready to: • • • Start early Invest in customization Partner with experienced developers Pay for installation, hosting, and routine maintenance Control your delivery platform

Looking to the Future • Can TAO or other solutions be generalized to meet

Looking to the Future • Can TAO or other solutions be generalized to meet a higher percentage of state assessment program needs “out of the box”? • Can TAO or other solutions be used to support multiple assessment programs? • Can enhancements made by individual programs be coordinated to reduce redundant efforts and gain greater efficiency?