Kentuckys Experience with ThreeDimensional Science Assessment National Conference

Kentucky’s Experience with Three-Dimensional Science Assessment National Conference on Student Assessment Philadelphia Marriott June 21, 2016 Kentucky’s Experience with Three-Dimensional Science Assessment Page

Session Presenters Stephen Pruitt—Commissioner of the Kentucky Department of Education (KDE) Karen Kidwell—Director of the Division of Program Standards (KDE) Roger Ervin—System Administrator for the Office of Assessment and Accountability (KDE) Nicolle Romero—Science Assessment Development Manager, West. Ed Joanne Jensen—Director of Assessment Client Relations, West. Ed Kentucky’s Experience with Three-Dimensional Science Assessment Page 1

2 Kentucky’s Guiding Principles }Equity }Achievement }Integrity Kentucky’s Experience with Three-Dimensional Science Assessment Page 2

3 Science Assessment: A Team Process The Commissioner } Office of Teaching and Learning } Office of Assessment and Accountability Kentucky’s Experience with Three-Dimensional Science Assessment Page 3

A SYSTEM of Science Assessments • Built on teaching and learning in classroom • Each component adds and elicits new evidence of students’ learning • Defensible evidence of student attainment of the standard, as the standard was intended, is generated Classroom Embedded Assessments Through. Course Tasks PE Statewide Summative Assessment • Supports different purposes/users/uses Kentucky’s Experience with Three-Dimensional Science Assessment Page 4

Leverage THINKING PARTNERS for Quality • Started with BOTA report to conceptualize vision of system • Supported by Commissioner to seek expert advice • Established an RFP for a THINKING PARTNER separate from/ahead of seeking a vendor • Simultaneously supported in-field teachers/administrators with deep, ongoing professional learning around the KAS for Science (3 years)—and the vision for the system – taking their feedback into account • Balanced involvement of ‘assessment’ and ‘content’ specialists in bringing the vision to reality Kentucky’s Experience with Three-Dimensional Science Assessment Page 5

Lessons Learned (so far!) • The WHOLE (of PEs) is greater than the sum of their parts • Seek clarity on the role of the PEs and all other supporting resources available for the standards • Storylines bring coherence and purpose to item clusters/tasks • 3 -Dimensionality is challenging but not impossible • Balance desire to innovate on ‘items’ with the reality/need to try things out in order to make progress • Plan for investment in professional learning to support classroom change • Must rethink the ‘metrics’ of scoring/reporting with clusters Kentucky’s Experience with Three-Dimensional Science Assessment Page 6

Role of Assessment and Accountability Science Assessments Ø Balancing Act Ø Supporting the goals of the Commissioner, Kentucky Board of Education and the Department related to science education Ø Connecting program staff and vendor services Ø Building for appropriate administration of state-required tests Ø Goals Ø How to ensure Kentucky students have a proper learning trajectory for science and become science literate? Ø How can assessment support quality science instruction and curriculum? Ø Challenges Ø What is doable within limits of technologies, vendor capacity, time and money? Ø Science Assessment Plan Kentucky’s Experience with Three-Dimensional Science Assessment Page 7

Science Assessment - Goals Ø Measure the richness of the standards (i. e. , content spectrum and three dimensions) Ø Provide for student accommodations and consistent test administration Ø Support student attainment of science standards at every grade throughout the school year (i. e. , classroom items and tasks) Ø Develop a robust and sustainable bank of items/clusters Ø Yield valid and reliable student/school performance results Ø Report detailed results at the school level Ø Develop potentially online interactive assessments (i. e. , simulations and tasks) Kentucky’s Experience with Three-Dimensional Science Assessment Page 8

Science Assessment - Challenges Ø Every Student Succeeds Act (ESSA) Ø U. S. Department of Education interpretation of the law Ø State interpretation of the law Ø Kentucky State Laws Ø All state summative assessments (i. e. reading/ELA, on-demand writing, mathematics, science and social studies) administered in a five-day window in the last 14 days of the school Instructional calendar Ø Limitations on testing time (2 – 2½ hours) Ø Timelines for reporting student and school performance results Ø Limited money $$$ Kentucky’s Experience with Three-Dimensional Science Assessment Page 9

Science Assessment – Challenges Ø Paper/pencil assessments Ø Does not easily allow support for technology-based accommodation solutions Ø Costs for printing and shipping are increasing; processing can be slowed and test results delayed Ø Technology-based assessments Ø Technology is constantly changing Ø Considerations need to be given to schools with technology limitations versus schools with one-to-one technology for students Ø Vendor concerns include: reliable delivery of an online assessment, limitations of processing platforms and limited experience with effectively supporting simulations and tasks Kentucky’s Experience with Three-Dimensional Science Assessment Page 10

Science Assessment Plan Ø For the first time, science test development is grounded in science phenomenon Ø On-going approach to item and item cluster development to allow for continuous improvement of the test and development of teacher and administrator science literacy Ø Test administration – measure richness of standards across years Ø Field test new items/item clusters each year to refresh a robust and sustainable bank Ø Paper and pencil assessments initially; however, building items with an eye toward future technology and use of online interactive platforms that may include simulations and tasks Kentucky’s Experience with Three-Dimensional Science Assessment Page 11

Summative Assessment Plan Elementary and middle school Ø Alternate assessment (grades 4 and 7) Ø Full operational test in 2016 -17 Ø Standard setting in summer 2017 Ø Report student performance levels to students, parents and schools/districts in fall 2017 Ø Not included in accountability until new system in 2017 -18 Ø Regular assessment (grades 4 and 7) Ø Field test administered in 2016 -17 Ø Goal to build item bank Ø No standard setting or reporting of student performance levels in 2016 -17 Ø Not included in accountability until new system in 2017 -18 Kentucky’s Experience with Three-Dimensional Science Assessment Page 12

Science Assessment Plan Ø High school assessment Ø TBD but not end-of-course (EOC) (Biology EOC measures only part of science standards) Ø Option 1 – Summative assessment Ø Option 2 – Assessment modules that cover student’s high school experience Ø Option 3 – Hybrid of Options 1 and 2 Kentucky’s Experience with Three-Dimensional Science Assessment Page 13

Accountability Ø Historically, Kentucky has included science performance in state accountability Ø New accountability system under development Kentucky’s Experience with Three-Dimensional Science Assessment Page 14

State Summative Assessment (SSA) Prototype • Demonstrates a starting point for the vision and expectations for the summative assessment of the Kentucky Science Assessment System • A collaborative effort with KDE staff, science content experts, and assessment designers and developers • Intended uses: 1. To serve as an initial model for measuring the three-dimensional science learning as called for with the NGSS (NGSS Lead States, 2013) and in A Framework for K– 12 Science Education (hereafter “the K– 12 Framework”) (NRC, 2012). 2. To support Kentucky in guiding its test vendors with the design and development of the three dimensional-aligned state summative assessment portion of the Kentucky Science Assessment System. Kentucky’s Experience with Three-Dimensional Science Assessment Page 15

State Summative Assessment (SSA) Prototype • The SSA prototype is aligned to a PE bundle and presented within the context of a phenomenon. • The purposes of PE bundles are: (1) to ensure that PEs are not assessed in isolation from one another (as recommended in the BOTA report; NRC, 2014); (2) to allow for an assessment to assess as broad a range of PEs as possible without compromising validity; and (3) to support the identification of appropriate phenomena that would provide a basis for assessing the given PEs. Kentucky’s Experience with Three-Dimensional Science Assessment Page 16

State Summative Assessment (SSA) Prototype Kentucky’s Experience with Three-Dimensional Science Assessment Page 17

State Summative Assessment (SSA) Prototype Kentucky’s Experience with Three-Dimensional Science Assessment Page 18

State Summative Assessment (SSA) Prototype Kentucky’s Experience with Three-Dimensional Science Assessment Page 19

Science Standards Alignment Considerations • The basis of a PE bundle is the individual PEs and the supporting information found within the PE foundation boxes. • Item clusters must be determined as meeting the intent of the PEs (i. e. , Kentucky Academic Standards (KAS) for Science) to which they are aligned. • The intent of the PEs is informed by any item specifications developed by KDE and by analyses based on the information specified in the PE foundation boxes and additional documentation (e. g. , K– 12 Framework, NGSS Progression Appendices). Kentucky’s Experience with Three-Dimensional Science Assessment Page 20

Science KAS Alignment Considerations Kentucky’s Experience with Three-Dimensional Science Assessment Page 21

Science KAS Alignment Considerations Context judgments • PE bundle (appropriate? yes/no judgment) • Phenomenon (appropriate? yes/no judgment) • Stimulus (appropriate? yes/no judgment) Individual item judgments • Item to SEP (0– 3 alignment judgment) • Item to DCI (0– 3 alignment judgment) • Item to CCC (0– 3 alignment judgment) • Item to PE (0– 3 alignment judgment) Overall item cluster judgments • Item cluster to PE (appropriate? yes/no) • PE bundle to item cluster (appropriate? yes/no) • Item cluster (acceptable for use? yes/no) Kentucky’s Experience with Three-Dimensional Science Assessment Page 22

Kentucky Spring 2016 Pilot • 4 ‘versions’ of a cluster based on same storyline • 13 student responses required (MC, short answer, matching, extended response) • 1300 7 th grade students; 26 teachers/classrooms • 40 individual cognitive lab interviews (10 each version) • 1 class period (from 40 – 75 minutes) • No accommodations were made for the pilot Kentucky’s Experience with Three-Dimensional Science Assessment Page 23

Kentucky Spring 2016 Pilot Findings • ~25% of students successfully completed claim, evidence, reasoning items – when presented as MC; fewer succeeded when they were written responses • Interpreting and ‘completing’ a model was very challenging; ~5% of students were successful with all elements across all versions (Though in cog labs, several students could explain the model correctly while still not completing it correctly) • “Mark all that apply” presented difficulty; most students got at least ONE of the correct choices, yet few marked both correctly Kentucky’s Experience with Three-Dimensional Science Assessment Page 24

Kentucky Spring 2016 Pilot Findings, cont. • Use of technology to support reading, additional graphics, enhance the storyline through video was not generally viewed as an enhancement to students • The storyline did engage students – helped them see the point of the questions • The reading load of the storyline was challenging (time, length, complexity) • Teachers viewed the structure and ‘content’ of the cluster as significantly more challenging than other science assessments Kentucky’s Experience with Three-Dimensional Science Assessment Page 25

Contact Information Stephen L. Pruitt Karen Kidwell Commissioner of Education Director Division of Program Standards Kentucky Department of Education Stephen. Pruitt@education. ky. gov Karen. Kidwell@education. ky. gov Roger W. Ervin Nicolle Romero Systems Administrator Science Assessment Development Manager Kentucky Department of Education and Technology Specialist, West. Ed Roger. Ervin@education. ky. gov nromero@wested. org Joanne Jensen Director of Assessment Client Relations West. Ed jjensen@wested. org Kentucky’s Experience with Three-Dimensional Science Assessment Page 26