North Carolina Career Technical Education Assessments Using Assessment

North Carolina Career Technical Education Assessments: Using Assessment and Data for Continuous Growth and Improvement Tammy Howard, Ph. D Director, Accountability Services North Carolina Department of Public Instruction

General Overview of the System • 215 Courses at the State Level • 215+ End of Course Assessments • Ability to Deliver Assessments Online or Paper/Pencil • System to Capture and Validate Credentials and Certifications 2

Vision and Concept • System Initially Designed in the Early 1990’s • Originally Conceptualized as a Method to Reward Teachers for Continuous Improvement • Recognized Nationally as Model for Formative and Summative Assessment • Aligned to Carl D. Perkins Career and Technical Education Act Technical Attainment Performance Indicators 3

Assessment Models • Types of Test Item Banks – Field Test – Classroom (Teacher – Formative) – Secure (Accountability – State End-of-Course) • Types of End-of-Course Summative Assessments – Developed – Adopted – Industry-Based • Pre- and Post-Assessment Model • EVAAS 4

Curricular Alignment • All Courses in NCCTE are required to have: – “Blueprints” of Essential Instructional Standards Aligned to: – Instructional Content Aligned to: – Formative Assessment Aligned to: – Summative Assessment 5

Validity, Reliability, and Analysis • Validity Required for All Secure and Classroom Test Items • Reliability Study Required for Each New Assessment • Data Analysis Performed at State and District/Local Level • State-level Item Analysis • Scale Scores Determined After Reliability and Item Analysis 6

Growth Analysis • In the early 1980 s, the EVAAS approach to measuring growth was founded at the University of Tennessee Knoxville by Dr. William Sanders • Revolutionized the way educators and policymakers viewed schooling effectiveness and the ability of students to make growth • In 2006, EVAAS was implemented statewide as a school improvement resource. • In 2012, EVAAS became a formal part of North Carolina’s teacher evaluation and accountability after recommendation by West. Ed and UNC researchers. 7

How is growth measured? • Predictive-based model – Based on students’ prior testing history • all previous tests for which correlation has been established • Students must have three prior test scores in any grade/subject – What is the difference between students’ expected score and observed score? • Use all available testing history for each student to minimize impact of measurement error • Use standard errors to address uncertainty inherent in any growth model and protect against misclassification 8

Inclusion in Growth Model (EVAAS) Each CTE assessment must have sufficient: • Stretch in the scales – To measure growth of both high- & low-achieving students – No floor or ceiling effects • Reliability in the scales – Preferably have at least 40 -50 test questions per subject/grade/year • Number of districts/schools/teachers administering test – Preferably 25 at each level 9

Inclusion in Growth Model (EVAAS) Each CTE assessment must have sufficient: • Correlation with curricular objectives • Predictive relationship – Results of each CTE test must be related or correlated with results of other tests from different grades/subjects – i. e. prior math or reading test scores and the CTE test score were not strongly related for Carpentry, but they were for Biomedical Technology 10

EVAAS Reporting • Dozens of reports for use in school improvement – Reflective analytics, such as value-added and diagnostic reports for districts, teachers and schools – Proactive analytics, such as student projections – Comparison reports, such as value-added summary and scatterplots • Roster verification for the student-teacher linkages in teacher value-added reports • Help supports, such as video clips, online ticketing system, and help pages • Available through a secure web application with customized access 11

Sampling of CTE Assessments Receiving EVAAS Reporting • 50 CTE assessment met criteria for inclusion in growth model Source: http: //www. dpi. state. nc. us/docs/effectiveness-model/evaas/resources/vam-assessments. pdf 12

13

14

15

16

17

For More Information…. Contact: Trey Michael - trey. michael@dpi. nc. gov Tammy Howard - tammy. howard@dpi. nc. gov Tomberlin thomas. tomberlin@dpi. nc. gov 18