Quality Assurance in a Chaotic World Including Quality

Quality Assurance in a Chaotic World Including Quality Assurance Within Theory of Action Presented to: CCSSO 2012 National Conference on Student Assessment June 27, 2012

Quality Assurance as a Continuous Cycle • Quality Assurance - A continuous cycle that improves and refines the processes necessary to produce valid assessments to the highest standards. Ensures that excellence is inherent in all components of the process. – – Plan Do Study Act

Edcount’s relevant work: • Evaluated assessment systems in most states and the U. S. territories through the federal standards and assessment peer review process. • Developed interpretative arguments for the validity of assessments and conducted studies to test the claims and assumptions in the interpretive argument for a number of states. • Provided technical assistance for reviewing the technical quality of assessment and data quality of assessment systems.

Quality Assurance Within the Theory of Action - Claims • Students are appropriately tested for identified testing options • Test scores accurately reflect student knowledge and skills • Reports provide valid scores and are used appropriately

Students Are Appropriately Tested for Identified Testing Options – Special Studies • The Learner Characteristics Inventory (LCI) – A survey designed to collect information about the population of students who participate in alternate assessments based on alternate achievement standards (AA-AAS). – Helps states identify and evaluate “who” is participating in the alternate assessments. Provides information to help direct participation rules and guidance.

Students Are Appropriately Tested for Identified Testing Options – Special Studies • Accommodation Studies – The extent to which accommodations are provided on the state assessments. – Literature review on best practices for accommodation use. – Provides information and evidence to improve best practices, guidance, and monitoring of accommodation use.

Test Scores Accurately Reflect Student Knowledge and Skills – Special Studies • Performance Level Descriptor (PLD) studies – Examine the extent to which assessment cut scores appropriately distinguish between student performance levels, and whether the performance level descriptors adequately capture meaningful distinctions in grade-level performance. – Engage teachers in rating and feedback processes to help administrators understand how teachers use and interpret PLD’s – Evaluate the PLD’s for coherence, consistency, and accuracy.

Test Scores Accurately Reflect Student Knowledge and Skills – Special Studies • Performance Level Descriptor (PLD) studies cont. – Provides information to help states refine their standard setting processes, focus their professional development and support their interpretations on the state assessments.

Test Scores Accurately Reflect Student Knowledge and Skills – Specialized Studies • Alignment Studies – Evaluations of the alignment between academic content standards and alternate assessments. – Evaluations of the alignment between academic content standards and general assessments. – Provides information to support and improve the design and development of assessments.

Test Scores Accurately Reflect Student Knowledge and Skills – Special Studies • Cognitive Labs – Collect information about whether assessments are eliciting the intended cognitive processes. – The cognitive labs can help states understand why students may struggle with certain items or content areas on an assessment, and, as a result, provide feedback about ways to develop accessible assessments and ways to improve instruction.

Reports Provide Valid Scores and Are Used Appropriately - Special Studies • Consequential Validity Studies – Evaluate and collect evidence about the intended and unintended consequences of a states large scale assessment system. – Provides evidence to inform policies and processes and focus professional development.

Work Within Puerto Rico • Special Studies – PLD study – Accommodation Study – Consequential Validity Evaluation – Cognitive labs • Professional Development – Data Use Workshop – Curriculum Mapping Boot Camps

Work Within the Laurent Clerc Center at Gallaudet University • Special Studies – PLD studies – Cognitive Process Validity Study • Technical Assistance For: – Development of a balanced assessment system with formative and interim assessments – Development of an accountability System – Small school review – Development of a data management system

How Did the Work Inform Quality Assurance and Processes • Provided focus for professional development • Provided information to improve the design and development activities for assessments and reports • Provided context as well as empirical evidence to support policies and processes

Contact Information • Jennifer Stegman, Senior Associate • jstegman@edcount. com • • ed. Count LLC 5335 Wisconsin Avenue NW Washington DC 20015 ed. Count. com