Establishing the Validity of Test Accommodations for Students

Establishing the Validity of Test Accommodations for Students with Disabilities: A Collaborative of State-based Research CTEAG Project Summary of Accomplishments and Dissemination of Final Products John F. Olson, Ph. D. Principal Investigator & Project Coordinator, CTEAG Project CCSSO NCSA and Joint ASES-TILSA Meeting Detroit, MI June 23, 2010

CTEAG Project - Overview Goals • Conduct research that focuses on validity of test accommodations and interpretations of scores from accommodated state assessments • Use a common research design to conduct the studies and to analyze data for comparisons across states and accommodations • Build a shared body of research-based validity evidence on test accommodations across multiple states • Share findings from project that provide states with additional evidence (e. g. , to submit to the USED for peer reviews) in support of their decisions on the use of various accommodations 2

CT EAG Project Structure • Lead State – Connecticut • Other Participating States – Kentucky, Michigan, and Nevada • Additional State Input and Involvement – from ASES and TILSA members • Project Management Team – group of SCASS advisors, experts, and consultants that oversaw and managed all validity studies and reporting activities • CCSSO -- provided logistical support, staffing, and consultants to do tasks for the project

CTEAG Research Questions 1. Does the accommodation affect item and test form statistics? 2. Does the accommodation change the content structure of the test, i. e. , is there an impact on the test structure under accommodated and standard administrations? 3. Does the accommodation increase performance of students? Does it increase performance more for SWD than it does for SWOD?

CT EAG – Validity of Accommodations Project State Participation – Details of Studies Topics/Issues CT #1 CT #2 Accommodation Computerized text reader (computer voice) Grade 7 Computerized Read aloud text reader with human voice and other mods made since CT#1 5 4 Content Area Reading Math comprehension May 2008. Computerbased test forms with 20 MC items per form. Administration: Test Date Study Design Items/Forms development Spring 2009. Computer-based test forms. Used old secure math test with 26 MC items per parallel form. KY MI NV Enhanced directions and fewer response options Read aloud by a trained individual (e. g teacher or aide) 6 7 Reading ELA Math Mar 2009. Developed forms from a pool of nonstate items that were similar in content to KY forms with 22 MC items per form. Oct 2008 and Nov-Dec 2008. Jan. 2009. Used 20 MC items Used released per form. items – 24 MC and 1 CR per form. 5

Summary of Results • In most cases, accommodations did not alter item stats (difficulty, discrimination) • Accommodations also did not change test form stats/characteristics (mean item difficulty, reliability) • Accommodations did not have an effect on content structure of test forms • Accommodations did not change dimensionality of tests John Olson Establishing the Validity of Test Accommodations June 2009 6

Summary of Results (continued) • Overall, results from ANOVA were mixed • Accommodations increased performance of students in KY study; enhanced directions increased performance in MI • Differential boost effect found in some cases, i. e. , SWD scores increased more than SWOD in CT#2, KY, and MI studies • Key areas of focus – • Differential boost effects • Increasing the accessibility of the assessment John Olson Establishing the Validity of Test Accommodations June 2009 7

CTEAG Project Accomplishments • Assisted and guided states in conducting 5 different validity studies on test accommodations • Analyzed all data from the studies and reviewed results with the PMT and states (and sometimes with states’ vendors) • Summarized results in a series of detailed state reports • Shared updates and information on progress of project at 8 ASES and TILSA meetings from 2007 to 2010 • Completed and disseminated 3 reports and a Project Database • Presented information on the project at 4 national conferences and meetings (2007 -2010) • Conducted 3 evaluations of the work, surveying states, 8

CTEAG Final Reports/Products • Guidebook for Test Accommodations Validity Studies --provides overall summary of results/findings and guidance to states on validity studies in a nontechnical manner [report was distributed to ASES and TILSA in Feb. 2010] • CTEAG Technical Report – provides technical information on the project and details of each state’s study, e. g. , designs, methods, analyses, statistics, results, etc. 9

CTEAG Final Reports/Products • Lessons Learned from State Validity Studies – provides information on the background and context in which the validity studies were conducted, issues that states may need to consider, and lessons learned by the four participating states • CTEAG Project Database -- provides additional info from the five studies, such as, samples, variables, data files, and statistical data for use in further research 10

Dissemination Activities • Presentation Session at 2010 NCSA on June 22 − State participants present results and perspectives from conducting validity studies − Representative from USED discusses importance of findings from federal perspective • Dissemination Workshop at Joint ASESTILSA meeting on June 23 − Share all remaining products and summary of accomplishments − State participants discuss lessons learned and how data and findings can be used to benefit states − Expert advisors and USED representative 11

Final Dissemination Activities • Distribute the reports/products widely to states, the USED, and others • Place PDFs of all final products on websites (CSDE , CCSSO/EAG) • Issue formal announcement (press release? ) and Chiefline newsletter article • Coordinate with project partners (NCEO and NASDSE) to share information with others • Submit final report on project to USED 12

Questions & Comments For more information, contact: John F. Olson President/Founder Olson Educational Measurement & Assessment Services 617 -965 -1490 (home/office) 210 -859 -5593 (cell) jmclkolson@yahoo. com