Qualitative MetaAnalysis and Data Synthesis of Recent Research

Qualitative Meta-Analysis and Data Synthesis of Recent Research on Test Accommodations for Students with Disabilities: Summary and Implications John F. Olson, Ph. D. Olson Educational Measurement & Assessment Services NCSA Annual Meeting Minneapolis, MN June 2012

Overview: Study of Recent Research on Test Accommodations In 2011, conducted a “qualitative metaanalysis and data synthesis” of recent accommodations research in large-scale student assessments 1. Reviewed literature for relevant studies 2. Collected existing data on studies that examined impact of test accommodations, preferably those using a random-control groups experimental research design 3. Analyzed/summarized data and entered it into a new and expanded summary table from CTEAG database 2

Research Procedures • Searched NCEO Online Accommodations Bibliography (searchable accommodations database on NCEO website) − identified several dozen experimental or quasiexperimental studies in published reports • Reviewed literature for relevant studies, such as those noted in key reports or sources summarizing recent accommodations research, e. g. , in the Bibliography for the CTEAG Guidebook and in NCEO Technical Reports • Obtained copies of published reports 3

Studies and Data Sources • Focus was on relatively recent (since 2000) 2 x 2 randomized-control group studies that had test data (Note: they were not that easy to find) • Categorized studies into − Useable quantitative data − Partially useable quantitative and/or qualitative data − Not useable or relevant data • Found 17 empirical studies with useable quantitative data for inclusion in project − Note caveats on ability to find appropriate studies or data (costs, limited stats reported, no data for calculating effect sizes, etc. ) 4

Analysis • Obtained data from studies (Ns, Means, Std Deviations) for both groups of students – SWD and SWOD – under accommodated and non-accommodated conditions • Analyzed effect sizes from studies (if possible) − Note: Effect size calculations are effective in summarizing different studies; it provides a practical way to measure the strength of the relationship between two variables and the effectiveness of a treatment/effect. • Obtained stats from ANOVA (if available) and data on any other analyses that were done • Summarized information into an expanded accommodations database summary table 5

Information included in Expanded Summary Table on Effects of Test Accommodations • Title of study (publication), name(s) of author(s), and date • Accommodation studied • Grades and content area tested • Sample sizes for SWD and SWOD • Information on analysis of the accommodation’s impact on items and test forms, dimensionality, and test construct • F-values and p-values from ANOVA (main and interaction effects) • Effect sizes (if obtainable) • General findings from study 6

(see Summary Table – separate file) 7

General Comments on Findings • Validity studies conducted by the CTEAG project were among the most comprehensive ones that were found investigating effects of test accommodations • Looking across all studies that were summarized – in general , findings are mixed − Differential boost found in some studies but not others − Signif main effects found in 7 of 17 studies − Signif interaction effects found in 6 of 17 − Large ES found in a some studies, small to no 8

Implications for States and Others • Meta-analysis provides more evidence for states to use and reference on validity of test accommodations from a variety of sources and types of accommodations • Information from this study supplements and expands that from the CTEAG project, i. e. the project database • No one study is able to answer all the questions concerning validity of test accommodations • Researchers have a new summary table that will be easily expandable to include results from other studies of accommodations as they become available in the future 9

Questions & Comments For more information, contact: John F. Olson Educational Measurement & Assessment Services 617 -965 -1490 (home/office) 210 -859 -5593 (cell) jmclkolson@yahoo. com