Reducing Cognitive Load in 2 Assessments What Works

Reducing Cognitive Load in 2% Assessments: What Works (Or Doesn’t Work) for Eligible Students? Caroline E. Parker, Sue Bechard, Joanna Gorin NCSA, Detroit, Michigan June 22, 2010

Study Summary I. Using three different analysis methods (cognitive interviews, item difficulty modeling, distractor analysis), identified cognitive barriers in existing 10 th grade reading items. II. Manipulated 34 items from four released passages based on those results. III. Pilot study examined impact of manipulations. IV. Developed student profiles from cognitive interviews to understand student characteristics.

I. Identified Cognitive Barriers � Working memory capacity � Limited executive functioning (ordering, organizing) � Inability to identify important information from text � Challenging vocabulary � Inability to draw inferences � Inappropriate use of prior/outside knowledge

II. Item Manipulation Process Following an item from start to finish

Example: Original Item The author’s “difficulty” (line 1) was caused primarily by the A. B. C. D. E. long distances that had to be traveled unanticipated changes in the project refusal to question some widespread assumptions cultural limitations that hindered communication challenge of mastering a new musical form

Round 1 Coding (n=7) Incorrect recall Correct recall Elaborative/prior knowledge/ irrelevant Literal/Direct match Mentioned ‘long distance’ Mentioned ‘music’ Mentioned ‘culture’ # correct 0 1 0 # incorrect 5 0 1 1 0 0 0 4 3 1 1 6

Linguistic Complexity Cognitive Challenges Cognitive errors made by students Recommended Changes Use complex vocabulary for matching even Reduce cognitive load Specific complex though inappropriate: of vocabulary in vocabulary assumptions, distractors and key words unanticipated, communications Links between distractors and text Tendency to choose distractor that links to text Move commonlychosen distractor to end to make it less distracting

Formatting Cognitive Challenges Cognitive errors made by students Changes Passage divided into Recall/Short term shorter sections to memory: Short term memory decrease ‘distance’ 'Distance' challenges/depend between important between ence on inaccurate information and items important recall item is no more than two information and paragraphs from items important information Difficulty in locating specific words/phrases Unable to locate Provide visual clue (bold) important words in to find specific passage words/phrases

Original and Revised Item Original The author’s “difficulty” (line 1) was caused primarily by the A. long distances that had to be traveled B. unanticipated changes in the project C. refusal to question some widespread assumptions D. cultural limitations that hindered communication E. challenge of mastering a new musical form Revised In line 1, what causes the author to have “difficulty” describing why he traveled to West Africa? A. His need to master a new musical form B. The unexpected changes in the project C. His refusal to question some common beliefs D. The cultural differences and language barriers E. His exhaustion from walking long distances 9

Open to closed stem Original: The author’s “difficulty” (line 1) was caused primarily by the A. long distances that had to be traveled B. unanticipated changes in the project C. refusal to question some widespread assumptions D. cultural limitations that hindered communication E. challenge of mastering a new musical form Revised: In line 1, what causes the author to have “difficulty” describing why he traveled to West Africa? A. His need to master a new musical form B. The unexpected changes in the project C. His refusal to question some common beliefs D. The cultural differences and language barriers E. His exhaustion from walking long distances

Vocabulary Original: The author’s “difficulty” (line 1) was caused primarily by the A. long distances that had to be traveled B. unanticipated changes in the project C. refusal to question some widespread assumptions D. cultural limitations that hindered communication E. challenge of mastering a new musical form Revised: In line 1, what causes the author to have “difficulty” describing why he traveled to West Africa? A. His need to master a new musical form B. The unexpected changes in the project C. His refusal to question some common beliefs D. The cultural differences and language barriers E. His exhaustion from walking long distances

Move distractor and change ‘travel’ to ‘walking’ Original: The author’s “difficulty” (line 1) was caused primarily by the A. long distances that had to be traveled B. unanticipated changes in the project C. refusal to question some widespread assumptions D. cultural limitations that hindered communication E. challenge of mastering a new musical form Revised: In line 1, what causes the author to have “difficulty” describing why he traveled to West Africa? A. His need to master a new musical form B. The unexpected changes in the project C. His refusal to question some common beliefs D. The cultural differences and language barriers E. His exhaustion from walking long distances

Sample Manipulations Table 1 2 3 4 5 6 7 8 9 10 11 12 13 Linguistic Close stem Simplify language/vocab in options Change option order * ** * * * ***** * * * Bolding * Add quote to stem * * * Expand stem Spacing * ** Edit attractive distractors Format * ******* * *

III. Impact of Manipulations (pilot study) Do item manipulations minimize the identified cognitive barriers for students with disabilities? For this presentation, we focus on the pilot study, though the cognitive interviews study also analyzed the item manipulations

Formatting & LSA 7 11 16 Harder Than Original 2 1 1 2 Same as Original 32 26 22 16 Blues LSA 0 Totem Formatting Easier Than Original None Rebates Wrappings Read Aloud Summary of Chi-square Analysis of Individual Items 0. 28 0. 24 0. 32 0. 16 0. 33 0. 32 0. 24 0. 20 0. 27 0. 21 0. 40 0. 13 0. 53 0. 71 0. 62 0. 59 0. 30 0. 33 0. 37 0. 59 0. 34 0. 48 0. 39 0. 61 0. 55 0. 46 0. 56 0. 39 0. 51 0. 59 0. 38 0. 42 0. 49 Read Aloud 0. 28 0. 29 0. 32 0. 13 0. 28 0. 25 0. 16 0. 18 0. 24 0. 36 0. 13 0. 54 0. 67 0. 58 0. 65 0. 30 0. 39 0. 36 0. 63 0. 38 0. 45 0. 44 0. 60 0. 41 0. 52 0. 48 0. 64 0. 42 0. 48 0. 41 Formatting 0. 27 0. 40 0. 25 0. 16 0. 39 0. 32 0. 43 0. 26 0. 19 0. 22 0. 30 0. 43 0. 20 0. 65 0. 80 0. 68 0. 64 0. 48 0. 37 0. 51 0. 67 0. 36 0. 55 0. 40 0. 62 0. 47 0. 34 0. 57 0. 48 0. 47 0. 62 0. 41 0. 50 0. 52 LSA 0. 55 0. 23 0. 33 0. 17 0. 45 0. 48 0. 37 0. 45 0. 26 0. 25 0. 34 0. 49 0. 19 0. 52 0. 66 0. 65 0. 76 0. 52 0. 30 0. 42 0. 61 0. 38 0. 55 0. 49 0. 52 0. 56 0. 36 0. 60 0. 57 0. 63 0. 75 0. 59 0. 47 0. 43 Format/ LSA 0. 49 0. 23 0. 35 0. 16 0. 43 0. 41 0. 35 0. 48 0. 33 0. 29 0. 34 0. 52 0. 26 0. 60 0. 70 0. 67 0. 73 0. 55 0. 34 0. 55 0. 67 0. 34 0. 67 0. 55 0. 50 0. 62 0. 31 0. 59 0. 63 0. 64 0. 75 0. 56 0. 60 0. 47

Mean Percent Correct for All Passages Across Conditions No Manipulation Read Aloud Formatting LSA Format & LSA Blues Totem Wrappings Rebates . 27. 24. 29. 35. 36 . 48. 49. 55. 53. 58 . 51. 49. 52. 53 . 48. 49. 50. 57. 60

p-values for ANOVAs – Effects of Manipulation and Design Features on Item Difficulty Design Feature Format Manipulation Design Interaction ---- ---- 0. 003 0. 012 LSA Manipulation Design Interaction <. 001 0. 043 0. 011 0. 001 0. 155 0. 722 0. 003 0. 024 0. 711 0. 001 0. 711 0. 955 <. 001 0. 005 0. 323 <. 001 0. 652 0. 138 ---- Format & LSA Manipulation Compared to No Manipulation Simplify language/ Option Edit Close vocabulary - order attractive Expand stem options changed distractors stem Bolding stem Manipulation Design Interaction <. 001 0. 067 0. 004 <. 001 0. 198 0. 989 <. 001 0. 025 0. 705 <. 001 0. 784 0. 748 <. 001 0. 007 0. 335 <. 001 0. 335 0. 878 <. 001 0. 009 0. 606

All Analyses conducted � Analysis of Item Difficulty ◦ Individual Item Difficulties � Chi-square tests of independence across Manipulation ◦ Mean Item Difficulty � Two Way ANOVA � Manipulation (within-subjects factor) � Design Feature (between-subjects factor) � Analysis of Student Scores ◦ Two Way ANOVAs (separate for each passage) � Manipulation � Previous Reading Performance Level ◦ Two Way ANOVAs (separate for each passage) � Manipulation � Order of Passage � Performance of Students Previously At or Below Chance � Distractor Analysis for Items At or Below Chance � Item Order Effects � Analysis of Missing Data � Point Biserials 18

Conclusions from All Pilot Study Analyses � Blues items were more difficult under all conditions. � Linguistic manipulations (with or without Format modification) had the most positive effect on student scores/item difficulty. ◦ Closing the stem of items was associated with positive change in student scores/item difficulty. � The effects of item manipulations did not interact with prior reading ability (i. e. , Proficiency Levels). � Students perform better/items are easier when passages come earlier. 19

Pilot Study Conclusions (2) � Based on the Cognitive Modeling, it appears that many of the properties of items that make them difficult for the 2% population are the same as for the general population. � Manipulating items in expected linguistic ways will change the difficulty level of the items. � We don’t know enough about what specific manipulations need to be made, though the most conclusive evidence seems to be for the closing of the item stems. � Modifying existing tests may be too constrained to allow for the development of the type of test needed for this population.

Cognitive Interview Conclusions for Sample Item � Restructuring passage can address short-term memory issues � Changing complex vocabulary words seems to increase accessibility, and separates students who just identify key words for matching from those who understand passage � In R 2, students demonstrated greater understanding of item in their cognitive interview responses 21

IV. Student Profiles From target population of study (all students with disabilities who did not reach proficiency on high school reading assessment), how can we develop criteria for identifying students eligible for AAMAS? cp

Students by Disability Category Autism Emotional Disturbance Cognitive Disability Traumatic Brain Injury ADD/ADHD Specific Learning Disability Other Health Impaired Missing Round 1 0 1 2 1 6 10 3 2 Round 2 0 0 3 0 5 20 0 4

Students by Least Restrictive Environment Time in general education Round 1 Round 2 Less than 40% 1 5 40%-79% 7 12 80% or greater 16 14 Missing 1 1

Students by State Assessment Score Round 1 Round 2 Very Low 6 4 Mid-Level 12 15 Nearing proficiency 6 11 Missing 1 2 *All students were below proficient

Round 1 Student Profiles Disability LRE English classes Jennifer Shannon Susan Tony LD Cognitive delay missing OHI >80% <40% >80% 40 -79% Standard/ honors Almost proficient Special education 80% 27% Standard Special English education State score level Very low % Correct on Totem Items 82% Very low 36% Mid-level

Round 2 Student Profiles Disability LRE Richard LD <40% English classes Life Skills State score level Pilot Score (of 34) Pilot % Correct CI Score CI % Correct Almost proficient 15 44% 9/16 56% Emma LD 40 -79% Special education Tom ADD/ADHD 40 -79% Special education Very Low 11 21 32% 1/10 11% 62% 9/18 50%

Richard: Opportunity to learn? Richard Disability LRE English classes State score level Pilot Score (of 34) LD <40% Life Skills Almost proficient 15 Pilot % Correct 44% CI Score 9/16 CI % Correct 56% Mismatch between being almost proficient on state reading assessment and being placed in life skills classes Speech disability limited communication during interview Successful on cognitive interview (56%) but did not demonstrate metacognitive skills. Also successful on pilot. When prompted, he said he chose answers because of “the way it sounds to me”

Richard often said he chose answers because of “the way it sounds to me” A: Because I think it’s showing how they’re made and stuff, I mean, how they are concluded. Q. So why wouldn’t it be (a), (c) or (d)? A: The way it sounds to me.

Tom: Mixed performance, shy, easily distracted Disability LRE English classes State score level Pilot Score (of 34) Pilot % Correct CI Score CI % Correct Tom ADD/ADHD 40 -79% Special education Very Low 21 62% 9/18 50% Largest difference between state assessment score and cognitive interviews (and pilot) Described passions that demonstrate literacy skills but not in accepted contexts (graphic novels) Opportunity to learn? In special education and some very basic nonspecial education English

Tom often used matching and elaborative inference Tom correctly chose E as the answer. A: Because it’s kind of like the difference in the past- or in the years between the past to the modern day, how things change. Q: Okay, how do you know it wasn’t A, B, C, or D? A: It was like what the-it’s not even talking about any of these things right here.

Emma: Fairly consistent low performance Disability LRE English classes State score level Pilot Score (of 34) Pilot % Correct CI Score CI % Correct Emma LD 40 -79% Special education Very Low 11 32% 1/10 11% Not successful in any of the test-taking contexts (slight improvement in pilot) Found it challenging to describe her thinking process in the cognitive interview Placed in special education classes for English We didn’t find evidence that an ART 2% AA-MAS would help her demonstrate more knowledge/skills/ abilities than the regular assessment

Emma often responded: “[I] couldn’t find an answer that fit the question” “Just when I look back, sometimes in the question, they'll give you a list, and then somehow one of those (a), (b), (c), (d) or (e), they give you something that relates to it. …. And then when you go and you read something like this, you're like, ‘What are they expecting? ’ I think it's kind of frustrating when you don’t know what they mean or what they are looking for, and you can't really find an answer that fits the question. ”

Student Profile Conclusions � The 57 students varied in assessment scores, learning settings, ability to engage in cognitive interviews • 75% of Round 2 students varied in their scores across assessment types (state vs. pilot vs. cognitive interview) • Students in special education setting may not be receiving gradelevel instruction • Some students showed no change in demonstration of knowledge/skills/abilities in cognitive interview setting • Some students found the think-aloud strategy helpful as a learning strategy • Some students responded positively to human contact during test-taking � More successful students could: • Demonstrate metacognitive understanding • Use inference appropriately • Understand vocabulary • Filter out irrelevant prior knowledge • Identify important information

Overall Study Conclusion/Policy Implications � There are students who are not being measured well in the current assessment system � It is very difficult to isolate who those students are � Assessment development MUST acknowledge the difficulty of assessing some students � Access to curriculum is still an issue � Paper and pencil changes aren’t enough but tests can still be improved