2011 STAR Nuts and Bolts Riverside County Assessment

2011 STAR Nuts and Bolts Riverside County Assessment Network March 4, 2011 Wes Scott Key Data Systems 1

Overview • • What is new in 2011 Who Takes What Test To Be Gridded by All Students Hand-gridding demographic data Irregularities and Post test gridding Scorables and non-scorables packaging Special Education issues Considerations and Reminders 2

What is new in 2011 • CMA is now. . . – ELA: Grades 3 – 11 – Math: Grades 2 -7, Algebra I, and Geometry – Science: Grades 5, 8 and 10 • Grade 4 Writing is back • 2 minor editorial changes in box A 3 of the answer documents 3

CMA • • ELA grades 3 -11, Math grade 3 -7, Algebra I, Geometry, Science 5, 8 & 10 May test CMA in one content area and CST in another Can not test CMA and CST in same content area If CMA, can not assess with CAPA Take grade 4 or 7 CMA writing if taking CMA ELA One more test section than CST, all content areas, all grades Fewer answer options (3 in CMA versus 4 in CST) Fewer items – Grade 3 -5 – 57 items, 48 count – Grade 6 -8 – 63 items, 54 count – Grade 9 -11 – 70 items (Grade 10 science is 66 items), 60 count • Accountability – CMA grades 3 -8, ELA will count in AYP – CMA grades 3 -9 in 2010 Base API & 2011 Growth API – CMA Geometry, grade 8, will count in AYP, December release, not in API 4

CMA (cont. ) • Same participation criteria as last year – CST scores in any previous year of Far Below Basic or Below Basic OR Scoring Proficient or Advanced on TWO previous CAPA administrations – Multiple measures of assessments – Determined will not attain grade level proficiency – Response to grade level instruction – IEP must state the student will take the CMA, and in which content area, before STAR testing 5

Answer Document Change • Box A 3 added subsection identification (this is a grade 10 answer document) 6

Answer Document Review • Box A 2 – There is no ‘All Tests’ column – In alphabetical order – A column for each possible test for this grade level (varies with grade level). This sample is grade 5. 7

Answer Document Review (cont. ) • Box 7 c on CST answer document. Used to identify which CMA tests will be taken and scored • Separate answer document for CMA Grades 3, 8 -11 • Grades 4 -7 answer documents include CST and CMA answer sections. . . CST on left and CMA on right. • CST and CMA version numbers at top right portion of page. Required – sets scoring key. • CMA section has box around it and different icons. 8

Grade 5 Answer doc CMA Version Box around CMA answer section 9

Race/Ethnicity • Box 8 – Hispanic/Latino – Yes/No bubbles • Box 9 – Race. To be filled in regardless of box 8 answer (Federal rules) • California says “do not guess” if race is blank • API, AYP, If Box 8 is ‘Yes’, will be Hispanic regardless of Race bubbles • If Box 8 is ‘No’ and no Race, will not be in any ethnic group. • If Box 8 is ‘No’ and bubbles in two or more racial groups, ‘Two or More Races’ subgroup • Note: races listed alphabetical and not by group 10

Who Takes What Test • All students take CST, CMA or CAPA, except… – If student already tested in a previous school – Student enrolled after the first day of testing (testing optional) – Parent Exemption • If less than 85% tested in any grade level test, then the school will not get an API. Grade level tests include: – – ELA grades 2 -11 Math grades 2 -9 Science in grades 5, 8 and 10 HSS in grades 8 and 11 • AYP – Grades 2 -8 require 95% participation – High school without a grade 10, STAR results are used 11

Who Takes What Test (cont. ) • Math grades 7 -11 (pages 15 -17, D-1 in District/Site Coordinators Manual) – Take End-Of-Course (EOC) test if in standards aligned course • Grade 7 – Grade 7 Math or Algebra I • Grade 8 & 9 take EOC Math or General Math (based on what Math course the student will complete this school year) • Grade 10 & 11 take EOC Math unless not in a standards aligned course (e. g. Business math) • If passed Algebra I and II and Geometry (new in 2010) or Integrated Math III and not taking any Math course OR taking higher-level math, take Summative Math • Grade 10 & 11, take no Math test if will not complete standards aligned course and don’t qualify to take the Summative Math test • If no EOC Math test taken, API score of ‘ 200’ (Far Below Basic) at a reduced weight (. 10 versus. 20) 12

Who Takes What Test (cont. ) • Science grades 9 -11 (pages 17 -18, D-2) – Grade 10, all students, take Life Science test – Take EOC test if in standards aligned course • Some district flexibility on this. – AP courses take corresponding CST – Higher level science, may take most closely aligned CST • No EOC Science test – API score of ‘ 200’ (Far Below Basic) at a reduced rate (. 05 versus. 22) – If take Grade 10 Life Science test, no ‘ 200’ – Some grade 10 students take two Science tests 13

Who Takes What Test (cont. ) • HSS Grades 9 -11 – World History is an EOC test. • Assess when student takes World History course • May take more than once if retaking course • If taking in grade 11, will take two HSS tests – Grade 11 – U. S. History (all grade 11 students) 14

Who Takes What Test (cont. ) • Can retake EOC Math and/or Science if… – Student failed course and is retaking the course – Student is taking AP course after completing the standards based course (e. g. AP Chemistry) – Student is taking General Math in grades 8 and 9 – Student is taking the Summative Math test 15

STS Grades 2 -11, ELA and Math Taken in STAR testing window STS Math, grades 2 -7, Algebra I and Geometry Take Algebra I/Geometry if taking equivalent CST math test • Same criteria for who takes STS • • – English Learners in US school less than 12 months (cumulative) – EL’s receiving academic instruction in Spanish – Any Spanish speaking EL may take STS (district decision) – Non-EL’s (English Only, IFEP, RFEP) can NOT take STS 16

To Be Gridded by All Students • • Box 1 – student name, teacher, etc. Box 7 (grade) Box 7 c – CMA only Grade 7, page 2, Math test taken, version (Algebra 1). Will be scored as 7 th grade math if not bubbled Note Special stopping place depending on Math test taken for CMA • Grade 4 -7, page 3 – CMA version (if applicable) 17

To Be Gridded by All Students • Grades 9 -11 – Page 2 – CST Science test taken and version – Page 3 – CST Math test taken and version (grades 8 -11) – Page 2 – HSS version • Grade 11, page 2, EAP-ELA (bubble releasing EAP scores) • Grade 11, page 3, EAP-Math (bubble releasing EAP scores) 18

Box 1 Grade CMA – 7 c Hispanic or Latino SSID 19

ELA Version EAP (CSU and CCC) World History Version 20

Math Taken And Version EAP Science Taken and Version 21

Hand-Gridding Demographic Data • Why hand-grid an answer doc? – Student is new to the district since pre-id • Alternative - Late pre-id up to one week before end of testing window – Student changes schools or tracks after pre-id submitted • STAR web site option to change schools – only need to change CBEDS – Will result in a late label being created by ETS. • Can take pre-id document to new school – online update CBEDS • If change to new wave, void first pre-id and do late pre-id in new wave – Late pre-id available until week before end of testing period 22

Hand-Gridding Demographic Data (cont. ) – Believe pre-id data is incorrect • Alternative - Online data correction available until last day of testing – Pre-id answer document becomes unusable – Pre-id for incorrect assessment (e. g. should be CMA, not CST) • If should take CMA and pre-id CST, or visa versa, and grade 3 • If CMA but pre-id CST, grade 4 -7, use pre-id answer document • If CMA but pre-id CST, grades 8 -11. Hand-bubble CMA document – Only bubble boxes 1 -7 c, 11 and 12 – Be sure SSID and Name spelling exactly matches pre-id document 23

Hand-Gridding Demographic Data (cont. ) • Gridding by District staff – not students • All data is important – leave no blanks except those that don’t apply to student (e. g. no RFEP info if not RFEP student) • Accuracy is vital as it impacts AYP and API • Box 10, Parent Education – Similar Schools (Parent Ed has most impact of any component) – Socio-economically disadvantaged subgroup • Box 11, student id, do NOT add zeros on the right side. • Box 12, SSID, always 10 digits, never starts with zero – Used for matching Writing, CST, CMA, EAP • Grid all available bubbles (e. g. 2 bubbles for day of month and all bubbles for years), ETS does not default any values 24

Hand-Gridding Demographic Data (cont. ) • Box 14 (CBEDS). If left blank, CDE treats as YES – may be bad • Box 16 c (Special Ed Exit Date). Only if 16 a is ‘ 000’. Very important for AYP • Box 20 – only for students who live outside district but attend district to receive special ed services per an IEP. Must be valid county/district code • Box 22 (RFEP Proficient/Advanced 3 times on CST ELA, SINCE RFEP date). If left blank (grades 5 -11), CDE treats as YES – bad. Has to be blank for grades 2 -4. • Boxes 15 (English Proficiency), 22 (RFEP CST ELA 3 times) and 24 (Date first entered a U. S. School) used to determine EL subgroup • 3% rule: Gender, Hispanic or Latino, Parent Ed, SSID, CBEDS, English Proficiency, Disability code, NSLP (boxes 6, 8, 10, 12, 14, 15, 16 a, 23) – – If more then 3% blank for a school, ETS scoring is stopped District must correct (cost is $1. 32 per student corrected) Corrections need to be made within 7 days (approximately) Only after corrections made will processing continue. 25

CBEDS English Proficiency Special Ed Info County/District of Residence RFEP 3 Year RFEP Date NSLP EL Date First Enrolled in US Special Ed Exit Date 26

Most Frequent Errors • • • Missing CBEDS (D-283/S-265) Missing or bad EL Date (487) Missing RFEP 3 -Year Prof/Adv (141) Missing NSLP (924) Missing Parent Education (751) Missing English Proficiency (532) Missing Hispanic indicator (235) Missing Special Ed Exit Date ? ? ? Bad IEP County/District of Residence (262, 145=current district) 27

Hand-Gridding Demographic Data (cont. ) • Common questions – What is continuous enrollment? • Enrolled on or before CBEDS Information day (first Wednesday in October) and has not been disenrolled since that date • Requires rigorous record keeping • Used to determine who is in API and Proficiency on AYP – What value should be used for ‘Date First Enrolled in a US school’ if they were born in the U. S. ? • When they first entered school…likely kindergarten 28

Irregularities • See Appendix C (page 76) for guidelines and reporting forms • Irregularity – helping students or security breaches Examiner reading the test booklet (except grade 2) Anything left on the classroom walls that will help a student Test booklets not being locked up Test booklets can NOT be taken out of the room except when the teacher is returning them to the secure site – Preview/review of test booklets by anyone (exception for grade 2 teachers on the first day of testing) – Teacher interpreting items for students – Test booklets being used by more than one student – – 29

Irregularities (cont. ) • Report irregularities and Inappropriate Test Preparation to the Site Coordinator who reports it to the District coordinator • District coordinator fills out the form (no student or teacher names are to be mentioned) and emails or faxes to the CDE (no calling) • CDE will tell District coordinator which bubble, if any, to fill in box A 1 30

Post Test Gridding • Be sure Box 1 is complete • Be sure all student gridding is complete and accurate (see earlier slides) • If hand-gridded demographics, be sure it is complete (AYP and API impact and/or big dollar cost to correct later) • Boxes, A 2, A 3 and A 4 – See last page of DFA for listing of codes (content varies with grade and test) 31

Post Test Gridding (Cont. ) • Box A 2 – Testing Conditions – Valid values • Pages 115 -116 – Significantly impact AYP & API • E, L, M, T exclude from AYP enrollment, unless. . . – Sample is grade 5 32

A 2 Accountability Impact AYP API Enroll Tested Proficient Valid Score A-Absent N N* C-Cheat N B* E-Medical Emergency N N* F-Sick – did not complete ? N* ? L-Enroll after first day N^ N*^ M-Enroll first day then moved N^ N*^ P-Parent Exempt N N* T-Previously Tested N N X-Refused acc/mod n/a n/a n/a Y-Replaced n/a n/a n/a N Z-Not attempt ? = Depends on how many items attempted and score B* = Include in Base API, exclude from Growth API N* = Excluded if no items attempted N^ = Excluded unless CBEDS=Y 33

A 2 – Errors • L – Enrolled after first day and tested – ELA – 71, 13 had CBEDS=Y – Math – 62, 13 had CBEDS=Y • T – Tested in a previous district – ELA – 26, 4 had CBEDS=Y – Math – 20, 4 had CBEDS=Y • Most of L and T above were not proficient 34

A 2 – Errors (cont. ) A – Absent, 192 -ELA scores, 109 -Math scores E – Medical Emergency, 5 -ELA, 4 -Math scores P – Parent Exempt, 41 -ELA, 3 -Math scores M – Enrolled first day of testing, moved before test was given, 38 -ELA, 58 -Math scores • A, P, and M will be included in AYP/API. • • – E is excluded from AYP regardless of scores 35

Post Test Gridding (Cont. ) • Box A 3 – Accommodations/Modifications (page 83 -87 & 143 in District Coordinators Manual) – If in IEP and used in instruction, student must be allowed to use accommodation/modification – If not in IEP, can NOT use it – ONLY mark if student was observed using it, if not used, mark box A 2, row X. – Accommodations do not impact AYP or API – Modifications count as Far Below Basic for API and ‘not tested’ for AYP • A 4 – English Learner Test Variations – No IEP required – English Learners Only – Glossary/Word list • Can not use for ELA • Translations only – no definitions 36

Scorables and Non-scorables Packaging • Scorables (orange label) – All answer docs (includes grades 2 & 3 test booklets) – *If Grade 3 absent, pre-id CMA & CST, only return one answer document with the scorables. – SGID sheets (more on this later) – Master File Sheets – Processing Form (district only) – See pages 40 -42 of the District and Test Site Coordinators Manual for a comprehensive list and the required packing order 37

Scorables and Non-scorables Packaging (cont. ) • Non-Scorables (green label) – CAPA Examiner’s Manuals – Grade 2 DFAs – Test booklets – Voided answer docs – Braille and Large print booklets 38

Scorables and Non-scorables Packaging • Following packaging instructions to the ‘T’ – 1 Master File for school for each admin – SGID for each grade (one per teacher if want teacher reports) and one for CAPA – CAPA on top of lower grade (do not return stimulus cards) – Lowest grades on top – Highest grades on bottom – STS under CST 39

Scorables and Non-scorables Packaging • SGID – Sets school code and grade for all answer documents (that’s why one can move pre-id answer docs with student if they change schools) – *Filling in teacher name on back will save that data in the results file – School code must be ‘ 0000001’ for NPS – No grade for CAPA – Back • Instructional Materials for grades 2 -8 only – CAUTION: If SGID grade and answer doc grades are different • Student counts as participating • Student will receive no scores 40

Scorables and Non-scorables Packaging • When return Multiple-Choice and CAPA – Site: 2 working days after last test for that administration – District: 5 working days after last test for that administration – Keep administrations separate – Track all documents, from original receipt until shipping back to ETS, in order to not have to pay excessive order charges 41

Special Education Issues • No Below Level Testing • Be sure all Special Education students are tested • CAPA and CMA – Scores count the same as CST for AYP and API – Assess in STAR testing window – Assess all who qualify – Appropriate test for the student – Can only help AYP and API 42

Special Education Issues (cont. ) • CAPA and CMA limits – applied in March 2011 AYP release – CAPA 1% rule (AYP only) • Applies to non-mobile students only • If number Proficient or Advanced is more than 1% tested in district, those over 1% are considered not proficient for AYP purposes • *File exception request (due around July 1) if even close to 1% – CMA 2% rule (AYP only) • Applies to non-mobile students only • If CMA Proficient or Advanced is more then 2% tested, those over 2% are considered not proficient for AYP purposes • No exceptions allowed – CAPA and CMA 3% rule (AYP only) • Combined CAPA and CMA can not exceed 3% • If CAPA exception granted, ‘combined’ limit is 2% plus CAPA exception amount. 43

Special Education Issues (cont. ) • CMA – Grade 4 -7 answer document, cover up or fold under CST answer page when testing CMA – Test separately from CST because CMA DFA and number of sections is different than CST – Coordinate schedule with CST since some students take CMA and CST (e. g. Grade 8 CMA students will take some CSTs (at least HSS)) 44

Special Education Issues (cont. ) • Accommodations and Modifications – See Pages 83 -87, 143 for complete list • Separate list for CST (page 83 -87) and CMA (page 143) – IEP can not exempt a student from testing – Accommodations • Result in a valid test • Must match instructional accommodations – Modifications change test interpretation • Actual score on individual student reports • Far Below Basic for API • Considered not tested for AYP 45

Special Education Issues (cont. ) • CMA Accommodations that are CST Modifications – ELA • American Sign Language/Manually Coded English to present questions • ELA Test questions/answer options read aloud – Math • Use of Calculator on grade 5 math test • Math Manipulatives on Math or Science tests • If modification required, must take CST (e. g. use dictionary) 46

Testing Considerations • • • Test early in the day Test with normal teacher if possible (especially for Elementary) Consider not testing on Monday and Friday Allow plenty of time for make-ups Tests are untimed (within reason) – set up room for those needing additional time Grades 2 and 3, limit to one section per day No breaks within a section except supervised bathroom breaks Multiple sections must be administered sequentially Multiple sections in a day require a 10 minute break between sections If complete a section, after starting the next section, can not go back and review answers in first section Do not administer CMA and CST in the same room at the same time 47

Additional Reminders • Distribute non-secure DFAs (grades 3 -11) before first day of testing • Grade 2, secure version, DFA (includes items) can NOT be distributed before first day. Non-secure version is available on STAR web site. • *Grade 2, all test booklets need to be the same version in a classroom • Distribute test materials first day of testing…not earlier • Have student write their name on the answer document and all test booklets • Be sure students only get their answer doc and test booklets • Students can write in test booklets – Grade 2 & 3. Erase extraneous marks. Recommend no use of Highlighters. If too much bleed-through, transcribe to a new answer booklet and ‘void’ the original answer booklet • Excessive erasures (all grade levels) raises a red flag for CDE • Immediately report possible irregularities • Administrative errors (page 78 & Form A-5 page 80) need to be documented but are not reported to the CDE 48

Additional Reminders (cont. ) • Box A 3. Note that each DFA lists only accommodations/modifications applicable to that grade level/test. Writing has additional codes. • *Unlisted Accommodations/Modifications must be cleared through the CDE prior to testing…do it 30 days before • Grade 2, teacher read items…do not mark as ‘read aloud’ modification when DFA says to read the item aloud. • Directions for Administration (DFA) – Must read ‘Say’ boxes exactly as written – ‘T’ means can translate ‘Say’ box • *No modifications allowed for CMA. If modification required, must take CST • Before testing, give teachers list of any demographic information needing to be gridded. • Request ETS to send data as ASCII text, not XML • There is a late processing fee of $385 if ‘extra’ answer docs sent in late and ETS has already processed the district data 49

Additional Reminders (cont. ) • Miscellaneous – CMA – Braille for grades 3 -8 except for Algebra I – Call ETS if running short on materials. They can deliver quickly using Supplemental Order process (need to call ETS) – Be aware of excessive order charge (if use less than 90% of materials ordered. . . ETS overage is excluded from calculation). – No Teacher Reports – API includes CAPA science (grades 5, 8 and 10) 50

Final Thoughts • Be familiar with the following. They are all available on the STAR web site. – Directions for Administration (all grades and test types) – District and Test Site Coordinator Manual – Sample answer documents • Maintain strict security • Accuracy of demographic and other gridded data on the answer document is vital 51

Contractor Contact Information • STAR (ETS) – TAC: 1 -800 -955 -2594 – FAX: 1 -800 -541 -8455 or 1 -925 -808 -2118 – STAR@ets. org – www. startest. org • CDE STAR Office – Phone: 1 -916 -445 -8765 – STAR@cde. ca. gov – www. cde. ca. gov 52

QUESTIONS? Wes@Key. Data. Sys. com 951 -245 -0828 x 212 53