Monitoring and PreScoring Virginia Alternate Assessment Program VAAP

Monitoring and Pre-Scoring Virginia Alternate Assessment Program (VAAP) Virginia Department of Education Department of Student Assessment, Accountability & ESEA Programs 1

Purpose The purpose of this presentation is to provide guidance to school divisions on monitoring and conducting prescoring activities for Virginia Alternate Assessment Program (VAAP) Collections of Evidence. Department of Student Assessment, Accountability & ESEA Programs 2

Monitoring and Pre-Scoring Plans • Monitoring and pre-scoring plans can be essential to early detection and prevention of issues within the Virginia Alternate Assessment Program (VAAP) Collection of Evidence (COE). • School divisions are encouraged to develop and implement monitoring and pre-scoring plans to ensure VAAP COE are developed under testing conditions. • Ensure scoring rules outlined in the VAAP Implementation Manual and the Virginia Alternate Assessments Administrator’s Manual are adhered to. Department of Student Assessment, Accountability & ESEA Programs 3

Monitoring Periodic and systematic review of evidence-based assessments in the development process with opportunities for feedback and intervention. Why Monitor? • To create better collections of evidence(COE). • To increase the alignment between the student’s level of achievement and the evidence presented in the COE. • To catch potential problems early. Department of Student Assessment, Accountability & ESEA Programs 4

What Monitoring is Not? • Checking with the teacher • Flipping through a collection of evidence • Review without feedback Prevent the “MAD DASH” before the COEs are due Department of Student Assessment, Accountability & ESEA Programs 5

Why collections of evidence are not successful • • • Missing or incomplete Student Evidence Identification (SEI) tags Ungraded or incorrectly graded evidence Use of unacceptable evidence Missing evidence to support the Aligned Standards of Learning (ASOL) Use of evidence not aligned with the ASOL Evidence does not support the level of performance for the ASOL Department of Student Assessment, Accountability & ESEA Programs 6

Monitoring Decisions • • • Who will monitor? How will you train them? When will monitoring occur? Who will receive feedback? How will feedback be given? What are your intervention options? Department of Student Assessment, Accountability & ESEA Programs 7

Components of Monitoring • Trained Reviewers • Feedback Loops • Intervention Options Department of Student Assessment, Accountability & ESEA Programs 8

Trained Reviewers • Trained reviewers may include: • Persons with Content Knowledge and Special Education Experience • • School Test Coordinator Assigned or Hired Staff Building Level Teams Central Office Level Teams Department of Student Assessment, Accountability & ESEA Programs 9

Feedback Loops • • • Teachers preparing collections Principals Building Level Coordinators (VAAP) Trainers/Technical Assistance Staff Service Providers Department of Student Assessment, Accountability & ESEA Programs 10

Intervention Options • Attendance at local training • Access to state and local ASOL resources or VAAP assessment resources • Peer or small group sessions • Individualized consultation and support Department of Student Assessment, Accountability & ESEA Programs 11

A Sample Division Monitoring Plan • School-based Review Teams created and trained • Teachers turn in collections of evidence every nineweeks • All collections of evidence are reviewed by team according to division pacing chart • Team reports shared with principal and central office • Central office assigns Content Teams to address concerns with evidence. Department of Student Assessment, Accountability & ESEA Programs 12

A Sample Monitoring Plan Department of Student Assessment, Accountability & ESEA Programs 13

Pre-Scoring A detailed review of evidence-based assessments prior to submission for scoring. Why Pre-score? • To provide a last check to ensure that all the scoring rules have been followed • To ensure that no last-minute detail has been overlooked • To provide an opportunity to make changes before the COE is submitted for scoring Department of Student Assessment, Accountability & ESEA Programs 14

What Pre-scoring is Not? • A “mad dash” to create a COE • The time to provide instruction Department of Student Assessment, Accountability & ESEA Programs 15

Pre-scoring Decisions • Who will pre-score? • How will you train them? • When will pre-scoring occur in relation to the submission date? • Who will receive feedback? • How will feedback be given? • What are your intervention options? Department of Student Assessment, Accountability & ESEA Programs 16

Sample Pre-scoring Plan Building Level Review • • Review of COE using the 2019 -2020 – Teacher Checklist – Administrator Checklist – Completed Collection of Evidence Check list School-based oversight teams review Building Administrator review and sign-off Return to Teacher to address concerns Department of Student Assessment, Accountability & ESEA Programs Central Office Review • Review of COE using the 2019 -2020 – Administrator Checklist – Completed Collection of Evidence Check list • Reviewed by central office teams Return to schools for correction of identified errors or omissions • 17

Tools for Pre-Scoring VAAP Implementation Manual • Teacher Checklist for Collections of Evidence p. 37 – 39 • Administrator Checklist for Collections of Evidence p. 40 – 41 • Completed COE Submission Checklist p. 42 - 43 Department of Student Assessment, Accountability & ESEA Programs 18

Media Considerations • Do videotapes and audiotapes have transcripts? • Are videotapes and audiotapes clearly labeled with SEI tags? SEI tags should be on the case, not on the DVD. • Is equipment/software needed to review the media clearly identified on the case? Department of Student Assessment, Accountability & ESEA Programs 19

Organizational Considerations • Is the evidence organized according to the scoring worksheet? • Are all required state and local forms completed? Department of Student Assessment, Accountability & ESEA Programs 20

VAAP COLLECTIONS OF EVIDENCE Department of Student Assessment, Accountability & ESEA Programs 21

Collection of Evidence (COE) • The VAAP provides the student the opportunity to demonstrate what he or she knows through a nontraditional method of testing. • By completing products and work samples that demonstrate his/her understanding or skill, under testing conditions the student proves that he/she knows the content. Department of Student Assessment, Accountability & ESEA Programs 22

Collections of Evidence All persons who are responsible for collecting, reviewing, scoring, or handling the COE, or authorized to have access to the COE, must ensure that the student’s information remains confidential and that the evidence is created under testing conditions and maintained in a secure manner. Department of Student Assessment, Accountability & ESEA Programs 23

Testing Conditions • Evidence collected for the VAAP must be completed in the presence of the teacher or other school personnel and under testing conditions. • All curricular materials related to the content and test-taking strategies that might influence student performance or provide an unfair advantage when the student is completing work samples, interviews, and other types of evidence must be removed from the site. • These materials include, but are not limited to: class notes, study guides, graphic organizers, maps, charts, posters, projections, computer programs, textbooks, dry-erase board displays, chalkboard displays, and bulletin board displays. Department of Student Assessment, Accountability & ESEA Programs 24

Testing Conditions (2 of 2) • In addition, directions or examples on work samples must not include hints, clueing, or prompts that provide answers or an unfair advantage for the student. • Students may not have access to any items not approved for use during testing. These items include, but are not limited to: cell phones, electronic devices including wearable technology, class notes, course content review materials, test-taking strategy reminders, notes, “cheat sheets, ” textbooks, and other curriculum materials. Department of Student Assessment, Accountability & ESEA Programs 25

Secure Measures • All COE must be kept in a secure location when not in use; • Establishing and maintaining appropriate monitoring activities for COE while in development; • The School Division Personnel Test Security Agreement for the Virginia Alternate Assessment Program has been signed by all staff who have access to the COE; Department of Student Assessment, Accountability & ESEA Programs 26

Secure Measures (2 of 2) • The Affidavit of Student Performance has been signed by all staff involved with implementation and development of COE; and • Ensuring evidence submitted in the COE was completed under testing conditions. Department of Student Assessment, Accountability & ESEA Programs 27

Violations of Confidentiality • COE left unattended or in a location accessible to persons not involved with the development of the collection; and • COE left in an inappropriate location such as a vehicle or home. Department of Student Assessment, Accountability & ESEA Programs 28

Components of a Collection of Evidence • A COE submitted for scoring to the local school division must contain the following components in this order: – A signed 2019 -2020 VAAP Affidavit of Student Performance ( this includes a signed Interpreter’s Affidavit. ) – Any locally required documentation – A 2019 -2020 VAAP Content Area Cover Sheet – Evidence that demonstrates individual achievement on ASOL selected from each reporting category for a specific grade level and content area. Department of Student Assessment, Accountability & ESEA Programs 29

VAAP Affidavit of Student Performance Ensures that all the work samples included in the COE were completed independently by the student, in the presence of a teacher or other school personnel, and under testing conditions in which the student did not have access to hints, clueing, or prompts that would provide answers. Department of Student Assessment, Accountability & ESEA Programs 30

VAAP Affidavit of Student Performance (2 of 2) • Persons who have supervised the student while completing evidence are required to sign an affidavit. • The principal or designee who reviewed the COE prior to submission for scoring must also sign the affidavit. Department of Student Assessment, Accountability & ESEA Programs 31

Examples of VAAP Affidavit of Student Performance Administrator Signature is dated after the spring 2019 administration has concluded. Has the collection of evidence been reviewed by an administrator? Dates missing from Teacher Signature Department of Student Assessment, Accountability & ESEA Programs 32

Examples of VAAP Affidavit of Student Performance (2 of 3) Date missing for Administrator’s Signature. Department of Student Assessment, Accountability & ESEA Programs Missing Administrator’s Signature and Date 33

Examples of VAAP Affidavit of Student Performance (3 of 3) Department of Student Assessment, Accountability & ESEA Programs 34

VAAP Content Area Cover Sheet • All evidence for an assessed content area should be organized using the 2019 -2020 VAAP Content Area Cover Sheet. • A separate Content Area Cover Sheet must be submitted for each content area. • The Level of Performance is not indicated in the content area of History/Social Science. Department of Student Assessment, Accountability & ESEA Programs 35

VAAP Content Area Cover Sheet (2 of 2) • In addition to providing student information, the cover sheet should – Identify the content area – Identify the Reporting category; and – Identify the ASOL and bullet • The student’s level of performance must be indicated on the Content Area Cover Sheet for the content areas of Reading, Writing, Mathematics, and Science. Department of Student Assessment, Accountability & ESEA Programs 36

VAAP Content Area Cover Sheet and SEI Tag Discrepancy Department of Student Assessment, Accountability & ESEA Programs 37

Scoring Rule 4 Incomplete Student Evidence Identification (SEI)Tags If there is no SEI Tag or if the SEI Tag is mislabeled or incomplete, the evidence cannot be considered in scoring. • Top: SEI Tag is missing the ASOL number. • Middle: SEI Tag is missing the bullet. • Bottom: SEI Tag is missing the content area and the level of performance. Department of Student Assessment, Accountability & ESEA Programs Student Evidence ID (SEI) Tag VAAP CONTENT AREA: RREADING ASOL: 3 E-RW e BULLET: C LEVEL OF PERFORMANCE: III d Student Evidence ID (SEI) Tag VAAP CONTENT AREA: MATHEMATICS ASOL: 7 M-NSCE 2 BULLET: LEVEL OF PERFORMANCE: Student Evidence ID (SEI) Tag VAAP CONTENT AREA: ASOL: 8 S-ECO 2 BULLET: D LEVEL OF PERFORMANCE: 38

VAAP Evidence Department of Student Assessment, Accountability & ESEA Programs 39

VAAP Collections of Evidence Various types of acceptable evidence submitted: • • Work Samples Audio Video Anecdotal Record Department of Student Assessment, Accountability & ESEA Programs Interviews Data Charts/Graph Captioned Photographs 40

VAAP Evidence (2 of 2) Anecdotal records, interviews, and other evidence that are “observations” of student skills and/or knowledge should be written in such a manner that the scorer can determine the level of individual achievement of the ASOL demonstrated. Department of Student Assessment, Accountability & ESEA Programs 41

VAAP Reading, Writing, Mathematics, and Science § § For Reading, Writing, Mathematics, and Science, teachers must select Aligned Standards of Learning (ASOL) only from the grade level in which the student is enrolled. For the content areas listed above, the teacher must use the Level of Performance system to indicate whether the student: § demonstrated the ASOL with significant support and modification (Level I), § partially demonstrated the ASOL (Level II), or § fully demonstrated the ASOL (Level III). Department of Student Assessment, Accountability & ESEA Programs 42

Selecting ASOL The Levels of Performance provide flexibility for instruction and assessment. Examine the ASOL: • Consider all the skills listed in the stem and the bullet • Is the student able to fully demonstrate all the requirements of the ASOL? • Are there specific skills of the ASOL that the student can demonstrate? • Are significant supports and modifications needed to assist the student in demonstrating the ASOL? Department of Student Assessment, Accountability & ESEA Programs 43

Selecting ASOL and Level of Performance Determine the Level of Performance: If the student • can demonstrate all the skills required by the ASOL, consider Level of Performance III. • can demonstrate some of the skills required by the ASOL, consider Level of Performance II. • needs significant support and modification to show understanding of the skills required by the ASOL, consider Level of Performance I. Department of Student Assessment, Accountability & ESEA Programs 44

Science ASOL Example 5 S-FME 1 The student will investigate and understand characteristics and interactions of moving objects. Key concepts include d) moving objects have kinetic energy. Department of Student Assessment, Accountability & ESEA Programs Evidence must show an understanding of: 1. Characteristics of moving objects 2. Interactions of moving objects 3. Moving objects have kinetic energy 45

Reading ASOL Example 4 E-CF 1 The student will b) determine the main idea of a fictional text Department of Student Assessment, Accountability & ESEA Programs Evidence must show an understanding of: 1. Main idea AND 2. Use a fictional text 46

Math ASOL Example 4 M-NSCE 4 The student will b) Add and subtract double-digit whole numbers Department of Student Assessment, Accountability & ESEA Programs Evidence must show an understanding of: 1. Adding double-digit whole numbers AND 2. Subtracting double-digit whole numbers 47

History ASOL Example HS-H 1 The student will recognize that history describes events and people of other times and places by b) identifying the people and events honored by the holidays of Thanksgiving Day; Martin Luther King, Jr. , Day; Presidents’ Day; and Independence Day (Fourth of July). Department of Student Assessment, Accountability & ESEA Programs The student will need to show evidence of: 1. History describes events and people of other times and places. 2. Identify the people and events honored by the holidays of : a) Thanksgiving Day b) Martin Luther King Jr Day c) President’s Day d) Independence Day (Fourth of July) 48

Levels of Performance The Levels of Performance are not considered technical errors. Department of Student Assessment, Accountability & ESEA Programs The scoring event coordinator does NOT have the flexibility to change or adjust the Levels of Performance on the SEI Tag. 49

Hints, Clues, and Prompts Department of Student Assessment, Accountability & ESEA Programs 50

Examples and/or Directions The explanation of how to add and subtract fractions gives the student a hint on how to answer the problems. Department of Student Assessment, Accountability & ESEA Programs 51

Directions on a Worksheet • Instructions provide the student with directions on how to decompose three digit numbers. • This is considered too much information and violates scoring rule #2 “Evidence must be student generated. ” Department of Student Assessment, Accountability & ESEA Programs 52

Directions 8 E-CF 1: The student will (b)Provide a summary of a familiar text This work sample provides direction of what should be included in the student’s summary. This is considered too much information provided to the student. Department of Student Assessment, Accountability & ESEA Programs 53

Explanations 7 E-CF 1: The student will (e) Identify how a character’s point of view is the same or different from another character. • The top of this work sample defines narrator and character’s point of view. Additionally, first person and third person narration are explained with examples provided. • In this work sample the student identifies the first and third person narration. There is no evidence of how a character’s point of view is the same or different from another character. Department of Student Assessment, Accountability & ESEA Programs 54

Explanations (2 of 2) Explanation of how to the solve an algebraic expression using simple addition and subtraction was provided to the student. Department of Student Assessment, Accountability & ESEA Programs 55

Definitions The work sample clues the student by providing definitions of predator and prey, the evidence cannot be considered for credit. Department of Student Assessment, Accountability & ESEA Programs 56

Verbal Hints The definition of fact and opinion were removed. A note is provided indicating the definitions of fact and opinion was read to the student. This considered as hinting and clueing the student to what facts and opinions are. Department of Student Assessment, Accountability & ESEA Programs 57

Physical Prompts Department of Student Assessment, Accountability & ESEA Programs 58

Physical prompts: Hand-over-Hand Department of Student Assessment, Accountability & ESEA Programs 59

Gestural Prompts The teacher pointed to the correct answer, providing too much help. Department of Student Assessment, Accountability & ESEA Programs 60

Written Prompting Department of Student Assessment, Accountability & ESEA Programs 61

Restating the ASOL HSS-COT 1: The student will investigate and understand that many aspects of the history and evolution of Earth and life can be inferred by studying rocks and fossils. Key concepts include: Red hi (a) traces and remains of ancient, often extinct, life are preserved by various means in many sedimentary rocks; Restating the ASOL does not show any demonstration of the student’s individual achievement. Department of Student Assessment, Accountability & ESEA Programs 62

Open Book Scoring Rule 2: If the student work is based on homework or an openbook, the work cannot be considered for scoring. The student is instructed to find the answers in the text book. The evidence cannot be considered for credit. Department of Student Assessment, Accountability & ESEA Programs 63

Score of “ 0” • A score point of “ 0” may be assigned if the evidence does not demonstrate any level of individual achievement for the ASOL being defended. Department of Student Assessment, Accountability & ESEA Programs 64

Example of Score of “ 0” • 8 E-CN 1 a The student will – a) cite text to support inferences from informational text. The evidence defends statements that are a fact or an opinion. Department of Student Assessment, Accountability & ESEA Programs 65

Evaluative Comments Evaluative comments can include: • Check Marks • Smiley Faces • “Good Job” Evaluative comments are not: • A statement that a student completed the activity “independently Department of Student Assessment, Accountability & ESEA Programs 66

Accuracy Score Examples Grade Check Marks Department of Student Assessment, Accountability & ESEA Programs Comment Stickers 67

Monitoring & Pre-scoring Eliminates Surprises by reducing or eliminating COEs with: • Incomplete or mislabeled SEI tags • Missing evidence • Inaccurate or ungraded evidence • Unacceptable evidence (copied from textbooks, homework, hand-over-hand) Department of Student Assessment, Accountability & ESEA Programs 68

Monitoring & Prescoring • Ensures collections required for students educated outside of the division are completed • Prevents incomplete collections in the division • Prevents collections not driven by IEP, 504 or EL Student Assessment Participation Plans Department of Student Assessment, Accountability & ESEA Programs 69

For questions, please contact: Student_Assessment@doe. virginia. gov (804) 225 -2102 Department of Student Assessment, Accountability & ESEA Programs 70