Consequential Validity A presentation of methods and selected

Consequential Validity A presentation of methods and selected results 1

Background Policy, Validity Framework, Study Design 2

Peer Review Guidance - Critical Element 4. 1 (p. 35) � “Has the State ascertained that the decisions based on the results of its assessments are consistent with the purposes for which the assessments were designed? ” �“Has the State ascertained whether the assessment produces intended and unintended consequences? ” 3

An Argument-based Validity Framework State the intended consequences Define the evidence that supports each consequence Collect data to ascertain the intended consequences Collect data to ascertain the unintended consequences 4

Original Validity Arguments (Lane & Stone, 2004) 1. Teacher and administrator motivation and effort will be evident. 2. Professional development support will be provided. 3. Instruction and curriculum will be adapted. 4. Student motivation & effort will be evident. 5. Performance will improve and be related to the changes above. 5

Additional Validity Arguments NCRRC & 3 State Consortium 6. Access to augmentative and alternative communication will be improved. 7. Parental involvement in student’s academic career will be increased. 6

Collecting Data Teacher Surveys Administrator Surveys Classroom Observation 7

Survey Data Analysis Procedures Design • Arguments • Stakeholders Sample • Stratify • Random Cluster Collect Data • Pilot • Version 1 • Version 2 8

Survey Data Analysis Procedures Prepare Data • CFA • Rasch Scaling Analyze • Descriptive • HLM • Discriminant Function Conclude? Evidence of Consequential Validity 9

Classroom Observation Procedures Identify Elements • Teaching • IEP Development • Student Profile Observe Sample in a sample Modified NAAC Observation Tool Evaluate Is there evidence of consequential validity? 10

Examining Changes Contextual Variables (Relatively Stable) Evidential Variables (Slope of Change) Student Performance (Slope of Change) 11

Baseline Data CONTEXTUAL VARIABLES Gender Race SES Grade/Age School Size LEP EVIDENTIAL VARIABLES Teacher* familiarity Teacher attitudes and beliefs Professional development Instruction in content areas Access to AAC Parent involvement *Includes administrators 12

Transferring the Framework GENERAL EDUCATION A history of assessment Adequate psychometric methods to construct and evaluate assessments SPECIAL EDUCATION Assessment is relatively new Limited psychometric methods to construct and evaluate assessments Academic instruction is special education reform Limited opportunity to learn Academic instruction is the foundation of education In the absence of disability, opportunity to learn and do well on the assessment 13

Selected Results Descriptive Data and Frequency Counts 14

Validity Evidence 1. a Teachers are familiar with the assessment 15

I believe I have a sufficient understanding of the purpose of the AA-AAS. 89% Agree/Strongly Agree 89% 91% State 1 State 2 11% Disagree/Strongly Disagree State 3 11% 9% 16

I believe I have a sufficient understanding of using the results to improve instruction. 61% Agree/Strongly Agree 57% 77% State 1 State 2 40% Disagree/Strongly Disagree State 3 44% 28% 17

Validity Evidence 1. b What are attitudes and beliefs toward the assessment? 18

I believe the current AA-AAS is an effective way to assess students. *Results are similar for math & science 19

It is important to include students with disabilities in statewide assessment and accountability systems 49% Agree/Strongly Agree 42% 48% State 1 State 2 51% Disagree/Strongly Disagree State 3 59% 52% 20

It is important to include students with disabilities in statewide assessment and accountability systems (Responses by assessment complexity) State A (Least complex level) 37% 54% State B (Least complex level) 38% 62% State A (Most complex level) 50% State B (Most complex level) 47% 53% Agree/Strongly Agree Disagree/Strongly Disagree 21

Unintended (negative) consequences From a list generated from the pilot study 22

Unintended (negative) Consequences Creates unrealistic expectations of students Students' educational needs not being met 21% 61% 10% 14% 75% State 1 State 2 Individualization of student programming diminished 13% State 3 33% 64% 13% Limits scope of instruction 31% 72% 23

Validity Evidence 2 Professional development opportunities are being provided 24

Rate the usefulness of the professional development activities that you have participated in since June 2007 25

Validity Evidence 3 Curriculum is aligned with the State’s Alternate Academic Standards 26

The AA-AAS has helped me align my classroom instruction with my State’s Alternate Academic Content Standards * Results are similar for math & science 27

Validity Evidence 4 Students are motivated to learn the material and do well on the assessment 28

Teaching academic content based on the Alternate Academic Content Standards for students with the most significant cognitive disabilities had the following effect on student performance in the classroom. 26% 27% 35% Increased classroom performance 66% 68% 49% No change in classroom performance Decreased classroom performance State 1 State 2 State 3 2% 0% 4% 29

Validity Evidence 5 Student scores are improving as a result of these factors 30

Some Take-Away Messages This framework can be used by any state to gather evidence of consequential validity. Regardless of the maturity of the Alternate Assessment, studies such as these remind us that development of the AAAAS is a beginning, not an end. 31