PM 1 Assessment Literacy Series PA Module 1

PM 1 Assessment Literacy Series: PA Module #1 Design © Pennsylvania Department of Education

PM 2 © Pennsylvania Department of Education 2

§ Content Documents § Item and operational form specifications § Theory of Action § Developmental readiness § Test and score descriptions STUDENT ACHIEVEMENT © Pennsylvania Department of Education 3

PM 2 § 1. number of items and score points § 2. range of difficulty (easy to hard) § 3. decisions made based on test results © Pennsylvania Department of Education 4

OBJECTIVES Participants will be able to: 1. Create an assessment’s purpose statement. 2. Identify targeted content standards to be measured. 3. Develop specification tables that create an assessment’s blueprint. © Pennsylvania Department of Education 5

RESOURCES Power. Point Handouts § Handout #1. 1 – Purpose Statement Examples § Handout #1. 2 – Targeted Content Standards Examples § Handout #1. 3 – Specification Tables Examples Templates § Template #1. 1 – Purpose Statement Framework § Template #1. 2 – Targeted Content Standards Framework § Template #1. 3 – Specification Tables Framework § Table 1. Content Match and Depth of Knowledge § Table 2. Content Pattern and Item/Task Sufficiency § Table 3. Passage and Writing Prompt Distributions § Template #6. 3 – Quality Control Checklist Participant Materials © Pennsylvania Department of Education 6

ASSESSMENT CYCLE Establish Purpose and Design Apply QC & Refinements Select Content Standards n sig Re vie w De Implement Data Analysis Build Test Specifications Conduct Alignment Reviews Create Test Forms & Guidelines © Pennsylvania Department of Education Develop Scoring Key. Rubrics Develop Items/ Tasks 7

Task: Review Approved Content Standards Tool: SAS Portal pdesas. org © Pennsylvania Department of Education 8

Objective/Task 1. Create an assessment’s purpose statement. Module 1. 1 Purpose Statement Objective/Task 2. Identify targeted content standards to be measured. Design Module 1. 3 Specification Tables © Pennsylvania Department of Education Objective/Task 3. Develop specification tables that create an assessment’s blueprint. Module 1. 2 Targeted Content Standards 9

§ SAT § Text complexity § Passage selection procedures § Key features of the assessment § PSSA/Keystone Exams § Qualitative Evaluation of the Text § Quantitative Evaluation of the Text © Pennsylvania Department of Education 10

MODULE 1. 1 PURPOSE STATEMENT © Pennsylvania Department of Education 11

PM 3 VALIDITY The validity of a test refers to the degree of accuracy with which it measures what it intends to measure. Without a relatively high degree of validity, a test is in effect a useless device as it is not serving the purpose for which it was designed. © Pennsylvania Department of Education 12

Template #1. 1: Creating an Assessment’s Purpose Statement PURPOSE STATEMENT An assessment’s purpose statement outlines why the assessment was developed, what the assessment is measuring, and how the results (scores) can be used. © Pennsylvania Department of Education 13

PM 3 Statement outlining… § why the assessment was developed. Types of validity evidence related to the purpose: • Convergent/ Discriminant • Test-criterion • Consequential © Pennsylvania Department of Education 14

Statement outlining… §what the assessment is measuring. CHARACTERISTICS OF TEST TAKERS © Pennsylvania Department of Education age gender academic preparation language skills ethnicity and so on… 15

Statement outlining… §how the scores should be used. USE OF TEST SCORES Reporting to Parents Rank order of students Promotion Certification Teacher evaluation © Pennsylvania Department of Education 16

Handout #1. 1: Creating an Assessment’s Purpose Statement Identify key decision makers Create a draft purpose statement Evaluate the assessment's design against its intended purpose © Pennsylvania Department of Education Schedule and plan meeting Discuss the assessment and rationale for its use Examine draft, refine, and reach consensus Document the process and decisions made Publish results 17

Handout #1. 1: Creating an Assessment’s Purpose Statement 1. Individually create a statement about the performance measure in terms of the content standards it is intended to measure. 2. Build consensus by focusing on three components of the statement: Why, What, and How. 3. Draft a sentence reflecting the group’s consensus for each component and review. 4. Merge sentences to create a single paragraph “statement”; review to ensure that the statement reflects the group’s intent. 5. Finalize the purpose statement and review for editorial soundness. © Pennsylvania Department of Education 18

Handout #1. 1: Creating an Assessment’s Purpose Statement T 1. 1 “The Grade 8 Pre-Algebra assessment is provided as a post-test to determine student mastery of the content standards. This grade-level summative assessment is intended to measure student proficiency of grade-level expectations in the sequence of the district’s algebra curriculum. Scores represent degrees of content knowledge, and are used as part of pupil progression for 8 th grade students. ” © Pennsylvania Department of Education 19

Template #1. 1: Creating an Assessment’s Purpose Statement T. 1. 1 QC CHECKLIST q Statement is clear, concise, and free of technical jargon. q Statement identifies which targeted content standards the performance measure is designed to evaluate. q Statement articulates how the information from the performance measure is intended for use, and provides insight about what the scores mean. © Pennsylvania Department of Education 20

Item Construction Design Purpose Form Construction Validity © Pennsylvania Department of Education 21

MODULE 1. 2 TARGETED CONTENT STANDARDS © Pennsylvania Department of Education 22

Opportunity to Learn Teaching Testing © Pennsylvania Department of Education 23

Constraints Testing Format Testing Time Developmental Readiness Volume Reading Comprehension in Grade 10 BIG IDEA-ESSENTIAL QUESTION-CONCEPTS-COMPETENCIES Over 20 STANDARDS! © Pennsylvania Department of Education 24

Handout 1. 2 Template #1. 2: Quality Control Checklist ASSESSMENT DESIGN “Targeted Standards” Design Tasks § Before the assessment is designed§ Task: Review Approved Content Standards § Tool: § After the assessment is designed§ Task: Verify all design tasks are completed § Tool: Quality Control Checklist: § Template 1. 2 © Pennsylvania Department of Education 25

© Pennsylvania Department of Education 26

An assessment’s targeted content standards are those standards you are choosing to assess. Select standards that… § represent Big Idea/Enduring Understanding statements within the content area. § focus on the essential knowledge and skills for which students are expected to demonstrate achievement. § create transparency for families and the community about what is most important for student success. © Pennsylvania Department of Education 27

Targeted Content Standard - Is this standard a component or dimension of a Big Idea /Enduring Understanding statement? • Targeted standards are a sample of the available standards within a key concept. • When comprised of multiple tasks, performance measures typically sample skills and knowledge from several standards. © Pennsylvania Department of Education 28

TARGETED STANDARDS SELECTION CRITERIA PM 4 -7 1. Endurance - Will this standard provide students with knowledge and skills that will be of value beyond a single test date? 2. Leverage - Does this standard provide knowledge and skills that will be of value in multiple disciplines? 3. Readiness for Next Level of Learning - Will this standard provide students with essential knowledge and skills necessary for success in the next level of learning? © Pennsylvania Department of Education 29

Handout 1. 2: Targeted Content Standards Workflow Examine those standards that reflect the key concept being measured Identify requirements (time, money, educators, student needs, policies, etc. ) © Pennsylvania Department of Education Identify those standards that can be measured by the assessment's design Examine draft listing of standards, refine draft, and reach consensus Flag those identified standards as "draft" targeted standards Document the process and publish 30

Handout 1. 2: Procedural Steps Template 1. 2: Framework PROCEDURAL STEPS 1. Place the course/subject’s name and Big Idea/Enduring Understanding statement above the Targeted Content Standards table. 2. Place the code for each standard/content strand in the Standard ID column along with the description for each content standard in the Standard Statement column. 3. Have a subject matter expert work collaboratively to identify an initial (i. e. , draft) set of content standards associated with the Big Idea/Enduring Understanding column. 4. Review the list of targeted content standards, look for gaps and/or redundancies, and then finalize the list by placing an “X” in the Final column. 5. Verify that the “final” targeted content standards will be those used to develop the test specification tables. © Pennsylvania Department of Education 31

TARGETED CONTENT STANDARDS ELA 6 EXAMPLE T 1. 2 Big Idea/Enduring Understanding: Students will interact with non-fictional text, providing critical thinking and responses. Students need to be able to question, reflect on, and interpret essential content across texts. They also need to be able to cite evidence to support analysis. Handout #1. 2: ELA Gr 6 Example Targeted Content Standards Standard Statement ID CC. 1. 2. 6 A CC. 1. 2. 6 B CC. 1. 2. 6 G CC. 1. 2. 6 H CC. 1. 2. 6 I Determine the central idea of a text and how it is conveyed through particular details; provide a summary of the text distinct from personal opinions or judgments. Cite textual evidence to support analysis of what the text says explicitly as well as inferences and/or generalizations drawn from the text. Integrate information presented in different media or formats (i. e. visually, quantitatively) as well as in words to develop a coherent understanding of a topic or issue Evaluate an author’s argument by examining claims and determining if they are supported by evidence. Examine how two authors present similar information in different types of text. © Pennsylvania Department of Education Draft Final X X X X X 32

T 1. 2 Template 1. 2: QA Checklist q Do the targeted content standards have endurance and leverage? q Do the targeted content standards represent a Big Idea/Enduring Understanding within the greater content area? q Do the targeted content standards prepare students for the next level of learning? © Pennsylvania Department of Education 33

MODULE 1. 3 SPECIFICATION TABLES (BLUEPRINTS) © Pennsylvania Department of Education 34

If the purpose is to build a cruise ship, this may not be the best blueprint! © Pennsylvania Department of Education 35

• • • • Number and types of items on the final assessment; Number and types of items to be developed; Content, skills, and any other attribute to be measured; Details about critical content included or excluded; Degree of match to content standards; Degree of match to alternate achievement or assessments; Intended level of difficulty; Target distribution of item difficulties; Intended examinee population, including major subgroups; Mode of administration; Material that represents different population subgroups; Rules for selecting and sequencing items; Relative weights assigned Timing of the total administration/separate sections or subparts Directions for examinees. © Pennsylvania Department of Education 36

Definition § A chart that describes the relationship between the attributes and characteristics of an assessment. § An assessment’s design is articulated in a series of specification tables. These tables (taken together) describe the content and structure of the assessment. Specification Table + Specification Table = Blueprint § The specification tables in this module define the § total number of items/tasks, § the content standards associated with the items/tasks, item/task type, § cognitive demand, and § other relevant information. © Pennsylvania Department of Education 37

SPECIFICATION TABLES (CONT. ) Importance § Guide item/task development process. § Ensure sufficient sampling of targeted content standards. § Assist in aligning the assessment. § Outline the general structure and design of the assessment. © Pennsylvania Department of Education 38

PM 8 SPECIFICATION TABLES (CONT. ) Components § Big Idea/Enduring Understanding • Statement summarizing the core knowledge and skills enumerated within the content standards. § Targeted Content Standards • Identified standards representing the Big Idea/Enduring Understanding statement that can be measured in a summative manner. Big Idea/Enduring Understanding Standard ID 4. 2 © Pennsylvania Department of Education 39 39

PM 8 SPECIFICATION TABLES (CONT. ) Big Idea/Enduring Understanding Standard ID Mathematical statements can be justified through deductive and inductive reasoning and proof. 2. 4. G. A Write formal proofs (direct proofs, indirect proofs/proofs by contradiction, use of counter-examples, truth tables, etc. ) to validate conjectures or arguments. Item Types PT Total Points 44 2 0 0 2 0 0 20 SR SCR ECR SR-Selected Response (includes passage-based, stand-alone, and evidencebased) SCR-Short Constructed Response/Short Answer (includes passage-based) ECR-Extended Constructed Response (includes writing prompts and textdependent analysis) PT- Performance Task (includes portfolio-based) © Pennsylvania Department of Education 40

PM 8 SPECIFICATION TABLES (CONT. ) Cognitive Level • Identified depth of knowledge (Do. K) represented in the targeted content standards. Big Idea/Enduring Understanding Mathematical statements can be justified through deductive and inductive reasoning and proof. Standard ID 2. 4. G. A Write formal proofs (direct proofs, indirect proofs/proofs by contradiction, use of counter -examples, truth tables, etc. ) to validate conjectures or arguments. © Pennsylvania Department of Education Depth of Knowledge Do. K 1 Do. K 2 Do. K 3 Do. K 4 4 2 0 0 4 2 0 0 Total 6 6 41

PM 8 SPECIFICATION TABLES (CONT. ) Reading Level • Defined readability of selected passages, given the developmental readiness of the test-takers. Standard ID CC. 1. 2. 6 A Passage Genre Reading Level(s) 860 L; 880 L Writing Literature Informational Prompt Response Total 0 1 1 1 STANDARD: Determine the central idea of a text and how it is conveyed through particular details; provide a summary of the text distinct from personal opinions or judgments. Reading Level is reported on the Lexile scale. © Pennsylvania Department of Education 42

Items and test forms must meet minimum standards Item specifications: Test blueprints item types number of items response options difficulty levels language load and artwork. content coverage assessment’s difficulty level balance of items targeted distribution of item difficulties rules for sequencing items total number of items timing. © Pennsylvania Department of Education 43

Defined • The degree to which content standards and assessments address the same construct (Webb, 1997). Relevance • Relevance means that alignment characteristics of the assessment supports inferences about the latent construct being measured. Relevance is a key validity characteristic of an assessment. © Pennsylvania Department of Education 44

RELEVANCE § ALIGNMENT © Pennsylvania Department of Education CHARACTERISTICS ARE A SOURCE OF VALIDITY § SCORES REPRESENT CONTENT BEING MEASURED § ALIGNMENT PRACTICES § IMPLEMENTED AS PART OF INSTRUCTIONAL DESIGN § LINK ASSESSMENT TO INSTRUCTION 45

Alignment is a bi-directional Design Task. Targeted Standards © Pennsylvania Department of Education Operational Form Items/ Tasks 46

Items/Tasks • The degree to which the items/tasks address the targeted content standards in terms of • (a) content match, and • (b) cognitive demand/ higher order thinking skills. © Pennsylvania Department of Education 47

1. ALIGNMENT FOCUS (CONT. ) Operational Form • The degree to which the completed assessment reflects (as described in the specification table and blueprint) the • (a) content pattern of emphasis, and • (b) item/task sufficiency. • Also, focuses on the developmental appropriateness and linguistic demand. © Pennsylvania Department of Education 12 48

ALIGNMENT DESIGN PROCESSES 1. SEC & Achieve Models 2. Webb Alignment Model 3. PA Assessment Alignment Model © Pennsylvania Department of Education 49

§ SEC (Surveys of Enacted Curriculum) Model § Survey and Assessment Data Process § Focus on Content Topics-Cognitive Demand § Compares Instructional Data with Assessment Data § Achieve Model § Panel Review Process § Focus on content, performance, challenge, balance, and range § Compares the content and performance challenge of an item /task to the standards and reviews levels of “coverage” © Pennsylvania Department of Education 50

§Webb Model § Panel Review Process § Focus on a reliable set of procedures and criteria for conducting alignment analysis studies § Analyzes alignment of an assessment to content standards based on: § a. Categorical Concurrence § b. Balance of Representation § c. Range of Knowledge § d. Depth of Knowledge § e. Source of Challenge © Pennsylvania Department of Education 51

• Categorical Concurrence • The same categories of the content standards are included in the assessment. • Items could be aligned to more than one content standard. • Balance of Representation • Ensures the distribution of the content standards across the operational form matches the test blueprint. © Pennsylvania Department of Education 52

• Range of Knowledge • The extent of knowledge required to answer parallels the knowledge the standard requires. • Depth of Knowledge • The cognitive demand of the content standard must align to the cognitive demand of the test item. • Source of Challenge • Students give a correct or incorrect response for the wrong reason (e. g. , linguistic demand). © Pennsylvania Department of Education 53

PM 9 1. Content Match (CM) • Items/tasks match a specific content standard based upon the narrative description of the standard and a professional understanding of the knowledge, skill, and/or concept being described. 2. Cognitive Demand/Depth of Knowledge (Do. K) • Items/tasks reflect the cognitive demand/higherorder thinking skill(s) articulated in the standards, with extended performance tasks typically focused on several, integrated content standards. © Pennsylvania Department of Education 54

PM 9 3. Content Pattern (CP) • Item/task distributions represent the emphasis placed on the targeted content standards in terms of “density” and “instructional focus”, while encompassing the range of standards articulated on the test blueprint. 4. Item/Task Sufficiency (ITS) • Item/task distributions consist of sufficient opportunities for test-takers to demonstrate skills, knowledge, and concept mastery at the appropriate developmental range. © Pennsylvania Department of Education 55

Handout 1. 3: Test Specification Workflow Determine the number of items/tasks for each standard given the desired length of the assessment Distribute the items/tasks across the differing Do. K given the cognitive demand in the standard Evaluate the specification tables to ensure items/tasks are "mapped" to at least one standard Evaluate the specification tables to ensure item/task points are sufficient to measure the standard, while reflecting the pattern of emphasis Determine (when applicable) passage/prompt type and readability Adjust specification tables after reveiwing Finalize process and publish tables © Pennsylvania Department of Education Determine the number of items/tasks for each standard by item type Distribute item/task point values among the standards Ensure text complexity/reading levels are developmentally appropriate for the testtaker 56

PROCEDURAL STEPS § Review the targeted content standards identified in Handout 1. 3: Procedural Steps Module 1. 2. § Insert selected Big Idea/Enduring Understanding statements and targeted content standards (Standard ID) into the applicable test specification tables. Big Idea/Enduring Understanding © Pennsylvania Department of Education Standard ID 57

PROCEDURAL STEPS (CONT. ) Handout 1. 3: Procedural Steps § For Table 1 (Content Match & Depth of Knowledge), determine the number of items/tasks across the four cognitive levels for each content standard, and then tally the rows and place the values in the Total column. Big Idea/Enduring Understanding Standard ID 2. 4. G. A Mathematical Write formal proofs (direct statements can be proofs, indirect justified through proofs/proofs by contradiction, use of deductive and inductive reasoning counter-examples, truth tables, etc. ) to validate and proof. conjectures or arguments. © Pennsylvania Department of Education Depth of Knowledge Do. K Tota 1 2 3 4 l 0 4 2 0 6 58

Template 1. 3 Developing Specification Tables SPECIFICATION TABLE 1 EXAMPLE Table 1. Content Match (CM) and Depth of Knowledge (Do. K) Big Idea/Enduring Understanding Standard ID Depth of Knowledge Do. K 1 Do. K 2 Do. K 3 Do. K 4 Total Artists make thoughtful choices in creating works of art. 1. 1 0 4 1 0 5 Artists create works of art employing both conscious and intuitive thought. 1. 2 0 0 0 2 2 Natural resources have influenced the creation of indigenous art forms. 4. 1 3 2 0 0 5 © Pennsylvania Department of Education 59

Handout 1. 3: Procedural Steps PROCEDURAL STEPS (CONT. ) For Table 2 (Content Pattern & Item/Task Sufficiency), determine the number of items/tasks across the four item types for each content standard. In parentheses, record the total points using the point values for each item type (e. g. , SCR weighted at 2 points). Tally the rows and columns and place the weighted point values in the Total column and row. Big Idea/Enduring Understanding Standard ID Mathematical statements can be justified through deductive and inductive reasoning and proof. 2. 4. G. A Write formal proofs (direct proofs, indirect proofs/proofs by contradiction, use of counter-examples, truth tables, etc. ) to validate conjectures or arguments. © Pennsylvania Department of Education Item Types SR SCR ECR PT Total Points 4 2 0 0 20 60

Template 1. 3 Developing Specification Tables SPECIFICATION TABLE 2 EXAMPLE Table 2. Content Pattern (CP) and Item/Task Sufficiency (ITS) Big Idea/Enduring Understanding Artists make thoughtful choices in creating works of art. Artists create works of art employing both conscious and intuitive thought. Natural resources have influenced the creation of indigenous art forms. Grand Totals © Pennsylvania Department of Education Standard ID Item Types SR SCR ECR PT Total Points 1. 1 0 0 5 (4) 0 20 1. 2 0 0 0 2 (35) 70 4. 1 0 5 (2) 0 0 10 20 70 100 61

Handout 1. 3: Procedural Steps For Table 3 (Passage & Writing Prompt Distributions), determine the number of items/tasks across the two-passage genre, and record the reading level of passages identified in the Passage Genre columns. If applicable, identify any writing prompts or tasks requiring a detailed “essay-type” response. Also, record the reading levels of each standard’s respective item types. Lastly, tally the rows and place the values in the Total column. Passage Genre Standard ID CC. 1. 2. 6 A Reading Level(s) 860 L; 880 L © Pennsylvania Department of Education Literature Information 0 1 Writing Prompt Response Total 1 2 62

Template 1. 3 Developing Specification Tables SPECIFICATION TABLE 3 EXAMPLE Table 3. Passage and Writing Prompt Distributions Technical Note: Reading Level is reported on the Lexile scale in the below example (i. e. , Table 3). Passage Genre Writing Reading Level(s) Literature Information Prompt Response Total CC. 1. 2. 6 A 860 L; 880 L 0 1 1 2 CC. 1. 2. 6 B 930 L 0 1 CC. 1. 2. 6 G 950 L; 940 L 1 0 0 1 CC. 1. 2. 6 H 1070 L 0 1 CC. 1. 2. 6 I 900 L; 920 L 0 1 Standard ID © Pennsylvania Department of Education 63

QC CHECKLIST The cognitive demands reflect those articulated in the targeted content standards. Template 1. 3: Procedural Steps q q The targeted content standards match the range within the Big Idea/Enduring Understanding statement. q There is a sufficient sampling of the targeted content standards (e. g. , 5 items/task points). q The item/task point distribution reflects the emphasis found among the targeted content standards. © Pennsylvania Department of Education 64

T 1. 3 QC CHECKLIST (CONT. ) Template 1. 3: Procedural Steps q. The readability approximates the instructional range of the test-takers. q There is a balance between developmental readiness and item/task complexity. q The overall assessment blueprint reflects a coherent, focused, and rigorous measurement approach. © Pennsylvania Department of Education 65

PM 1 • Purpose Statement Module 1. 1 © Pennsylvania Department of Education Module 1. 2 • Targeted Content Standards • Specification Tables Module 1. 3 66

§ Outlined steps needed to create purpose statement § Establish a procedure to select content standards to be measured § Created specification tables that support assessment’s overall blueprint © Pennsylvania Department of Education 67