Overview of the PADI Design System Robert Mislevy

Overview of the PADI Design System Robert Mislevy, University of Maryland Geneva Haertel, SRI International Robert Murphy, SRI International AERA April 2005

The Problem § Creating good science inquiry tasks is hard. § How to do it both soundly and efficiently? § What’s the link with science standards? § What’s the psychological perspective represented? § How to design assessments for different content areas & ages of students? § How do you score it? § How to sort out confounded evidence? § How to leverage the use of technology? 2

The PADI Solution (1) § Conceptual frameworks and representational (“epistemic”) forms… § Design patterns for assessing inquiry § Task-design templates § Assessment delivery system § Object model for task design § Software tools to aid the design process § Optional scoring and calibration engine to sort out evidence 3

The PADI Solution (2) § Libraries of exemplars… § Design Patterns for assessing inquiry § Task templates and pointers to tasks § Worked-through applications § FOSS, Bio. KIDS; GLOBE and others § Evaluation studies § FOSS, Bio. KIDS 4

Project Objectives To produce a conceptual framework and collection of design tools and resources for supporting the design of assessments of science inquiry. § Formulate a design framework for science inquiry that is extensible, shareable, and practical § Lay out the framework in a open-system object model § Develop supporting software to create and work with design patterns and task templates § Develop an optional scoring and calibration engines for the sorting out of evidence § Provide an initial set of high quality exemplars to highlight the elements of a coherent assessment argument. § Test out the utility of the framework, object model, and tools with applications (GLOBE, FOSS, Bio. Kids etc. ) 5

Scope of PADI project § Not an authoring system. Conceptual framework and object model provide infrastructure around which tailored authoring systems can be built. § Not a delivery system. Shared conceptions, representations, and object definitions enhance the efficiency of delivery system design by providing a common infrastructure (i. e. delivery system should be consistent with principles of ECD). § Does not set out to define inquiry based on a single perspective. Structures for expressing assessment arguments (Design Patterns) and instantiating them in tasks (Task Templates) independent of the psychological perspective. 6

Main Components of PADI Design Models (Conceptual) Evidencecentered design "Scoring engine" Run. Time PADI Science Inquiry Delivery System Architecture Calibration Other examples Science Standards PADI Applications Libraries Cog Psych Literature Design Tools (Specific incarnation) Exemplars Three-tiered architecture Use Cases Object Model Design Software Templates Design Patterns Globe Bio. KIDS Conceptual models Application Design FOSS Conceptual models Application Design Implementation Evaluation study 7

Evidence Centered Design § The assessment problem: To draw inferences about student proficiencies based on limited observations. § Evidence-Centered Design: Defines elements of the assessment argument and relationships between them. 1. What does it mean to know and do inquiry? 2. What constitutes evidence of knowing? 3. How can that evidence be elicited from students? 4. What are the appropriate statistical techniques for making valid student inferences about what students know from what students do? PADI is implementing a special case of ECD to the study of science inquiry. 8 §

Evidence-Centered Design (1) Design Models (Conceptual) "Scoring engine" Run. Time Evidencecentered design Delivery System Architecture Calibration 9

Assessment Design Models (2) Design Models (Conceptual) "Scoring engine" Run. Time Evidencecentered design Delivery System Architecture Calibration Student Model(s) Evidence Models Stat model Task Models Evidence Rules Feature s 1. xxxxx 2. xxxxx 3. xxxxx 10

Assessment Design Models (3) Design Models (Conceptual) "Scoring engine" Run. Time Evidencecentered design The Four-Process Delivery System Architecture Calibration Administrator Summary Feedback Presentation Process Activity. Selection Process Examinee Task/ Evidence Composite Library Evidence Accumulation Process (Summary Scoring) Evidence Identification Process Task Level Feedback (Response Processing) 11

Assessment Design Models (3 a) Design Models (Conceptual) "Scoring engine" Run. Time Evidencecentered design The Four-Process Delivery System Architecture Calibration Administrator Presentation Process Activity. Selection Process "Scoring engine" Run. Time Summary. Calibration Feedback Examinee Task/ Evidence Composite Library Evidence Accumulation Process (Summary Scoring) Evidence Identification Process (Response Processing) Task Level Feedback 12

Paradigms for Assessing Science Inquiry Other examples PADI Applications Science Standards Cog Psych Literature 13

Software Support & Architecture for Designing Assessment Objects Design Tools (Specific incarnation) Three-tiered architecture Use Cases Object Model Design Software 14

PADI Applications Globe Bio. KIDS Conceptual models Application Design FOSS Conceptual models Application Design Implementation 15

Evaluation Studies Bio. KIDS FOSS Conceptual models Application Design Implementation Evaluation study 16

Libraries of Assessment Objects Evidencecentered design Science Inquiry PADI Applications Libraries Design Tools (Specific incarnation) Exemplars Templates Design Patterns 17

The PADI Solution § PADI focus is on. . . § Design framework (Design patterns, templates, delivery system) § Assessing inquiry in science § PADI focus is not on. . . § Tasks, per se (although will be examples, through FOSS, Bio. KIDS, other exemplars) § Authoring systems § Delivery systems § (exceptions: scoring and calibration engine, Grade. Book)