Principled Assessment Designs for Inquiry PADI Geneva Haertel

Principled Assessment Designs for Inquiry (PADI) Geneva Haertel, SRI International Robert Mislevy, University of Maryland Partners: Mark Wilson & Cathleen Kennedy, University of California-Berkeley Nancy Butler Songer, University of Michigan, Bio. KIDS Project Kathy Long, University of California-Berkeley, Lawrence Hall of Science, FOSS Project Larry Hamel, Codeguild Inc.

PADI Goals • Conceptual frameworks for: – Design Patterns for assessing inquiry – Task-design – Assessment delivery • Object model for task design • Software tools to aid the design process • “Scoring engine” to sort out evidence • Worked-through applications – Globe, FOSS, Bio. KIDS

Method Student Model(s) Evidence Models Stat model Task Models Evidence Rules Feature s 1. xxxxx 2. xxxxx 3. xxxxx • What complex of knowledge, skills, or other attributes should be assessed, presumably because they are tied to explicit or implicit objectives of instruction or are otherwise valued by society? • What behaviors or performances should reveal those constructs? • What tasks or situations should elicit those behaviors? (Messick, 1992)

OBJECT MODELING AND SYSTEMS DEVELOPMENT Strand Accomplishments • Drafted use-cases and initial UML models • Prototyped web application to support example-based modeling • Entered examples and iterated models for design patterns, templates, student models, etc. • Expanded measurement model to support MRCML scoring engine • Created Grade. Book sample application • Abstracted EMO framework for CSCW 2004 paper and NSF Science of Design proposal • Implementing wizards for assessment design

Object Models Grade. Book Design System Wizards

MODELING Strand Purpose Designs to assess students’ creation, use, and understanding of scientific models Problem While there has been rapid growth in the development of modeling software and tools for instructional purposes, few of these modeling tools are accompanied by assessments that provide high-quality evidence of student understanding Approach • Prepare literature review of modeling tools and how student use of modeling tools and activities have been assessed in recent years • Create design patterns for creation and use of models • Organize AERA Interactive Symposium Accomplishments • AERA Interactive symposium, Modeling, Inquiry, and Assessment: Developing a Principled Assessment Argument for Computer-Based Modeling Tools • Modeling design patterns in the PADI design system • Draft review of modeling research and implementation in education and recommendations for principled assessment designs

Examples of Modeling Tools STELLA model of market growth Model. It interface for air quality model Net. Logo model for Connected Chemistry Thinker. Tools model

FOSS Strand Purpose To use the PADI Design System to develop new science inquiry assessments with the following features and capabilities: • Accommodates alternative student models to measure student learning in different ways from the same assessment activities. • Provides immediate feedback to students and teachers when assessments are online. • Uses assessment feedback to help students monitor and direct their learning activities.

The FOSS Online Assessment The Online Assessment Integrates Instructional and Assessment Purposes This item can be used in “Practice Mode” to generate tutorial hints, or in “Quick Check” mode to generate a proficiency estimate for the student at this time point.

NAEP/TIMSS Strand Purpose To understand the surface and deep characteristics of NAEP and TIMSS items through the lens of the PADI system Problem What are the characteristic and variable features of NAEP and TIMSS items that measure inquiry? Method Reverse engineer NAEP and TIMSS items into PADI templates

NAEP Floating Pencil Template Student Model Variables Task Model Variables - Content: Density / relative density - Level of scaffolding (inquiry, content) - Content: Errors of measurement - Format of student response (multiple choice, short constructed response, extended constructed response) - Inquiry: Tools and techniques used to gather data - Inquiry: Evidence, content knowledge, and logical argument used to predict or explain - Level of student explanation (none, choice of response, studentgenerated) - Data representation (drawing, table, graph)

SCORING ENGINE Strand Purpose To develop a stand-alone module that can be called by an assessment application to compute student proficiencies for: • • • Multivariate student models Multivariate observations Application-managed model specifications: − Student model parameters − Item parameters − Scoring parameters − Estimation parameters − Proficiency type (posterior distribution, maximum likelihood estimate, plausible values)

The Scoring Engine Servlet An Assessment Application Calls the Scoring Engine via HTTP to Estimate Student Proficiencies Student Responses Database PADI Design System Database Scoring Engine via HTTP or local Extracts measurement model details from the PADI Design System. via HTTP Assessment Application Requests student proficiency estimates from the Scoring Engine.

EPISTEMIC FORMS AND GAMES Strand Epistemic Forms & Games (Collins & Ferguson 1993) • Epistemic forms - target structures that guide inquiry • Epistemic games - procedures for “filling out” a form PADI as Complex of Epistemic Forms & Games • Types of inquiries guided by PADI – “What are potential constituent elements of assessment tasks that can provide evidence about (such and such) skill? ” – “Given this conceptual assessment argument, what operational assessment can elicit, identify, and synthesize evidence in a way that instantiates the conceptual argument? ” • If TOMS = PADI’s epistemic forms, then what are PADI’s games and in what ways are they epistemic?

Method of Investigating PADI’s Epistemic Nature What exactly are PADI’s games? • Some rules and constraints are “built in” • Investigate and catalog constraints and strategies that have been discovered by PADI’s first users through interviews How do PADI’s design principles make its games epistemic? • Evidence Centered Design • Object Oriented Design

GLOBE Strand Purpose • To develop GLOBE template(s) with the PADI system • To use GLOBE template as an example for PADI Wizard Why GLOBE Is a Good Use-Case for PADI • GLOBE assessments address multiple aspects of science inquiry and content knowledge • GLOBE generic template and samples of GLOBE assessments created by SRI facilitate development of the GLOBE template for PADI Global Learning and Observations to Benefit the Environment (GLOBE) is a worldwide hands-on, primary and secondary school-based education and science program (www. globe. gov).

GLOBE Generic Template [Insert GLOBE data or graphs here] Present problem requiring use of GLOBE data archives. Planning Investigations: Ask students to pose relevant questions. Analyzing and Interpreting GLOBE Data: Ask questions about data in the table. Conducting Investigations: Assure data quality. Analyzing and Interpreting Data: Ask questions about the relationships between variables. Analyzing and Interpreting Data: Ask to generate new representations to analyze trends. Analyzing and Interpreting Data: Ask for interpretation of specific features of graphs. Planning Investigations: Ask to set up another problem using new data set. Communicate Findings: Ask to summarize and report findings.

GLOBE PADI Template Activity 1: Posing Research Questions Student Model Variables Evidence Model Measurement Model Materials & Presentation GLOBE Data Ability to Plan GLOBE Investigations Observable Variable: Score on Posing Research Questions Ability to Understand GLOBE Concept Work Product Task Model Student. Developed Research Question Based on GLOBE Data Problem Situation Description Prompt to Pose Questions Sample Question Specification of Required Components Task Model Variables GLOBE Activity Scaffolding Number of Questions Requested Sample Question TMV Specification of Required Components TMV

Bio. KIDS Strand Purpose • Longitudinal studies focusing on 5 th-7 th graders’ critical thinking in science • Audience is diverse, high poverty urban children • Students analyze data, look for patterns, build explanations of phenomena • Students learn basic science concepts and complex reasoning in concert

Content-Inquiry Matrix for “Formulating Scientific Explanations from Evidence” complexity of science content required to perform task amount of inquiry scaffold provided

Student Performance on Posttest Inquiry Reasoning By Complexity Type