Assessment of Student Learning in Integrated Science Technology

Assessment of Student Learning in Integrated Science, Technology, Engineering, and Mathematics (i. STEM)

Assessment of Student Learning in i. STEM Goals l Features of cognitively-principled assessment design l Design principles for the assessment of i. STEM learning outcomes l Limitations of current i. STEM assessments l Challenges for assessing i. STEM learning outcomes l Promising innovative approaches l Needed research

i. STEM Definition Two or more STEM disciplines, including technology or engineering

i. STEM Models l Integrated and balanced l Asymmetrical l Hierarchical/layered

i. STEM Learning Environment Models l Tacit or explicit l Learning outcomes l Pedagogy l Transfer near or far

i. STEM Models l Implementation features l Duration/dosage-within experience, multiple experiences l Settings-formal/informal l Staff expertise l Resources

Cognitively-Principled Assessment Design Learning science research (E. g. , How People Learn l Measurement theory and research on measuring learning (E. g. Knowing What Students Know) l Assessment argument linking claims of learning, to evidence of learning, to tasks eliciting the evidence l

Evidence-Centered Design Student Model What complex of knowledge, skills, or other attributes should be assessed? Evidence Model Task Model What behaviors or performances should reveal the relevant knowledge and skills described in the student model? What tasks or situations should elicit the behaviors or performances described in the evidence model? Messick, 1993 Mislevy, Almond, & Lucas, 2004

i. STEM Constructs (Student Model-Assessment Targets) From national frameworks and standards for science, mathematics, engineering, technology l Cross-cutting concepts-E. g. , Systems and System Models (Next Generation Science Standards) l Cross-cutting practices-E. g. , problem solving, communication, collaboration (NAEP Framework for Technology and Engineering Literacy) l

Task Models l Integrated applications to natural and designed world l Applied, significant, recurring problems in the natural and designed world l Scenario-based tasks building across a problem solving/inquiry/design sequence l Within a discipline and/or in integrated scenarios

Evidence Model l What evidence is collected Explicit responses l Logged processes l l How the evidence is evaluated and summarized Scoring l Rubrics l l How the evidence is reported for intended purposes and users

Limitations of i. STEM Assessments l Little readily available information/documentation of measuresdescriptive or technical quality l Lack of attention to alignment of outcome measures to i. STEM learning outcomes and standards l No descriptions of coverage/balance l Outcome measures tend to emphasize content, declarative knowledge l Little attention to application of practices

Limitations of i. STEM Assessments l Practices –not measured well by static, conventional formats l Few measures during (processes, formative) vs. at end (summative) l Little measurement of collaboration and communication l Lack of deliberate design to measure for transfer of cross -cutting concepts and practices l Little attention to establishing/documenting technical quality

Challenges for Designing i. STEM Assessments Specification of desired learning outcomes l Need for focus, coherence of knowledge and processes and whether situated in a domain and/or in integrated problems Distinguish if the learning is attributed to initial teaching vs. application of previously learned l E. g. , Are all targets at grade level, vs. grade level and prerequisites? Coverage-What is the balance of assessment targets? What is the balance of proximal (embedded, tailored) vs. distal (state test)?

Challenges for Designing i. STEM Assessments (cont’d) Need to tailor assessment design to assessment purposeintended use of the data l Formative/summative Embedded to monitor (use of feedback and coaching) and adjust vs. l Culminating to report proficiency status l Duration, scope, time l l More extended, spread over multiple classes/periods Embedded vs. external Documentation of measures l Descriptions, technical quality (validity of interpretation, reliability)

Promising New Assessment Designs for i. STEM Technology Affordances l Alignments l Access to resources and expertise l Network with collaborators, experts l Collections l Delivery l Entry of rubrics, ratings, work in progress, final artifacts l Scoring-auto and online training and scoring, moderated rating sessions l Reporting-customized to users

Promising New Assessment Designs for i. STEM Technology Supports within Hands-On Projects l Blended model of technology tools, e. g. CAD, prototypes, test data with engineering projects l Entry of rubric ratings, calibrated training sessions l Annotated postings of designs, prototypes, tests l Embedded tasks to test knowledge and skills during projects l Electronic project notebooks l Electronic portfolios l Juried exhibitions posted, streamed, archived

Promising New Assessment Designs for i. STEM Interactive Task Design Features Dynamic presentations of spatial, causal, temporal phenomena in a system l Multiple overlapping representations Interactivity l Supports iterative, active inquiry and design Multiple response formats l Reduce reliance on text Rapid, customized interaction, feedback, reporting

Research on Learning in Science Simulations l Facilitate formation of organized mental models of system components, interactions, and emergent behaviors l Facilitate transfer l Facilitate use of systematic problem solving & inquiry l Situate in authentic, significant, recurring problems in the natural and designed world l Highly engaging

NAEP 2014 Framework and Specifications for Technology and Engineering Literacy l Sim. Scientists: Force and Motion-Fire Rescue l PISA: Reactor l http: //www. nagb. org/publications/frameworks/te ch 2014 -framework/ch_toc/index. html

Sim. Scientists Test Effects of Pollution on Cells

Sim. Scientists Test Effects of Calories on Activity Level

Research Needs l Analysis of extant assessments-large scale and classroom, formative and summative l Analyses of assessment opportunities within i. STEM models l Review of promising exemplars l Formulation and testing of different purposes, designs, and evidence collection strategies l Pilot studies of assessment design models for established and new genre of i. STEM learning environments

Assessment of Student Learning in i. STEM equellm@wested. org http: //simscientists. org