Assessing K12 Students Learning Progression of Carbon Cycling

Assessing K-12 Students’ Learning Progression of Carbon Cycling Using Different Types of Items 2010 NARST Presentation Written by: Jing Chen and Charles W. Anderson (Michigan State University), Jinnie Choi, Yong Sang Lee, Karen Draney (University of California, Berkeley) Culturally relevant ecology, learning progressions and environmental literacy Long Term Ecological Research Math Science Partnership April 2010 Disclaimer: This research is supported by a grant from the National Science Foundation: Targeted Partnership: Culturally relevant ecology, learning progressions and environmental literacy (NSF-0832173). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

ASSESSING K-12 STUDENTS’ LEARNING PROGRESSION OF CARBON CYCLING USING DIFFERENT TYPES OF ITEMS Jing Chen Charles W. Anderson Michigan State University Jinnie Choi Yong Sang Lee Karen Draney University of California, Berkeley

• analyze how well ordered multiple choice (OMC), multiple True or False (MTF) items differentiate students among achievement levels by comparing students’ responses to these items to their responses to the same problems set in open-ended (OE) formats • inform the development of OMC and MTF items to measure students’ achievement levels more reliably OMC and MTF items • OMC item-- each of the possible answer choices is linked to a particular developmental level of student understanding for the construct being measured. (Briggs, Alonzo, Schwab, & Wilson, 2006). • MTF item-- a set of true or false questions asks students to judge True or False. PURPOSE OF THIS STUDY PURPOSE

• How well can our mixed types of items assess students’ performances? RESEARCH QUESTIONS • How well can OMC and MTF items diagnose students’ achievement levels?

LEVELS Explaining Progress variable LEVELS Naming Progress Variable 4 Linking processes with matter and energy as constraints 4 Scientific statements 3 Changes of Molecules and Energy Forms with Unsuccessful Constraints 3 Scientific words of organic molecules, energy forms, and chemical change 2 Force-dynamic accounts with hidden mechanisms 2. 5 Macroscopic force-dynamic accounts 1. 5 1 • Related studies 2 1 Easier scientific words with mixed meanings Hidden mechanism words Easier hidden mechanism words Words about actors, enablers, and results • Test scores based on OMC items have greater validity than test scores based on TMC items, without sacrificing reliability. There a weak to moderate positive correlation between students’ scores on OMC items and their scores on traditional MC items (Briggs, Alonzo, Schwab, & Wilson, 2006). • Compare to OE items, OMC items appear to provide more precise diagnoses of students’ learning progression levels and to be more valid, eliciting students’ conceptions more similarly to cognitive interviews (Alonzo & Steedle, 2008) • Inconsistency in students’ responses to items in different formats, students performed differently on multiple-choice and short-answer items (Steedle, 2006) THEORETICAL FRAMEWORK • Carbon cycle learning progression framework

• students in grade 4 to 12 • 1044 assessment (454 pre, 550 post, 181 elementary, 377 middle, 446 high) • collected during 2008 -2009 Assessments • 48 items in total • include 8 OMC, 8 MTF items and their paired OE items in this study (32 in total) • matter energy transformation in 5 macroscopic events and large scale events. RESEARCH METHOD Participants

• OMC items Ø cross-tabulation Ø Spearman rank-order correlation between OMC and OE levels • MTF items Ø cross-tabulation Ø point-biserial correlation between MTF and OE responses • For all items Ø IRT partial credit model; IRT analysis Ø qualitative evaluation by a group of researchers RESEARCH METHOD Data Analysis

A loaf of bread was left uncovered for two weeks. Three different kinds of mold grew on it. Assuming that the bread did not dry out, which of the following is a reasonable prediction of the weight of the bread and mold together? A) The mass has increased, because the mold has grown. (Level 1) B) The mass remains the same as the mold converts bread into biomass. (level 2) C) The mass decreases as the growing mold converts bread into energy. (level 2) D) The mass decreases as the mold converts bread into biomass and gases. (level 3) OE response levels OMC response levels 1 (A) 2 (B, C) 3 (D) 0 1 3 4 1 42 15 2 2 1 24 32 3 0 0 12 RESULTS OF OMC ITEMS BREAD ITEM

OMC ITEM PAIRED OE SAMPLE SIZE CORRELATI ON WTLOS_N 306 . 272** WTLOS_E 306 . 342** BODYTE_N 138 . 313** BODYTE_E 138 . 476** BREAD_N 137 . 581** BREAD_E 137 . 651** MATCH_N 283 . 217** MATCH_E 283 . 258** LEAVES_N 306 . 300** LEAVES_E 306 . 320** STOREN_E 287 . 360** TROP_E 269 . 411** LIGHT_E 289 . 219** TOTAL_N . 654** TOTAL_E . 755** BODYTE BREAD MATCH LEAVES ** correlation is significant at the. 01 level (2 -tailed) • There are weak to moderate positive correlations between students’ OMC levels and their OE levels. • The correlation between overall OMC scores and overall OE naming scores is. 654, and the correlation between overall OMC scores and overall OE explaining scores. 755. RESULTS OF OMC ITEMS Correlation between OMC scores and paired OE item scores

OMC (5 items administrated at high school form C) OE (5 paired items administrated at high school form C) Cronbach’s alpha (N of cases =138) . 441 . 704 Expected reliability for a longer test (using spearman-brown prophecy formula) 0. 703 (a test with 15 OMC items) RESULTS OF OMC ITEMS Reliability of OMC and OE items

• Compare the item difficulties for 8 OMC and 8 paired OE items (both naming and explaining), OMC items are easier than OE naming, which is easier than OE explaining. Test information comparison (one test includes 8 OMC items ****** only, the other included 8 paired OE items only) • The test that contains 8 OE items will get more information at the higher ability range compare to a test that includes 8 paired OMC items. RESULTS OF OMC ITEMS Item difficulties for OMC items, paired OE items (naming and explaining)

• Students’ choices to “sunlight”, “air”, and “plant create their own energy” do not have significant correlation to their levels in the paired OE item. • Students who circled N for water (water 0), N for nutrients (nutrients 0) are more likely to be at higher levels compare to students who circled Y for these 2 indicators. RESULTS OF MTF ITEMS PLANT ENERGY: Which of the following are sources of energy for plants? Circle yes or no for each of the following: a). Water Yes / No b). Light Yes / No c). Air Yes / No d). Nutrients in soil Yes / No e). They make their own energy Yes / No Explain what you think is energy for plants

MTF ITEMS A B C D E F G TREEG ROWTH SUN WATER AIR SOIL HUMAN ENERGY SUN WATER NUTRIENTS FOOD EXER CO 2 N N Y N N N SUN WATER NUTRIENTS OWN ENERGY PLANT ENERGY AIR N HUMAN GROW RUN . N SUN WATER AIR FOOD WATER AIR ENERGY SLEEP SOIL PLANT CO 2 INSECT ENERGY HEAT WATER+ GAS WASTE OTHER MATERIALS Y CARBON ANIMAL WINTER Y STONE WARM RUN CLOTH FOOD Y RESULTS OF MTF ITEMS (INDICATORS)

RESULTS OF ALL ITEMS • The classical item discrimination index shows the correlation between students’ scores on this item and their total scores. There is a moderate to strong correlation between students’ score on most of the items to their total score. • The red dot items are the OMC items. They generally show lower correlation between students’ scores on the OMC item and their total scores compare to the OE items.

• All the items are fitting well to the model (for each item, the weighted MNSQ is within the range from. 77 to 1. 33). • This graph indicate that our assessment items can fairly differentiate students with different abilities among levels. RESULTS OF ALL ITEMS Average person location for each score

• OMC items are generally easier than OE items, OMC items do not measure students at high ability levels as precise as OE items do • OMC items have weak to moderate correlation with OE items that measure the same content • lower reliability compare to OE items. • The choices in many MTF items did not associate with students’ levels in OE format • The entire assessment differentiate students among levels well Implications 1. Though it’s hard to write OMC options at higher achievement levels without using “science-y” terminologies, we still need to develop options for students at high level. 2. The design of MTF items need to be informed by more research. So the options should be better indicators that actually differentiate students among levels. 3. The number of OMC items in a test should be relatively large in order to get equivalent amount of test information (test reliability) compare to a test with OE items or with mixed types of items. CONCLUSIONS AND IMPLICATIONS Conclusions

Thank you for your attention!

a b c d 1 8 5 11 17 2 3 1 65 10 3 0 0 10 1 4 0 0 1 0

• OMC, MTF items are informed by educational research, options in OMC items represent typical students’ understandings, from the least to most scientifically sophisticated accounts, found in science education research. • OMC and MTF items are more diagnostic compare to traditional MC and T or F items, can be used to provide teachers quick feedbacks. • however, OMC and MTF items do not give students opportunity to create their own responses • given the advantages and limitations of the OMC and MTF items, the analysis of how well the OMC and MTF items can diagnose students’ achievement levels can direct the future use of OMC and MTF items. WHY STUDY THIS? • OMC and MTF items can be more easily used in large-scale assessments compare to OE items