Semantic Map Assessment Project Overview for Mc REL

Semantic Map Assessment Project Overview for Mc. REL Dr. Roy B. Clariana Penn State, Great Valley RClariana@psu. edu http: //www. personal. psu. edu/rbc 4 12/18/02

Project intent n n n Semantic maps (SMs) are considered to be valid and reliable measures of science content knowledge (Ruiz-Primo, Schultz, Li, Shavelson). SMs are described as “authentic”, teachers and students use semantic maps in the science classroom as a tool to represent their understanding of that content (Novak). The intent of this project is to establish an automatic computer system that scores semantic maps by comparing students’ maps to an “expert” map.

Student-developed SMs The student recalls important terms from a science lesson and drags related terms closer together and unrelated terms further apart to form clusters or categories, and then draws lines between directly related terms.

Scoring Semantic Maps n n To date, SMs are scored by teachers or trained raters using scoring rubrics (Lomask) Although this marking approach is time consuming and fairly subjective, map scores usually correlate well with more traditional measures of science content knowledge (multiple choice, fill-in-the blank, and essays) No one has tried automatic assessment yet To automatically mark SMs, the graph is converted into an array (matrix)

SMs can be represented by two kinds of arrays n n Link array – if a line (link) is used to connect two terms, a “ 1” is placed in the corresponding array cell, and a “ 0” is used in array cells to show that there is not a link between terms. Association array – represents the strength of relationship between pairs of terms as distances scaled from 0 (highly related) to 1 (unrelated). This array can be converted into a link array of implicit clusters equivalent to a Path. Finder neighborhood.

Semantic map w/ link array Most studies use link information, usually called “propositions”.

Semantic map w/ distance array 0. 17 Some use association information, usually called “neighborhoods”.

SM automatic scoring Pilot n n n A group of 12 practicing teachers enrolled in CI 400 at PSU completed SMs to describe the structure and function of the heart and then wrote essays on this topic from their maps. SM “distance” data were obtained using Concept Mapper software (available at: http: /www. personal. psu. edu/rbc 4/cm 1. htm) that I developed last March for this purpose. SM “link” array data were colleted by manually entering 1’s for linked terms and 0’s for unlinked terms into an Excel spreadsheet.

. . . Pilot n n n Computer-derived LSA Essay scores were obtained by pasting the participant’s essays into the Web-based form available at: http: //www. personal. psu. edu/rbc 4/frame. htm Manually determined SM and essay scores were determined by 5 pairs of judges Variables and correlation results are shown on the next slides

Internal variables n n Links (L) – arithmetic sum of all links in the map Link agreement with an expert (L/Exp) – the arithmetic sum of the links that exactly match the expert map Associations (A) – first convert the proposition closeness to a link array (. 13 as cut-off), then the arithmetic sum of all links in the map Association agreement with an expert (A/Exp) – convert proposition closeness to links (. 13 as cutoff), then the arithmetic sum of the links that exactly match the expert map

Criterion Variables n n H U M A N n LSA Essay – Essay score established by Latent Semantic Analysis software, using Landau and Kintsch's Web site Semantic Map (Map) – Map scores established by averaging the scores from 6 pairs of judges using Lomask et al. , 1992 rubric for assessing semantic maps Essay – Overall essay scores established by averaging the scores from 5 pairs of judges (using a rubric)

Correlation matrix Computer Human Significant correlations shown in bold.

Maps and Essays n n n The strongest correlation, r = 0. 98, was shown for “Map” and “Essay”, both human derived metrics. Since the semantic maps were used as an aid in writing the essays, it seems reasonable that the two should be highly related. This high correlation (based on human raters) provides criterion-related evidence of semantic map validity. If confirmed in later studies, science teachers may reasonably use semantic map scores for student assessment.

A/Exp n n The automatically derived variable association agreement with an expert (A/Exp) was significantly correlated with the human derived “Map” (. 87) and “Essay” (. 82) scores, as well as with the computer derived “LSA essay” score (. 82). Thus the first pilot suggests that association agreement with an expert is a promising automatic measure of science content knowledge (like Path. Finder C scores).

L/Exp n n The automatically derived variable link agreement with an expert (L/Exp) was significantly correlated with the human derived “Map” score (. 87), as well as with the computer derived “LSA essay” score (. 83). Thus, link agreement with an expert is also a promising automatic measure of science content knowledge (most extant studies involving SMs use some variant of L/Exp).

Links VS. associations n n In a multiple regression analysis of these two automatic variables to human derived essay scores, association agreement with an expert accounted for 67% of the variance, link agreement with an expert accounted for an additional 6%, so the two together accounted for 73% of the variance in human rater essay scores (multiple r = 0. 85). So association (neighborhood) and link (propositions) information each account for some unique components of the essays.

Next steps Field trials – Confirm pilot results; examine criterion-related validity for SMs to MC, CR, essay, and other test forms; determine cutscore approach for association arrays; find the best algorithm for score generation and automate it; improve the software, especially automating link capture; much more…

Present follow-up investigation n n 60 undergraduate students in intro Ed. Psyc completed an instructional text on the heart, developed concept maps of the content on paper, then completed a verbatim- and an application-level multiple-choice posttest on the lesson content. I am using the Concept Mapper software to establish the distance arrays, and the same manual procedure for link arrays. Example concept map

Student TN-10 Concept Map

. . follow-up investigation n n So far, data is collected, I’m establishing the arrays now, Then I will determine the cut-score for transforming the distances to link arrays, Then, I’ll calculate correlations with the MC tests, and finally Submit a manuscript early 2003, and use these two as a basis for obtaining a grant for a larger field-trial and for software development

What is the potential of automatic SM assessment? n n n Can automatic semantic map marking support higher-level learning? Higher-level assessment? How could teachers/districts use an automatic semantic map marking system? What value would automatic semantic map marking have for “test” companies?

Next steps. . . A Roadmap to Professional Practice, Norm 5: - Choose teaching and assessment strategies that help students develop understandings of math and science. - Choose n teaching and assessment strategies that are compatible to one another. - Use multiple methods and tools and systematically gathering data about students’ scientific NCLB: and mathematical reasoning skills and their understandings about math and science concepts. -- State standards must be developed for science by assessment the 2005 Incorporate ongoing, embedded, diagnostic, prescriptive, and summative -06 school year. into instruction. --Beginning Provide opportunities students to demonstrate their knowledge, understandings, in thefor 2005 -06 school year, tests must be and skills in a variety of ways. administered every year in grades 3 through in math and - Use the results of assessments at different levels and in a variety 8 of ways to improve reading. teaching and learning. --Beginning Communicate student to the school student and his/herscience parent(s) or guardian. in theprogress 2007 -08 year, achievement - Review assessment tasks for the use of stereotypes, offensive or irrelevant language, or must alsothat bereflect tested. assumptions the perspectives or experiences of a particular group. - Recognize that the purpose of an assessment may be different in different situations. Is there a fit here at Mc. REL?

Next steps. . . n Is there a fit here at Mc. REL? n If so, what are our next steps?