Visual Summarization of Leg Sim Argumentation n Background

Visual Summarization of Leg. Sim Argumentation n Background Leg. Sim (Legislature Simulation): Students practice argumentation skills using an asynchronous discussion forum. A+ By analyzing the surface features of each discussion topic, we can see a huge space of improvement. F! Only one bill has more than 70 postings, most of the bills have less than 5 postings. B 0 One direction of improvements is to enhance teacher intervention. Teachers must evaluate students’ progress and choose where to give feedback Our toolkit focus on two problems: 1. What kind of data interests teachers? 2. How can we provide this data in a good form? Excel file of 480 rows… INFORMATION OVELOAD! n What type of Interventions do teachers use? n What are the criteria for evaluating students? Assumptions that survive critique are more likely to be sound, so teachers can use a Socratic/devil’s advocate approach to encourage students to justify or elaborate their positions. Intervention Scenario #1: comments on a piece of legislation consist of a majority consensus and a poorly argued dissenting minority. -The teacher can probe the minority to elaborate by asking questions. Intervention Scenario #2: there is only consensus - The teacher can challenge the group’s assumptions by playing devil’s advocate. n How can our tool help? Intervention Scenario #1: probe poorly argued dissent Many studies on argumentation have been based on Tolumin’s framework of argumentation theory. Toulmin suggests that the statements in argumentation have different functions and can be classified into six categories: claims, data, warrants, backings, qualifiers, and rebuttals. A summary that includes: - Sentiment ※ - Argumentation quality ※ - Number of posts - Number of legislation discussions participated - Participation over time - Argumentation quality over time ※ Sentiment and Argumentation quality are manually annotated. Intervention Scenario #2: challenge consensus Evaluation Scenario #3: weekly class participation Balanced! Skewed! Weighted Comparison! Set +/signs and weighted sort. Evaluation Scenario #4: student at the end of the semester PIE Chart of argumentation of each student. Weighted Comparison! Give proper weight to arguments. Evaluation Scenario #5: student’s progress over time Weekly count or Timeline shows progress over time Weekly count! You can check overall argumentation of students. Evaluation Scenario #6: student argumentation comparison Weighted Comparison! Low weight for no argumentation, high for good argumentation! n System Evaluation To test the effectiveness of these visual summaries we want to investigate whether teachers using the tool are: - able to give faster, more effective feedback, which results in greater improvement of argumentation skills among their students? We could investigate these questions with a between-subjects design where the control group of teachers has the raw data and the experimental group uses the visual summaries. - able to quickly evaluate students’ performance in a way that is consistent with time-consuming manual evaluations. Yue (Jenny) Cui () Moonyoung Kang (moonyoun@cs. cmu. edu) Cari Sisson (csisso@cs. cmu. edu)