Exploring the Relationship Between Novice Programmer Confusion and

Exploring the Relationship Between Novice Programmer Confusion and Achievement By: Diane Marie Lee Ma. Mercedes Rodrigo Ryan Baker Jessica Sugay Andrei Coronel

Affective States and Achievement • Recent studies have illustrated the relationships between affective states and achievement • Negative affective states have negative impact on student’s achievement (Craig et al, 2006; Rodrigo, 2009; Lagud, 2010)

Confusion • Double-edged/ Dual Nature (D’Mello 2009) ▫ Harmful ▫ Helpful

Goal • Discovery-with-models approach to finding the relationship between novice programmer confusion and achievement

Data Collection • 149 students enrolled in CS 21 a – Introduction to Computing I • Four lab sessions • Blue. J IDE • Blue. J Plug-in (Jadud and Henriksen, 2009)

Data Collection Compilation logs = all submissions made to the compiler • Compilation logs include ▫ ▫ ▫ Computer number Timestamp Code Error message (if any) And many more!

Data Collection • Total of 340 student-lab sessions • Total of 13, 528 compilation logs collected

Data Labeling • Sorted the compilations by student and by Java class name • Grouped the compilations into clips ▫ Clips = 8 compilations ▫ Total: 2, 386 clips • Raters were asked to label a sample of 664 clips

Data Labeling • Used low-fidelity text replays ▫ Maintains good inter-rater reliability and efficient in aiding coders to label student disengagement (Baker et al. 2006) • Labels ▫ Confused ▫ Not Confused ▫ Bad Clip • Cohen’s Kappa between raters: 0. 77

Data Labeling • Filter out “bad clips” • Remove clips where raters disagreed on the label • Left with 418 clips for model construction

Model Construction • Used Rapid. Miner version 5. 1 • Used J 48 Decision Trees • Features were mined from the clips

Model Construction • Feature set used: ▫ ▫ ▫ Average time between compilations Maximum time between compilations Average time between compilations w/ errors Maximum time between compilations w/ errors Number of pairs consecutive compilations ending w/ the same error

Kappa: 0. 86

Data Relabeling • Model was coded as a Java program • Had the program relabel all the 2, 386 clips • Generated three sets of confused-not confused sequences • Correlated the percentage of the sequences of each student to their midterm exam scores

Results Not Confused- Not-Confused-Not Confused Confused . 064 . 139 . 144 -. 229 (0. 539) (0. 180) (0. 163) (0. 026) Relationship with midterm

Results NNN NNC NCN NCC CNN CNC CCN CCC -. 015. 014 . 062 -. 046. 233. 163. 052 -. 337 Relationship with Midterm (. 901) (. 909) (. 610) (. 704) (. 05) (. 174) (. 665) (. 004)

Conclusion • Prolonged confusion has a negative impact on student’s performance • Resolved confusion has a positive impact on student’s performance • A certain amount of confusion is needed for learning

On-going Work • Support the incorporation of tools for automatic detection of confusion in computer science learning environments • Redoing the sampling and clipping method

Thank you Questions?