Modeling Student Benefits from Illustrations and Graphs Michael

Modeling Student Benefits from Illustrations and Graphs Michael Lipschultz Diane Litman Computer Science Department University of Pittsburgh

Introduction Visuals used to convey concepts [Graesser et al, 05; Van. Lehn et al, 05; Rau et al, 09] Illustrations Graphs • Frequent switching [Rau et al. , 09; Rau et al. , 12 a, b] • Concreteness fading [Goldstone & Son, 05; Mc. Neil & Fyfe, 12] • Better transfer [Leelawong & Biswas, 08; Van Heuvelen & Zou, 01] • Relatable [Kozhevnikov et al, 07; Goldstone et al, 12] 4

Adapt to Student • Best representation varies – Gender [Meltzer, 05] – Knowledge [Kozhevnikov et al, 07; Maries et al, 12] – Skills [Dancy & Beichner, 06; Kozhevnikov et al, 07; Velez et al, 05; Conati & Maclaren, 08] • Adapt to student & context – Successful in uncertainty & motivation -> persistence & better learning gains [Aist et al, 02; Forbes-Riley & Litman, 11] • In this work: develop modeling approach – Model better than baseline – Model with interesting properties 6

Outline • Introduction • Data • Modeling – Augmented Stepwise Regression • Quantitative Results • Best Model • Conclusions and Future Work 7

Data • Prior study [Lipschultz & Litman, 13] – Problem-solving [Van. Lehn et al, 05] + post-problem discussion [Katz et al, 11] – Saw either illustrations only or graphs only – Pretest & Post-test – to measure learning gains – 29 subjects: 2, 042 data points (dialogue answers) • Features: – – Student information [Arroyo et al, 00; Chi et al, 11; Lipschultz & Litman, 11] Student skill [Chi et al, 11; Lipschultz & Litman, 11] Domain information [Baschera et al, 11; Lipschultz & Litman, 11; Ward & Litman, 06] Contextual information [Baker et al; Beck, 06; Drummond & Litman, 10; D'Mello & Graesser, 10] 8

Modeling • Don’t know best visual – Know: visual seen & amount learned • Infer best visual from learning gains • Regression to predict learning – Syntactic constraints for useful model – Goal: good model that is useful 9

Modeling with Stepwise Regression Algorithm 1. Stepwise Linear Regression – Convert features to binary 2. Identify Problematic Rules – – Mutually Exclusive Non-Adaptive 3. Handle Problematic Rules – Remove Lesser Rule in Pair 4. Relearn Model – Regular Regression 5. Rank by absolute value of coefficient 10

Modeling with Stepwise Regression 1. Stepwise Linear Regression • Postscore = terms + prescore • terms: βi*representation*(tutoring context) • βi*representation*partition*rule – βi*Illustration*(Pre. Score=High)*(Response. Time=Fast) – For high pretesters, when Response. Time=Fast, show illustrations – Binary features • Learns coefficients (βi’s) • Keeps only predictive terms 11

Modeling with Stepwise Regression Example Starting Set of Terms (72) Illustration*(Pre. Score=High)*(Walk. Thru. Pct=Low) Illustration*(Pre. Score=High)*(Walk. Thru. Pct=High) Illustration*(Pre. Score=High)*(RQPct. Correct=Low) Illustration*(Pre. Score=High)*(RQPct. Correct=High) Illustration*(Pre. Score=High)*(Pct. Situation. Correct=Low) Illustration*(Pre. Score=High)*(Pct. Situation. Correct=High) Illustration*(Pre. Score=High)*(Pct. Session. Correct=Low) Illustration*(Pre. Score=High)*(Pct. Session. Correct=High) Illustration*(Pre. Score=High)*(Pct. Thru. Situation=Early) Illustration*(Pre. Score=High)*(Pct. Thru. Situation=Late) Illustration*(Pre. Score=High)*(Pct. Thru. Session=Early) Illustration*(Pre. Score=High)*(Pct. Thru. Session=Late) Illustration*(Pre. Score=High)*(KCUsage=State) Illustration*(Pre. Score=High)*(KCUsage=Apply) Illustration*(Pre. Score=High)*(Item. Diff=Easy) Illustration*(Pre. Score=High)*(Item. Diff=Hard) Illustration*(Pre. Score=High)*(Response. Time=Fast) Illustration*(Pre. Score=High)*(Response. Time=Slow) Graph*(Pre. Score=High)*(Walk. Thru. Pct=Low) Graph*(Pre. Score=High)*(Walk. Thru. Pct=High) Graph*(Pre. Score=High)*(RQPct. Correct=Low) Graph*(Pre. Score=High)*(RQPct. Correct=High) Graph*(Pre. Score=High)*(Pct. Situation. Correct=Low) Graph*(Pre. Score=High)*(Pct. Situation. Correct=High) Graph*(Pre. Score=High)*(Pct. Session. Correct=Low) Graph*(Pre. Score=High)*(Pct. Session. Correct=High) Graph*(Pre. Score=High)*(Pct. Thru. Situation=Early) Graph*(Pre. Score=High)*(Pct. Thru. Situation=Late) Graph*(Pre. Score=High)*(Pct. Thru. Session=Early) Graph*(Pre. Score=High)*(Pct. Thru. Session=Late) Graph*(Pre. Score=High)*(KCUsage=State) Graph*(Pre. Score=High)*(KCUsage=Apply) Graph*(Pre. Score=High)*(Item. Diff=Easy) Graph*(Pre. Score=High)*(Item. Diff=Hard) Graph*(Pre. Score=High)*(Response. Time=Fast) Graph*(Pre. Score=High)*(Response. Time=Slow) Illustration*(Pre. Score=Low)*(Walk. Thru. Pct=Low) Illustration*(Pre. Score=Low)*(Walk. Thru. Pct=High) Illustration*(Pre. Score=Low)*(RQPct. Correct=Low) Illustration*(Pre. Score=Low)*(RQPct. Correct=High) Illustration*(Pre. Score=Low)*(Pct. Situation. Correct=Low) Illustration*(Pre. Score=Low)*(Pct. Situation. Correct=High) Illustration*(Pre. Score=Low)*(Pct. Session. Correct=Low) Illustration*(Pre. Score=Low)*(Pct. Session. Correct=High) Illustration*(Pre. Score=Low)*(Pct. Thru. Situation=Early) Illustration*(Pre. Score=Low)*(Pct. Thru. Situation=Late) Illustration*(Pre. Score=Low)*(Pct. Thru. Session=Early) Illustration*(Pre. Score=Low)*(Pct. Thru. Session=Late) Illustration*(Pre. Score=Low)*(KCUsage=State) Illustration*(Pre. Score=Low)*(KCUsage=Apply) Illustration*(Pre. Score=Low)*(Item. Diff=Easy) Illustration*(Pre. Score=Low)*(Item. Diff=Hard) Illustration*(Pre. Score=Low)*(Response. Time=Fast) Illustration*(Pre. Score=Low)*(Response. Time=Slow) Graph*(Pre. Score=Low)*(Walk. Thru. Pct=Low) Graph*(Pre. Score=Low)*(Walk. Thru. Pct=High) Graph*(Pre. Score=Low)*(RQPct. Correct=Low) Graph*(Pre. Score=Low)*(RQPct. Correct=High) Graph*(Pre. Score=Low)*(Pct. Situation. Correct=Low) Graph*(Pre. Score=Low)*(Pct. Situation. Correct=High) Graph*(Pre. Score=Low)*(Pct. Session. Correct=Low) Graph*(Pre. Score=Low)*(Pct. Session. Correct=High) Graph*(Pre. Score=Low)*(Pct. Thru. Situation=Early) Graph*(Pre. Score=Low)*(Pct. Thru. Situation=Late) Graph*(Pre. Score=Low)*(Pct. Thru. Session=Early) Graph*(Pre. Score=Low)*(Pct. Thru. Session=Late) Graph*(Pre. Score=Low)*(KCUsage=State) Graph*(Pre. Score=Low)*(KCUsage=Apply) Graph*(Pre. Score=Low)*(Item. Diff=Easy) Graph*(Pre. Score=Low)*(Item. Diff=Hard) Graph*(Pre. Score=Low)*(Response. Time=Fast) Graph*(Pre. Score=Low)*(Response. Time=Slow) Terms from Step 1 (~10) 0. 1423*Illustration*(Pre. Score=High)*(Response. Time=Fast) 0. 0342*Graph*(Pre. Score=High)*(Response. Time=Fast) -0. 5820*Illustration*(Pre. Score=Low)*(Pct. Thru. Tutoring=High) 0. 2432*Graph*(Pre. Score=Low)*(Pct. Thru. Tutoring=High) 0. 3895*Illustration*(Pre. Score=High)*(Pct. Session. Correct=High) 0. 6921*Illustration*(Pre. Score=High)*(Pct. Session. Correct=Low) -0. 3382*Illustration*(Pre. Score=High)*(KCUsage=Apply) 0. 5243*Illustration*(Pre. Score=High)*(Item. Diff=Easy) 0. 1322*Graph*(Pre. Score=Low)*(Pct. Thru. Situation=High) -0. 8225*Graph*(Pre. Score=Low)*(Pct. Walk. Thru=Low) 12

Modeling with Stepwise Regression Example Starting Set of Terms (72) Illustration*(Pre. Score=High)*(Walk. Thru. Pct=Low) Illustration*(Pre. Score=High)*(Walk. Thru. Pct=High) Illustration*(Pre. Score=High)*(RQPct. Correct=Low) Illustration*(Pre. Score=High)*(RQPct. Correct=High) Illustration*(Pre. Score=High)*(Pct. Situation. Correct=Low) Illustration*(Pre. Score=High)*(Pct. Situation. Correct=High) Illustration*(Pre. Score=High)*(Pct. Session. Correct=Low) Illustration*(Pre. Score=High)*(Pct. Session. Correct=High) Illustration*(Pre. Score=High)*(Pct. Thru. Situation=Early) Illustration*(Pre. Score=High)*(Pct. Thru. Situation=Late) Illustration*(Pre. Score=High)*(Pct. Thru. Session=Early) Illustration*(Pre. Score=High)*(Pct. Thru. Session=Late) Illustration*(Pre. Score=High)*(KCUsage=State) Illustration*(Pre. Score=High)*(KCUsage=Apply) Illustration*(Pre. Score=High)*(Item. Diff=Easy) Illustration*(Pre. Score=High)*(Item. Diff=Hard) Illustration*(Pre. Score=High)*(Response. Time=Fast) Illustration*(Pre. Score=High)*(Response. Time=Slow) Graph*(Pre. Score=High)*(Walk. Thru. Pct=Low) Graph*(Pre. Score=High)*(Walk. Thru. Pct=High) Graph*(Pre. Score=High)*(RQPct. Correct=Low) Graph*(Pre. Score=High)*(RQPct. Correct=High) Graph*(Pre. Score=High)*(Pct. Situation. Correct=Low) Graph*(Pre. Score=High)*(Pct. Situation. Correct=High) Graph*(Pre. Score=High)*(Pct. Session. Correct=Low) Graph*(Pre. Score=High)*(Pct. Session. Correct=High) Graph*(Pre. Score=High)*(Pct. Thru. Situation=Early) Graph*(Pre. Score=High)*(Pct. Thru. Situation=Late) Graph*(Pre. Score=High)*(Pct. Thru. Session=Early) Graph*(Pre. Score=High)*(Pct. Thru. Session=Late) Graph*(Pre. Score=High)*(KCUsage=State) Graph*(Pre. Score=High)*(KCUsage=Apply) Graph*(Pre. Score=High)*(Item. Diff=Easy) Graph*(Pre. Score=High)*(Item. Diff=Hard) Graph*(Pre. Score=High)*(Response. Time=Fast) Graph*(Pre. Score=High)*(Response. Time=Slow) Illustration*(Pre. Score=Low)*(Walk. Thru. Pct=Low) Illustration*(Pre. Score=Low)*(Walk. Thru. Pct=High) Illustration*(Pre. Score=Low)*(RQPct. Correct=Low) Illustration*(Pre. Score=Low)*(RQPct. Correct=High) Illustration*(Pre. Score=Low)*(Pct. Situation. Correct=Low) Illustration*(Pre. Score=Low)*(Pct. Situation. Correct=High) Illustration*(Pre. Score=Low)*(Pct. Session. Correct=Low) Illustration*(Pre. Score=Low)*(Pct. Session. Correct=High) Illustration*(Pre. Score=Low)*(Pct. Thru. Situation=Early) Illustration*(Pre. Score=Low)*(Pct. Thru. Situation=Late) Illustration*(Pre. Score=Low)*(Pct. Thru. Session=Early) Illustration*(Pre. Score=Low)*(Pct. Thru. Session=Late) Illustration*(Pre. Score=Low)*(KCUsage=State) Illustration*(Pre. Score=Low)*(KCUsage=Apply) Illustration*(Pre. Score=Low)*(Item. Diff=Easy) Illustration*(Pre. Score=Low)*(Item. Diff=Hard) Illustration*(Pre. Score=Low)*(Response. Time=Fast) Illustration*(Pre. Score=Low)*(Response. Time=Slow) Graph*(Pre. Score=Low)*(Walk. Thru. Pct=Low) Graph*(Pre. Score=Low)*(Walk. Thru. Pct=High) Graph*(Pre. Score=Low)*(RQPct. Correct=Low) Graph*(Pre. Score=Low)*(RQPct. Correct=High) Graph*(Pre. Score=Low)*(Pct. Situation. Correct=Low) Graph*(Pre. Score=Low)*(Pct. Situation. Correct=High) Graph*(Pre. Score=Low)*(Pct. Session. Correct=Low) Graph*(Pre. Score=Low)*(Pct. Session. Correct=High) Graph*(Pre. Score=Low)*(Pct. Thru. Situation=Early) Graph*(Pre. Score=Low)*(Pct. Thru. Situation=Late) Graph*(Pre. Score=Low)*(Pct. Thru. Session=Early) Graph*(Pre. Score=Low)*(Pct. Thru. Session=Late) Graph*(Pre. Score=Low)*(KCUsage=State) Graph*(Pre. Score=Low)*(KCUsage=Apply) Graph*(Pre. Score=Low)*(Item. Diff=Easy) Graph*(Pre. Score=Low)*(Item. Diff=Hard) Graph*(Pre. Score=Low)*(Response. Time=Fast) Graph*(Pre. Score=Low)*(Response. Time=Slow) Terms from Step 1 (~10) 0. 1230*Illustration*(Pre. Score=High)*(Response. Time=Fast) 0. 7890*Graph*(Pre. Score=High)*(Response. Time=Fast) -0. 5820*Illustration*(Pre. Score=Low)*(Pct. Thru. Tutoring=Later) 0. 2432*Graph*(Pre. Score=Low)*(Pct. Thru. Tutoring=Later) 0. 4560*Graph*(Pre. Score=Low)*(Response. Time=Fast) 0. 1230*Graph*(Pre. Score=Low)*(Response. Time=Slow) -0. 3382*Illustration*(Pre. Score=High)*(KCUsage=Apply) 0. 5243*Illustration*(Pre. Score=High)*(Item. Diff=Easy) 0. 1322*Graph*(Pre. Score=Low)*(Pct. Thru. Situation=High) -0. 8225*Graph*(Pre. Score=Low)*(Pct. Walk. Thru=Low) But some of these rules don’t make sense… 13

Modeling with Stepwise Regression Algorithm 1. Stepwise Linear Regression – Convert features to binary 2. Identify Problematic Rules – – Mutually Exclusive Non-Adaptive 3. Handle Problematic Rules – Remove Lesser Rule in Pair 4. Relearn Model – Regular Regression 5. Rank by absolute value of coefficient 14

Modeling with Stepwise Regression 2. Identify Problematic Rules • Mutually Exclusive Rules 0. 123*Illustration*(Pre. Score=High)*(Response. Time=Fast) 0. 789*Graph *(Pre. Score=High)*(Response. Time=Fast) Different Representation Same context • Non-Adaptive Rules 0. 456*Graph*(Pre. Score=Low)*(Response. Time=Fast) 0. 123*Graph*(Pre. Score=Low)*(Response. Time=Slow) Same Representation Opposite Contexts 15

Modeling with Stepwise Regression Algorithm 1. Stepwise Linear Regression – Convert features to binary 2. Identify Problematic Rules – – Mutually Exclusive Non-Adaptive 3. Handle Problematic Rules – Remove Lesser Rule in Pair 4. Relearn Model – Regular Regression 5. Rank by absolute value of coefficient 16

Modeling with Stepwise Regression 3. Handling Problematic Rules • Mutually Exclusive Rules 0. 123*Illustration*(Pre. Score=High)*(Response. Time=Fast) 0. 789*Graph *(Pre. Score=High)*(Response. Time=Fast) • Non-Adaptive Rules 0. 456*Graph*(Pre. Score=Low)*(Response. Time=Fast) 0. 123*Graph*(Pre. Score=Low)*(Response. Time=Slow) 17

Modeling with Stepwise Regression Algorithm 1. Stepwise Linear Regression – Convert features to binary 2. Identify Problematic Rules – – Mutually Exclusive Non-Adaptive 3. Handle Problematic Rules – Remove Lesser Rule in Pair 4. Relearn Model – Regular Regression 5. Rank by absolute value of coefficient 18

Modeling the Best Representation: Experiment • Model Types – Baseline: just show one kind (illustration) – 1 Factor: 1 Tutoring Context factor in term – 2 Factors: Partition data along 1 variable • High pretesters vs. Low pretesters – 3 Factors: Partition along 2 variables • 10 -fold cross validation Partition variables: 1. Gender 2. Spatial. Reason 3. Pre. Score 4. Pct. Thru. Problem 5. Pct. Thru. Session 19

Modeling the Best Representation: Results 0, 6 Higher is Better 0, 5 Significantly better than baseline 0, 3 0, 2 0, 1 2 Factors n hr u. S e t. T Pc u. P ro bl ss io em n Pc t. T hr ia l. R ea so nd er Sp at hr u. S e t. T Pc Ge n ss io em bl ro u. P Pc t. T hr Pr e. S co r e n Sp at ia l. R ea so nd er Ge r ct o Fa 1 se l in e 0 Ba Adj. R^2 0, 4 3 Factors (Pre. Score and …) 20

Modeling the Best Representation: Results Significantly better than baseline 0, 6 Higher is Better 0, 5 Significantly better than baseline 0, 3 0, 2 0, 1 2 Factors n hr u. S e t. T Pc u. P ro bl ss io em n Pc t. T hr ia l. R ea so nd er Sp at hr u. S e t. T Pc Ge n ss io em bl ro u. P Pc t. T hr Pr e. S co r e n Sp at ia l. R ea so nd er Ge r ct o Fa 1 se l in e 0 Ba Adj. R^2 0, 4 3 Factors (Pre. Score and …) 22

Interpreting the Model Pre. Score*Gender Females (n=8) High Pretesters 1. If few correct answers in walk throughs or reflections, show graphs 2. If many correct answers in session or problem, show illustrations 3. If early in problem or session, show graphs (n=9) Low Pretesters 1. If many correct answers in session, show graphs 2. If early in session, show illustrations 3. If many correct answers in problem, show illustrations 4. If early in problem, show illustrations 5. If few correct answers in reflections, show illustrations Males (n=3) 1. If few correct in reflections, show illustrations 2. If many correct in session, show illustrations 3. If few correct in walk throughs, show illustrations (n=9) 1. If few correct answers in walk throughs or reflections, show illustrations 2. If many correct answers in session, show illustrations 3. If early in session or problem, show graphs 4. If many correct answers in problem, show illustrations 23

Conclusion • Developed modeling technique – Best visual unknown – Handles “problematic” rules • 5 models outperform baseline – Possible to model benefit • Partitioning Useful: Pre. Score & Gender 24

Future Work • Empirical Evaluation of Model – Is adapting visual representation helpful? • Develop method of selecting partition features – Partial correlation with postscore (covars=existing partitions)? 25

Thank you Thanks to IES for supporting this research.