Effective Helpseeking Behavior Among Students Using an Intelligent

Effective Help-seeking Behavior Among Students Using an Intelligent Tutoring System for Math: a Cross. Cultural Comparison Jose Carlo A. Soriano

Break down �Effective Help-seeking Behavior Among Students �Using an Intelligent Tutoring System for Math: �A Cross-Cultural Comparison

Intelligent Tutoring Systems �Seeks to simulate the effectiveness of a good personal human tutor �Individual tutoring is more effective than classroom instruction by 2 standard deviations (Bloom, 1984) � Knows what specific skills the student is having trouble with � Able to offer appropriate help at an appropriate time �Objective: help the student learn �ITS are better by 1 standard deviation (Koedinger et al, 1998; Corbett et al. , 2001)

Intelligent Tutoring Systems

Help-Seeking in ITS �The kind of help, and how the help is offered, affects learning (Aleven et al, 2003) �Students who use High-level help most frequently have the least learning(Matthews and Mitrovic, 2008) �Help-seeking is a Meta-cognitive skill �Meta-cognition is “cognition about cognition” � One’s knowledge of the processes in play � One’s active control of it during learning

Help-seeking Behavior 1. Both the EU and UNESCO declared: � developing metacognitive skills, or ‘teaching students how to learn’ should be among the highest educational priorities (Louizidu and Kotselini, 2007) 2. Help-seeking behavior is an achievement-related behavior (Karabenick and Knapp, 1991) 3. Higher-achieving students were more likely to ask for help when encountering personal difficulties (Taplin et al, 2001)

Effective Help-Seeking �Student is more likely to learn when: �Student seeks for help when encountering personal difficulties � Student knows what kind of help is needed such that student can work effectively on his/her own � Student knows how to ask for help �Student is not dependent on help �Student spends time understanding help given

Help-seeking in ITS �However, students generally do not know when they need help (Aleven and Koedinger, 2000) �Students “game the system” (Baker et al) �Meta-cognitive tutors have been developed by Aleven et al �“Scooter the tutor” developed by Baker et al �To teach students “how to learn”

The Problem �Is “effective help-seeking” the same across cultures? �Very few cross-cultural comparisons � Comparing ITS use between USA and Latin American students(Ogan et al, in press) � Comparing Disengaged behavior between USA and Filipino students (Rodrigo et al, 2010) �Implications on Meta-cognitive tutors

Method �Find out if effective help-seeking behavior is the transferrable across cultures, or are significantly different �Might encourage more cross-cultural comparisons �Implications on future efforts on meta-cognitive tutoring

Scatterplot tutor

Data �Costa Rica �Mexico �USA �Philippines

Feature Engineering 1. Helpavoidance 2. Nothelpavoidance 3. Helpnonuse 4. Unneededhelp 5. Bugmsg. Longpause 6. Bugmsg. Shortpause 7. Hintmsg. Longpause 8. Hintmsg. Shortpause

Feature Engineering 9. Hintmsg. Longpause. Correct 10. Hintmsg. Shortpause. Correct 11. Nothelpavoidance. Shortpause 12. Nothelpavoidance. Longpause 13. Unneededhelp. Shortpause 14. Unneededhelp. Longpause 15. Shortpause. Hintmsg 16. Longpause. Hintmsg 17. Firstattempt. Hintmsg

Feature Optimization �Most features require a threshold, either p-know or a time threshold �Brute-force grid search: �For p-know thresholds, grid-size is 0. 05 �For pause thresholds, grid-size is 0. 5 seconds �Single-parameter linear regression for each threshold for each feature in grid

Feature Selection �Cross-validated r is used as the goodness criterion �Correlation between the predicted learning values and the actual learning value �The threshold with the best cross-validated r becomes the threshold for each feature �As an additional control against over-fitting, features whose best threshold had negative cross-validated r is dropped from model creation

Model Creation and Evaluation �Models will be created using Forward Selection �Models will be evaluated by applying each country’s model to each country’s data set �A model will be created after combining the four data sets

Brute-Force Grid Search Feature CR Helpavoidance Helpnonuse Unneededhelp Bugmsg. Longpause Hintmsg. Shortpause Hintmsg. Longpause. Correct Nothelpavoidance. Longpause Unneededhelp. Shortpause Unneededhelp. Longpause MX Helpnonuse Bugmsg. Shortpause Hintmsg. Shortpause. Correct Nothelpavoidance. Longpause ALL Helpavoidance Nothelpavoidance Helpnonuse Bugmsg. Shortpause Hintmsg. Longpause. Correct Hintmsg. Shortpause. Correct Nothelpavoidance. Shortpause Nothelpavoidance. Longpause Unneededhelp. Shortpause Unneededhelp. Longpause cut-off r 0. 15 1 25. 5 47. 5 0. 5 41. 5 45. 5 13 0 0. 081 0. 012 0. 006 0. 054 0. 017 0. 294 0. 284 0. 019 0. 008 1 2. 5 0. 5 58. 5 0. 201 0. 044 0. 025 0. 018 0. 05 0. 4 0 2. 5 1 3 20 1 35 1 0. 023 0. 024 0. 094 0. 071 0. 022 0. 062 0. 027 0. 052 0. 04 0. 062 Feature PH Nothelpavoidance Helpnonuse Nothelpavoidance. Shortpause Nothelpavoidance. Longpause Unneededhelp. Shortpause US Helpavoidance Helpnonuse Unneededhelp Bugmsg. Longpause Bugmsg. Shortpause Hintmsg. Longpause Hintmsg. Shortpause Hintmsg. Longpause. Correct Hintmsg. Shortpause. Correct Nothelpavoidance. Shortpause Nothelpavoidance. Longpause Unneededhelp. Shortpause Unneededhelp. Longpause Shortpause. Hintmsg Longpause. Hintmsg cut-off R 0. 4 0. 95 1 0 0. 5 0. 087 0. 043 0. 108 0. 075 0. 061 0. 25 1 1 57 0. 5 6. 5 1 4 0. 5 0 12 0 32. 5 37. 5 0. 122 0. 014 0. 117 0. 131 0. 026 0. 003 0. 02 0. 039 0. 073 0. 265 0. 096 0. 203 0. 123 0. 026 0. 04

Analysis �Firstattempt. Hintmsg – only feature that was not able to pass in any country �Different nature of skills � Some can be hard at the very start, some can be easily understood from the start � Most skills may require a large number of actions �Nothelpavoidance. Longpause – had positive cross-validated correlation in all five data sets �In contrast to theory, this feature has negative directionality for most countries � This might be because students who exhibit the behavior simply do not understand the skill

Forward Selection Country CR MX PH US ALL Learning = 0. 132 * Helpavoidance(0. 15) + 7. 385 * Hintmsg. Longpause(47. 5) - 9. 096 * Hintmsg. Longpause. Correct(41. 5) - 21. 847 * Nothelpavoidance. Longpause(0. 25, 45. 5) + 53. 010 - 0. 147 * Helpnonuse(1) - 0. 754 * Bugmsg. Shortpause(2. 5) + 1. 187 * Hintmsg. Shortpause. Correct(0. 5) + 40. 652 0. 021 * Helpnonuse (0. 95) - 0. 763 * Nothelpavoidance. Shortpause(0. 4, 1) + 32. 423 - 1. 021 * Nothelpavoidance(0. 25) - 2. 870 * Bugmsg. Longpause(57) - 6. 680 * Nothelpavoidance. Shortpause(0. 25, 19. 5) + 5. 605 *Longpause. Hintmsg(37. 5) + 12. 086 0. 036 * Helpavoidance(0. 05) + 0. 082 * Helpnonuse(0) - 0. 491 * Bugmsg. Shortpause(2. 5) - 46. 861 Cross-validated r 0. 462 0. 229 0. 126 0. 350 0. 153

Analysis �Helpavoidance – negative directionality for CR and PH �reinforcces Aleven et al’s findings that avoiding help is negatively correlated to learning (Aleven et al. , 2006) �Butmsg. Shortpause and Nothelpavoidance. Shortpause also has negative directionality for MX and US, and PH and US. �Possibly effect of not spending enough time to understand bug or hint message provided

Cross-Cultural Evaluation Rows are models, columns are data sets applied to Country CR MX PH US ALL CR 0. 534 -0. 285 0. 051 0. 151 0. 08 MX 0. 04 0. 392 -0. 086 -0. 009 0. 088 PH 0. 004 -0. 174 0. 203 0. 146 0. 038 US -0. 085 -0. 164 0. 228 0. 476 0. 057 All -0. 032 0. 165 0. 047 0. 142 0. 215

Analysis �Eighteen out of 25 model applications produced positive correlation between model’s predicted learning and the actual learning �Negative correlation means model did worse than chance at predicting learning �LOOCV r values are very high compared to r when applied to other countries �Reinforces hypothesis that help-seeking might not transfer across countries

Analysis �MX and US – performance on each other’s data sets are low (-0. 009 and -0. 164) �This means that our model of effective help-seeking is not effective when applied to the other country �Reinforces findings in (Ogan et al, in press) which compares differences in behavior of USA and Mexico students � Collaborative nature of students from Mexico may be the reason why help-seeking is different � The help they ask from the tutor will only be help that they do not get from other students

Analysis �CR and MX – did not perform well on each other’s data sets (-0. 285, 0. 04) �Though collaborative tendencies might be common between Costa Rica and Mexico, help-seeking behavior with the ITS may differ �US and PH – performed very well on each other’s data set (0. 146 and 0. 228) �Reinforces findings in (San Pedro, 2011) wherein a carelessness detector is generalizable between the two countries �In contrast to (Rodrigo, 2010) which shows disengaged behavior is different between the two countries � But Help-seeking and Disengaged behavior are two different sets of behaviors

Conclusion �Results did not expose that effective help-seeking as a whole is very culture-specific (18 out of 25 applications returned positive r) �However, it is not apparent that effective help-seeking is transferrable across countries �Big difference between LOOCV r and cross-cultural evaluations

Conclusion and Recommendations �There are pairs of countries wherein effective help-seeking fail to generalize to each other’s data set. �Meaning effective help-seeking from one country may not necessarily be effective in another �Single effective help-seeking models used by metacognitive tutors may be effective in one culture but not in others �Future meta-cognitive tutors might have to use a more generalizable model �May have to switch models, depending on the culture where the ITS is used