Feedback visualization in a grammarbased elearning system for
Feedback visualization in a grammarbased e-learning system for German: A preliminary user evaluation with the COMPASS system Karin Harbusch and Annette Hausdörfer University of Koblenz-Landau, Department of Computer Science {harbusch, hausdrfer}@uni-koblenz. de 1
Overview 1. Motivation 2. COMPASS 3. Feedback in COMPASS 1. Feedback presentation 2. User evaluation 4. Conclusions and future work 2
1. Motivation One of the advantages of using natural-language processing in an ICALL (Intelligent Computer-Assisted Language-Learning)-system is that the learner can freely produce an utterance. The natural-language processing component evaluates the result and automatically generates personalized, reliable and immediate learner feedback as an important prerequisite for effective learning in ICALL. Especially, the feedback can be tuned to the learner’s current performance level and his/her understanding of the explicitly presented grammar rules.
2. COMPASS Via drag-and-drop dialogues, COMPASS (the acronyom stands for COMbinatorial and Paraphrastic Assembly of Sentence Structure) allows users to freely select wordforms from a lexicon and to combine them into phrases and sentences. The core component of COMPASS is a natural-language generator that, for every new word the user wishes to attach to the current string (as an extension of this string or as a replacement of a substring), checks whether this tentative attachment is grammatically well-formed or not.
COMPASS II On this basis, the system can compute and display online the grammatical structure of input strings in the form of syntactic trees, and identify and diagnose input errors. It confronts the learners only with errors they could have avoided, given their current proficiency level and their current understanding of the grammar rules. The proficiency levels are implemented as underspecifications of grammar rules.
2. 4 a All three strands Different proficiency levelsin Diehl et. Al (2000) for German as L 2 Table by Diehl et al. (2000; native speakers of French at various high school levels); also verified by Ballestracci (2005; native speakers of Italian at university level). Here we focus only on strand 3 – case marking. 6
Low proficiency level in strand 3 (case) Demo also available via http: //youtu. be/ptsv 2 d. GGv_4
3. Feedback in COMPASS 3. 1 Feedback presentation Widely practiced in e-learning systems are animated tutors that interact with the user in a socially engaging manner. Adopting this format, we have chosen an owl—portrayed as intelligent and wise in Aesop’s fables—as a character capable to attract and motivate children as well as young adults.
In concise mode, the owl has green eyes, sitting in a green box at the upper right corner as long as the user is building grammatically correct structures. In case of an error the color changes to red. In verbose mode, if COMPASS spots an error, the owl shows up at the word or phrase the learner has just attached erroneously, and displays a box with information about the error type. Additionally, the user can switch off feedback if desired.
Concise feedback the white dog fondle Peter and Anja ‘Peter and Anja fondle 3 rd, plural the white dog’
Verbose feedback the white dog fondles Peter and Anja ‘* Peter and Anja fondles 3 rd, singular the white dog’
3. 2 User evaluation In order to find out whether the feedback was sufficiently salient and informative to attract the attention of the users, we performed an eye-tracking experiment. We also asked the participants to express in a questionnaire their subjective impression of the usefulness of the linguistic illustrations and especially, the feedback.
User evaluation II Measures to run a normalized study: No free sentence production with COMPASS but presentation of prefabricated sentences. Study in with native speakers of German so that all learners have the same proficiency level. Let test subjects exercise relatively rare and errorprone agreement cases (see study by Bock & Miller, 1991). They will need feedback same as L 2 learners to delivered his/her grammaticality judgment.
User evaluation III Study controlled for: Pure sentences to be judged. Only syntactic trees after the test subjects had played around with COMPASS to become familiar with the trees. Concise feedback along with syntactic trees. Verbose feedback along with syntactic trees.
Heat map example for no feedback sowohl Peter als auch Hans sah die große Gefahr As_well Peter as also Hans saw 3 rd, singular the big danger 15
Heat map example for concise mode 16
Heat map example for verbose mode
Results of the study As indicated by the “hot spots” in the heat maps, the participants did pay attention to the feedback agent. The percentage of correct answers increases from 52% in the control condition (no tree, no feedback) to 67% in the experimental condition with syntactic trees (more precisely, even in the experimental subcondition without feedback where the owl did not give away the correct answer). This suggests the feedback was not only perceived but also yielded a learning effect. The questionnaire data showed that several participants (5 out of 20) found the trees baroque and confusing rather than helpful. These participants did 18 not profit from the feedback.
4. Conclusions and future work We have presented an e-learning system for German as L 1 and L 2 based on a natural-language generator. Basically, the feedback production takes into consideration the users proficiency level. Moreover the feedback can be presented by a tutor in a concise or verbose mode if desired by the user. The results of our preliminary user evaluation experiment suggest that our way of presenting feedback via an animated pedagogical agent is promising.
Conclusions and future work II However, the questionnaire revealed considerable dissatisfaction with the level of detail of the linguistic information provided by COMPASS. The participants’ complaints ranged from too many colors to too much linguistic sophistication. We take these results as recommendations to continue with feedback presentation in the form of animated tutors and with syntactic trees, but also to simplify the tree format considerably.
For more information about the COMPASS system see: https: //userpages. unikoblenz. de/~harbusch/COMPASS/ ex. html 22
Feedback design 23
- Slides: 23