Intelligent Learning Object Guide i LOG A Framework

Intelligent Learning Object Guide (i. LOG) A Framework for Automatic Empirically-Based Metadata Generation S. A. Rileya, L. D. Millera, L. -K. Soha, A. Samala, and G. Nugentb a. University of Nebraska—Lincoln: Department of Computer Science and Engineering

Overview �Introduction: �What is a Learning Object (LO)? �Why do we need LO metadata? �Metadata problems and i. LOG solution �i. LOG Framework �LO Wrapper �Meta. Gen (metadata generator) �Data Logging �Data Extraction �Data Analysis (feature selection, rule mining, statistics) �Conclusions and Future Work 2 Intelligent Learning Object Guide

Introduction: What is a learning object? �Self-contained learning content LO Metadata �Ideally, each covers a single topic �Serve as building blocks for lessons, modules, or courses �Can be reused in multiple 3 instructional contexts Intelligent Learning Object Guide Learning Object i. LOG Learning Object structure: • Content: tutorial, exercises, assessment • Metadata

Introduction: What is a learning object? �The i. LOG LOs contain a tutorial, exercises, and assessment �Each covers a ‘bite-sized’ introductory computer science topic. Tutorial Exercises Assessment 4 Intelligent Learning Object Guide

Introduction: Why do we need LO metadata? � Repositories for LOs are being constructed � However, there are barriers to effective utilization of these repositories: LO Metadata Learning Object LO Metadata �Learning Context: not all LOs, 5 even on the same topic, are suitable for use in a given learning context �Uncertainty: we cannot be certain what will happen with real-world usage �Search and Retrieval: current Intelligent Learning Object Guide metadata is not machine-readable, Learning Object LO Metadata Learning Object LO Repository

Introduction: Why do we need LO metadata? Learning Context: � Students are highly varied: �Pre-existing knowledge, cultural background, motivation, selfefficacy, etc. Uncertainty: � Cannot be certain what will happen when actual students use an actual LO: �Good for students with low self-efficacy �Inherent gender bias �Bad for students without Calculus experience Search and Retrieval: � Metadata is fundamental to an instructor’s ability to use LOs: �Guide in the LO selection process �Help prevent the feeling that e-learning is ‘too complicated’ 6 Intelligent Learning Object Guide

So … 7 …how do we enable instructors to locate appropriate LOs for their students? ? ? Intelligent Learning Object Guide

Introduction: Metadata problems and i. LOG solution • Current metadata standards are insufficient (Freisen, 2008) • There ample opportunities for making elearning more “intelligent” (Brooks et al. , 2006) Current Metadata Ideal Metadata � Manual generation by 8 course designer � Based only on designer intuition � Metadata format inconsistent / incomplete � Human- but not machine- readable Intelligent Learning Object Guide � Automated generation � Based on empirical usage � Consistent metadata suitable for guiding LO selection � Both human and machine-readable

Introduction: Metadata problems and i. LOG solution The i. LOG solution is: � General: i. LOG is based on established learning standards �We use the SCORM learning object standard, the IEEE LOM metadata standard, and the Blackboard LMS �Furthermore, it is compatible with existing LOs and does not require modification to the LOs (noninvasive) �The i. LOG framework can also be applied to other standards � Automatic: i. LOG metadata is automatically 9 generated and updated � Interpretable: i. LOG metadata is both human and Intelligent Learning Object Guide machine readable

Introduction: Metadata problems and i. LOG solution � LO Wrapper: logs student behaviors when using LO � Meta. Gen : generates empirical usage metadata using data mining techniques � Works noninvasively with pre-existing LOs using standard learning management systems (LMSs) 10 Intelligent Learning Object Guide LO Wrapper LO Metadata Learning Object Learning Managem ent System (LMS)

Related Work �Automatic metadata generation �Primarily focuses on content taxonomies (Roy et al. , 2008; Jovanovic et al. , 2006) �Mining student behavior log files �Mining has been shown to have a positive impact on instruction and learning (Kobsa et al. , 2007) �Standardization of educational log file data �Significant progress has been made with tutor- message format standard (PSLC Data. Shop) 11 Intelligent Learning Object Guide

Overview �Introduction: �What is a Learning Object (LO)? �Why do we need LO metadata? �Metadata problems and i. LOG solution �i. LOG Framework �LO Wrapper �Meta. Gen (metadata generator) �Data Logging �Data Extraction �Data Analysis (feature selection, rule mining, statistics) �Conclusions and Future Work 12 Intelligent Learning Object Guide

i. LOG Framework LO Wrapper Rules and Statistics Generation Meta. Gen i. LOG dataset LO Metadata Database Learning Object Rule Feature Data Mining e Selection. Analysis Subset Data Extraction Log Files and Existing Metadata 13 Two components: LO Wrapper and Meta. Gen Intelligent Learning Object Guide Data Logging

i. LOG Framework: LO wrapper LO Wrapper LO Metadata Learning Object 14 Intelligent Learning Object Guide LO Wrapper: �‘Wraps’ around an existing LO �Intercepts student interactions and logs them to a database �Does not require changing the LO

i. LOG Framework: Meta. Gen Rules and Statistics Generation Meta. Gen i. LOG dataset Database Rule Mining Data Feature Subset Selection Analysis Data Extraction Meta. Gen modules: • Data Logging, Data Extraction, Data Analysis 15 Intelligent Learning Object Guide Data Logging

i. LOG Framework: Meta. Gen— Logging Meta. Gen LO Wrapper LO Metadata Database Learning Object Data Logging Log Files 16 Potential data sources: • Interactions: clicks, time spent, etc. • Surveys: demographic, motivation, self-efficacy, evaluation Intelligent Learning Object Guide • Assessment scores

i. LOG Framework: Meta. Gen— Logging �Data sources used in our i. LOG deployment: 17 Static Learner Data Static LO Data Interaction Data Baseline motivation Baseline selfefficacy Gender Major GPA SAT/ACT score ⁞ Topic Length Degree of difficulty Level of feedback. Blooms’ level for assessment questions ⁞ Total time on tutorial Total time on exercises Total time on assessment Min time spent on a tutorial page Max time spent on a tutorial page Avg. time per assessment question ⁞ Intelligent Learning Object Guide

i. LOG Framework: Meta. Gen— Extraction LO Wrapper LO Metadata Meta. Gen i. LOG dataset Database Learning Object Data Extraction Data Logging Log Files and Existing Metadata Data Extraction: � Uses Java application to query the relational database and extract a ‘flat dataset’ suitable for data mining: �Student Behaviors: Average time per tutorial page, Total time 18 on assessment, etc. �Student Characteristics: Total motivation self-rating, GPA, Gender, etc. Intelligent Learning Object Guide

i. LOG Framework: Meta. Gen— Analysis Meta. Gen LO Wrapper i. LOG dataset LO Metadata Database Learning Object Feature Data Selection Analysis e Subset Data Extraction Data Logging Log Files and Existing Metadata 19 Data Analysis (feature selection): �Uses ensemble of feature selection algorithms �Seeks to identify student behaviors and characteristics that are relevant to learning Intelligent Learning Object Guide outcomes

i. LOG Framework: Meta. Gen— Analysis � Feature selection (FS) is used 20 to find a subset of variables (features) that is sufficient to describe a dataset (Guyon et al. , 2003) � Different techniques may generate different results � Instead, our goal was to find ALL features relevant to learning outcomes � Thus, the feature selection ensemble members ‘vote’ on Intelligent Learning Object Guide which features they identify as FS# 1 FS# 2 FS# 3 All features

i. LOG Framework: Meta. Gen— Analysis Notable Results: �Relevant features varied widely across LOs �Discovered unexpected patterns: �Possible gender bias , Calculus bias, etc. Logic 2 21 Attribute Number of Times Selected highest. Math gender taken. Calculus assess. Std. Dev. Sec. Above. Avg? was. Any. Part. Confusing? 16 13 13 Intelligent Learning Object Guide Searching Attribute GPA assess. Min. Sec. Page. Below. Avg? assessment. Min. Sconds. On. APage believe. LODifficult. To. Understand course. Level Number of Times Selected 14 11 10 10 9

i. LOG Framework: Meta. Gen—Analysis Meta. Gen LO Wrapper i. LOG dataset LO Metadata Database Learning Object Rule Feature Data Selection Analysis Mining e Subset Data Extraction Data Logging Log Files and Existing Metadata Rule Mining: • Uses Tertius algorithm for predictive rule mining • Generates rules from selected features (along with rule strength) taken. Calculus? = no fail (. 52) 22 current. Total. Motivation. Above. Avg? = no fail (. 52) gender = female fail (. 36) Intelligent Learning Object Guide

i. LOG Framework: Meta. Gen— Analysis Statistics Generation LO Wrapper Meta. Gen i. LOG dataset LO Metadata Database Learning Object Rule Feature Data Selection Analysis Mining e Subset Data Extraction Data Logging Log Files and Existing Metadata Statistics Generation: • Empirical data: time to complete, pass/fail rates, and student ratings of LO success. Rate = 51% average. Time = 433 seconds average. Student. Rating = 4. 3/5. 0 23 Intelligent Learning Object Guide

i. LOG Framework: Meta. Gen— Analysis Logic 2—Intro CS for non-majors assessment. Std. Dev. Seconds. Above. Avg? = yes fail (. 35) success. Rate = 51% assessment. Max. Seconds. On. AQuestion = high fail (. 33) average. Time = 433 seconds highest. Math = precalculus fail (. 28) average. Student. Rating = 4. 3/5. 0 gender = female fail (. 24) Logic 2 --Intro CS for majors baseline. Std. Dev. Motivation = low fail (. 72) success. Rate = 38% taken. Calculus? = no fail (. 52) average. Time = 688 seconds current. Total. Motivation. Above. Avg? = no fail (. 52) average. Student. Rating = 4. 16/5. 0 Logic 2—Honors Intro CS for majors Opinion. Of. LOUsability = negative fail (. 59) success. Rate = 55% Believe. LOAn. Aid. To. Understanding = yes pass (. 49) average. Time = 799 seconds Believe. LONeeds. More. Detail = yes fail (. 43) average. Student. Rating = 3. 43/5. 0 gender = female fail (. 36) 24 Appear to be different predictors of success for different learning contexts: • Honors: student impression of LO, gender • Majors: motivation, math experience • Non-majors: long time spent on assessment, math experience, Intelligent Learning Object Guide gender

i. LOG Framework: Meta. Gen— Analysis Logic 2—Intro CS for non-majors assessment. Std. Dev. Seconds. Above. Avg? = yes fail (. 35) success. Rate = 51% assessment. Max. Seconds. On. AQuestion = high fail (. 33) average. Time = 433 seconds highest. Math = precalculus fail (. 28) average. Student. Rating = 4. 3/5. 0 gender = female fail (. 24) Logic 2 --Intro CS for majors baseline. Std. Dev. Motivation = low fail (. 72) success. Rate = 38% taken. Calculus? = no fail (. 52) average. Time = 688 seconds current. Total. Motivation. Above. Avg? = no fail (. 52) average. Student. Rating = 4. 16/5. 0 Logic 2—Honors Intro CS for majors Opinion. Of. LOUsability = negative fail (. 59) success. Rate = 55% Believe. LOAn. Aid. To. Understanding = yes pass (. 49) average. Time = 799 seconds Believe. LONeeds. More. Detail = yes fail (. 43) average. Student. Rating = 3. 43/5. 0 gender = female fail (. 36) 25 Inverse relationship: time spent on LO and student ratings: • Advanced students may have higher expectations (lower ratings) Intelligent Learning Object Guide • Advanced students may care more about the material

i. LOG Framework: Meta. Gen— Analysis LO Wrapper Rules and Statistics Generation Meta. Gen i. LOG dataset LO Metadata Database Learning Object Rule Feature Data Selection Analysis Mining e Subset Data Extraction Data Logging Log Files and Existing Metadata Rules and Statistics: • Usage statistics and rules are combined to form empirical usage metadata 26 Intelligent Learning Object Guide

i. LOG Framework: Our Implementation LO wrapper: � HTML document that uses Java-script to record and timestamp student interactions with the LO (e. g. , page navigation, clicks on a page, etc. ). � Uses a modification of the Easy SCO Adapter 1 to interface with the SCORM API and retrieve student assessment results from the LMS. � Uses Java. Script to transmit interaction data to Meta. Gen: � Data logging: uses PHP to store student interaction data into a My. SQL database. � Data extraction: uses Java to query the database and process the data into the i. LOG dataset. � Data analysis: uses the Weka (Witten, 2005) implementations of several feature selection algorithms to generate the i. LOG data-subset and the (Flach, 2001) predictive rule mining algorithm to generate empirical usage metadata rules. 27 1[http: //www. ostyn. com/standards/demos/SCORM/wraps/easyscoadapterdoc. htm#li Intelligent cense]Learning Object Guide

Overview �Introduction: �What is a Learning Object (LO)? �Why do we need LO metadata? �Metadata problems and i. LOG solution �i. LOG Framework �LO Wrapper �Meta. Gen (metadata generator) �Data Logging �Data Extraction �Data Analysis (feature selection, rule mining, statistics) �Conclusions and Future Work 28 Intelligent Learning Object Guide

Conclusions i. LOG: a framework for automatic, empirical metadata generation: � LO Wrapper component: �“Wraps” noninvasively around pre-existing learning objects (LOs) �Automatically collects and logs student interaction data �Resulting LOs can be played on a standard LMS, such as Blackboard 29 � Meta. Gen component (metadata generator): �Uses data mining to create empirical usage metadata: �Feature selection: provides insights on which student characteristics and behaviors may contribute to success in different learning contexts. �Rule mining: uses salient features to generate rules predicting success Intelligent Learning Object Guide �Usage statistics: empirical evidence of time to complete,

Future Work: Closing the Loop LO Wrapper Rules and Statistics Generation Meta. Gen i. LOG dataset LO Metadata Database Learning Object Rule Feature Data Selection Analysis Mining e Subset Data Extraction Data Logging Log Files and Existing Metadata 30 • Method to automatically write empirical usage metadata to the LO metadata file • Method to integrate new metadata with existing metadata Intelligent Learning Object Guide

References � IEEE 1484. 12. 1 -2002 Standard for Learning Object Metadata (LOM). Retrieved January 7, 2009, � � � � � 31 from http: //ltsc. ieee. org/wg 12/files/LOM_1484_12_1_v 1_Final_Draft. pdf N. Friesen, The International Learning Object Metadata Survey. Retrieved August 7, 2008, from http: //www. irrodl. org/index. php/irrodl/article/view/195/277/ C. Brooks, J. Greer, E. Melis, C. Ullrich, Combining ITS and e. Learning Technologies: Opportunities and Challenges, Proc. 8 th Int. Conf. on Intelligent Tutoring Systems (2006), 278 -287. D. Roy, S Sarkar, S. Ghose, Automatic Extraction of Pedagogic Metadata from Learning Content, Int. J. of Artificial Intelligence in Education 18 (2008), 287 -314. J. Jovanovic, D. Gasevic, V. Devedzic, Ontology-Based Automatic Annotation of Learning Content, Int. J. on Semantic Web and Information Systems, 2(2) (2006), 91 -119. B. Jong, T. Chan, Y. Wu, Learning Log Explorer in E-Learning Diagnosis, IEEE Transactions on Education 50(3) (2007), 216 -228. E. Garcia, C. Romero, S. Ventura, C. Castro, An architecture for making recommendations to courseware authors using association rule mining and collaborative filtering, User Modeling and User-Adaptive Interaction (to appear). E. Kobsa, V. Dimitrova, R. Boyle, Adaptive Feedback Generation to support teachers in web-based distance education, User Modeling and User-Adapted Interaction 17 (2007), 379 -413. I. Guyon, A. Elisseeff, An Introduction to Variable and Feature Selection, Journal of Machine Learning Research 3 (2003), 1157 -1182. P. A. Flach, N. Lachiche, Confirmation-Guided Discovery of First-Order Rules with Tertius, Machine Learning 42 (2001), 61 -95. Ian H. Witten and Eibe Frank "Data Mining: Practical machine learning tools and techniques", 2 nd Edition, Morgan Kaufmann, San Francisco, 2005. Intelligent Learning Object Guide

Contact and Acknowledgement i. LOG project website: http: //cse. unl. edu/agents/ilog Authors: S. A. Rileya, L. D. Millera, L. -K. Soha, A. Samala, and G. Nugentb Email: sriley@cse. unl. edu, lmille@cse. unl. edu, lksoh@cse. unl. edu, samal@cse. unl. edu, gnugent 1@unl. edu � This material is based upon work supported by the 32 National Science Foundation under Grant No. 0632642 and an NSF GAANN fellowship. Intelligent Learning Object Guide