The Learning Online Network with CAPA A CrossInstitutional

The Learning. Online Network with CAPA A Cross-Institutional Open-Source Learning Content Management and Assessment System Background and Research Prospectus Gerd Kortemeyer Michigan State University Dec. 2003

Some of the History

1992 CAPA • Started by Ed Kashy et al. in Cyclotron • Individualized assessment system for science and math • Immediate feedback, multiple tries mastery based • Used paper copies of assignments and terminal input • X-Windows problem editing • Got Web student interface in 95

1997 Lecture. Online • Started by Wolfgang Bauer, Walt Benenson, Gary Westfall, and Gerd Kortemeyer in Cyclotron • Learning content management and individualized assessment system for science and math • Sharing of content between courses • Completely web-based interface

1999 LON-CAPA • The Learning. Online Network with CAPA • Collaboration of CAPA and Lecture. Online groups • “The best of both worlds” • Sharing of content between courses and institutions • Reusability of content on different levels of granularity • Distributed and Scalable

LON-CAPA Today • Useful and usable for “casual” (non-”power”) faculty user outside science and math • LON-CAPA is a full-featured course management system (calendar, communication, …) • 2003 Computer. World 21 st Century Achievement Award • Directly or indirectly attracted approx. $4 M in grant funding at MSU • Version 1. 0 was rolled out this Fall

Today LON-CAPA is used … • With material written by faculty teaching course or “re-used” from other faculty • For homework in addition to “traditional” lecture and textbook + helproom • Hybrid with “traditional” lecture, homework and textbook online + helproom • Completely “virtual” courses • AP courses • Prelab quizzes • In-class exercises • Class sizes: 10 -3, 000

LON-CAPA@MSU

Course Enrollments at MSU 13, 400 enrollments 9, 300 individuals

Subject Areas MSU Fall 2003

Support at MSU • Hosted out of central Computer Lab • Two fulltime support staff in CNS LITE Lab • One-Stop MSU Faculty Support Site • Request ticketing system • 24 hour line • CMS transition support • Coupling with central admin systems • LON-CAPA bubble-sheet exam support in central Scoring Office

Training at MSU • Part of regular LCTTP offerings in Spring 2004 • Offering department seminar, workshops, and colloquia

Cross-Institutional

System-wide Use • 26 user institutions • 24, 000 course enrollments/semester systemwide • Middle Schools: 300 • High Schools: 500 • Community Colleges: 50 • Four-year Colleges: 300 • Universities: >22, 000.

Distributed Server Network • LON-CAPA is built as a geographically distributed network of constantly connected servers

• The shared content repository appears as one searchable virtual filesystem • Built-in dissemination

Resource Assembly • Pages can be constructed from fragments and other pages • Sequences can be constructed from fragments, pages, and other sequences • Courses point to (top-level) sequences • Maps at every level are simply other content resources

Resource Assembly • Graphical resource assembly tool (RAT) to construct maps • Working on branching based on conditions using performance data, course data, and preferences • Individualized curriculum

Shared Resource Pool Shared resource pool currently holds approximately • 6, 000 original homework and exam problems • 5, 000 images • 150 movies • 180 java applets • 3, 000 content pages.

Content Material

Individualized Assessment

“Individualized Assessment” • Individualized problems: different numbers, different graphs, different options, … • “Classical” online homework elements: multiple choice, option response, mix-and-match, string, etc • “STEM” elements: numerical, multicomponent numerical, physical units, symbolic math, individualized simulations • Free combination of the above elements • Adaptive immediate feedback • Multiple attempts - mastery based

“Individualized Assessment” • Example: Individualized graph, numerical answer

“Individualized Assessment” Randomized labels and options

“Individualized Assessment” • Example: Multicomponent numerical with individualized animation and physical units

Applet: Initial and Final Action Frames

Online

Online

Offline Exam

Offline Exam

Item Analysis

Doing Item Analysis on Randomized Questions <conceptgroup concept=”Fruit”> <foil>Apple</foil> <foil>Orange</foil> </conceptgroup> <conceptgroup concept=”Metals”> <foil>Copper</foil> <foil>Gold</foil> </conceptgroup>

First 18. 6 % of responses 1. Two masses have the same acceleration if the string does not stretch. 2. Weight of the two masses is greater than the tension of the string attached to the ceiling 3. The top tension equals the bottom tensions. (massless pulley) 4. Tension holding the two masses are equal if mass of pulley = 0 5. Sub-system accelerates upwards or downwards accordingly 6. Center of mass accelerates downward

Learner Evaluations of LON-CAPA

Students usually find LONCAPA helpful. Example: Physics for Scientists and Engineers CAPA as Tool for Learning/Understanding

Students usually claim that they spend more time on task (and still find it helpful)

Students usually would like to use LONCAPA again in the next semester Question: If I had a choice between the CAPA scheme as used in PHY 184 and the recitation scheme used in PHY 183, I would be: Phy 184 Spring 95 N=156 Birge, Cowen A - Strongly in favor of recitations B - Moderately in favor of recitations C - Indifferent D - Moderately in favor of CAPA E - Strongly in favor of CAPA

Learning Outcomes (in terms of grades …)

Grades on Midterms and Finals Prof. Danielewicz (Physics)

Distribution of grades in Introductory Physics for Scientists and Engineers 1992 -1994 w/o CAPA 1996+ with CAPA

Current Research Projects

NSF ITR • “Investigation of a Model for Online Resource Creation and Sharing in Education Settings” • Uses LON-CAPA as model system • Five year project, $2. 1 M, funded by the National Science Foundation Information Technology Research Program

ITR: Research Goals • Open source code development concept for Instructional Management System • Pooling online educational resources between departments, colleges, universities and schools • Business models for “trading” such resources on small level of granularity • Business models for services and support • Metrics for resource quality and effectiveness • Curriculum adaptivity to learner types and remediation needs

NSF ASA • “Diagnostic Question Clusters: Development and Testing in Introductory Geology and Biology” • Joyce Parker, PI • Uses LON-CAPA as model system • Three year project, $0. 5 M, funded by the National Science Foundation Assessment of Student Achievement Program

ASA: Research Goals • Development of a pool of diagnostic question clusters to assess students' understanding of introductory biology and geology • Tools to better analyze students’ responses • Mechanisms for peer-review and publication

NSF National STEM Digital Library

Future Research Possibilities

Feedback and Help

Bad Feedback? M. Pascarella, CAPA in a Large University Setting, Ph. D. dissertation (Physics), University of Colorado (2002): 1. CAPA evaluated more helpful by students who have a less expert-like view of science 2. Multiple tries on problems led students to abandon some of their skills to verify the validity of results 3. Undesirable outcome attributed to immediate feedback and penalty-free multiple tries

1. Yet, feedback is a good thing and multiple cycles are a good thing: “Because many new technologies are interactive, it is now easier to create environments in which students can learn by doing, receive feedback, and continually refine their understanding and build new knowledge. ” Bransford et al. “How People Learn”

Questions: • What kind of feedback helps which learners? –Yes/No –Follow-up questions –Adaptive references to online material –No feedback? • How to use multiple attempts or assessment cycles to help “refine understanding” [Bransford] instead of “turning learners into guessers” [Pascarella]?

Helproom • Helproom at MSU started out: –Very few computers, no seating in front of computers –Round tables –Big sheets of paper –Whiteboard all around the walls –TAs present –Basement “cave” –Absolutely crowded • The all-new helproom since last two years: –Lots of desktop computers in cubicles –Less whiteboards, less round tables

Questions: • What are the characteristics of an effective helproom? • Use of wireless laptops, collaborative environment? • How might this best be implemented in the residential environment of LBS? • How to make a helproom work across disciplines?

Gender Differences

• Grade distributions for male and female students for a twosemester course (PHY 231 and PHY 232). • PHY 231 was taught the traditional way, and in PHY 232 CAPA was used.

Questions: • Is this gender difference real? • Does it occur across disciplines? • What are the reasons?

“Cheating”

Influence of Websites 3 rd Party Monitored (posting) Monitored (looking) Homework 0. 041 0. 118 -0. 109 Quizzes -0. 302 0. 098 0. 069 Midterm Exams -0. 352 0. 166 0. 160 Final Exam -0. 348 0. 147 0. 129

Questions: • How do learner and educator definitions of “cheating” differ? • What course and online design favors collaboration over cheating? • Which kind of curricular material discourages cheating and fosters conceptual discussions?

Just-in-Time-Teaching and In-Class Use

In-class use of LON-CAPA • Done in small courses at Westshore CC • Adapt to LON-CAPA and implement curricular material such as Project Galileo • PRS, PDA

Questions: • Does this approach scale to class sizes at MSU? • Can technological hurdles be overcome? • Does this approach increase the effectiveness of lecture time?

Predicting Success, Learners at Risk

B. Minaei, Ph. D. Computer Science (MSU) “Data-Mining Online Educational Systems” [LON-CAPA] Best Predictor Good Predictor Total Number of Correct Answers Predictor • Gets Problems Correct on the First Try • Time spent Minor Predictor Participation in online Discussions Total Number of Tries Preliminary, work in progress

Predictive power of performance on different types of assessments Covariates included ACT, College GPA, Initial Physics knowledge (Force Concepts Inventory), and number of absences

P. Kotas, Homework Behavior in an Introductory Physics Course, Masters Thesis (Physics), Central Michigan University (2000) [CAPA] • Best Predictor: Procrastination on homework (negative) • Second Best: interacting with the material on a daily base (positive) • Third: working on the homework immediately following the lectures (positive)

Questions: • What is cause, what is effect? • Can an effective “early warning” system be implemented? • Can the system guide students in their navigational behavior toward greater success?

Possible Funding Sources • NSF Assessment of Student Achievement (ASA) • NSF Course, Curriculum, Laboratory Improvement (CCLI) including Adaptation and Implementation track • NSF Research on Learning and Education (ROLE) • Sloan Foundation

On the web. . . www. lon-capa. org