An Amazing Day New Ways of Showing How

An Amazing Day!

New Ways of Showing How Good Students Use a CMS John Fritz, UMBC January 21, 2009 ELI Annual Conference

Terms of Use "Copyright John Fritz, 2009. This work is the intellectual property of the author. Permission is granted for this material to be shared for non -commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author. "

Overview • • About UMBC Problem Demo Perspectives to Consider – Students – Faculty Future Plans Other Examples & Policy Implications More Information Q&A

About UMBC • Founded in 1966 • “Research extensive university” Carnegie classification • Fall 2007 Stats – 12, 041 Students • 9, 464 undergrad, 2, 577 grad – 1, 319 Faculty • 636 FT, 683 PT • Selected Brags – One of 50 Best Colleges for Women – 1 st in undergrad chemistry degrees awarded to African Americans – Six-time National College Chess Champions

About Blackboard @ UMBC • Learning System Version 7. 2 • As of Fall 2007 – – 11, 614 students (96% of all students) 1, 074 Bb course sites (includes multi-section courses) 808 Instructors (61% of all instructors) 356 Communities • Includes all student, faculty and staff senates • Support Staff: – 2 FTE (Admin & Support) – 1 Server Admin

“So, is Blackboard making a difference? ” Former UMBC Provost Art Johnson in 2002 PROBLEM HOW DO YOU ANSWER?

Current Reporting in Blackboard

Course vs. System “Drill Down” View?

Questions • Functional – What is the relationship between Blackboard use and teaching and learning? – What tools can we give users to shed light on (and improve) their own performance within the system? • Technical – How do we query the system without breaking it? – How do we scale and maintain the process? • Bb – Core product? • Community – Building Blocks? • Other?

New Tools & Approaches

UMBC Solution: Transparency Show faculty what peers are doing through publicly available reports of student use. Bb Sysadmins shouldn’t have the only “birds eye” view. “Average hits per student” course rankings don’t favor large courses over smaller ones. Usage alone is no indicator of quality. But activity by students piques faculty curiosity. New user tools build on “activity” as an indicator (not a cause) of student success. We did not have access to “official” grades in SIS, so we studied “unofficial” grades in the CMS.

Most Active Blackboard Courses Rankings (Publicly Available) Self Service Tool: Check My Activity (UMBC Students Only) Self Service Tool: Grade Distribution Reports (UMBC Faculty Only) DEMO

Recent News

www. umbc. edu/blackboard/repor ts

Three Types of Usage

Rankings Based on Usage

Course Tool Usage

Rankings by Department

Rankings in Context

How Faculty Can Connect

Best Practice: “Show & Tell”

Best Practice: Q&A (i. Tunes. U)

Self Service Tool: Check My Activity (UMBC Students Only) DEMO

All Users Start Outside Bb Using my. UMBC

Students See Personal Report of Activity

Students See Their Summary to Date

Comparison with Class Summary to Date

Grade Distribution Reports (UMBC Faculty Only—For Now) DEMO

Grade Distribution Reports by Bb Activity

Per Item Grade Book Activity Analysis

Sample Report for Final Grades in a Class

Student Usage Results

FA 2007 Faculty GDRs (19 courses)

After we ran this article. . .

SP 2008 Faculty GDRs (28 courses)

UMBC Bb Reports: GDR of D & F students • Based on voluntary GDRs run by instructors in 61 courses, students earning a D or F tend to use Bb 35 percent less than students earning a C or higher. – FA 2008 (13 courses | 40 percent less) – SU 2008 (7 courses | 33 percent less) – SP 2008 (26 courses | 32 percent less) – FA 2007 (15 courses | 36 percent less) • “What is the institution’s ethical obligation of knowing? ”- John Campbell, Purdue

PERSPECTIVES TO CONSIDER

Student Perspective • Check My Activity Tool Hasn’t Been Used As Much As Faculty Grade Distribution Report Tool. Why? – May be too buried in my. UMBC portal for students to find it – Not yet integrated into Bb under “My Grades” (B 2? ) • Leery of “Big Brother” or “Academic Profiling”? – See “Colleges Mine Student Data to Predict Dropouts” (5/30/08 Chronicle of Higher Education) – Univ. of Northern Arizona researchers reported student concerns with “intrusive advising” (“Academic Analytics, ” Educause Review, Vol. 42, No. 4, July/August, 2007) • Maybe students don’t think this is what a CMS is supposed to do?

FA 2008 SCI 100 Findings • How would you describe the CMA’s view of your Bb activity compared to your peers? – 28% “I was surprised by what it showed me” – 12% “It confirmed what I already knew” – 42% “I’d have to use it more to determine its usefulness” – 16% “I haven’t used it. ” – 2% did not respond to this question

Faculty Perspective • Concerns about “course ranking” based on average Bb hits per user? – Not a problem at UMBC (so far) • Cheap feedback to students in terms of faculty effort to produce it. – Better if they “publish” grade distribution reports in their courses, so students see how activity informs success. • Can CMS feedback change student motivation? And behavior (performance)?

FA 2008 SCI 100 Findings • If your instructor published a GDR for past assignments, would you be more or less inclined to use the CMA before future assignments are due? – 54% “More inclined” – 10% “Less inclined – 36% “Not sure”

Online Gradebook: A Symbiotic Relationship?

FUTURE PLANS

my. UMBC Blackboard “Flyover”

What We Want – Building Block or New App

Promote? Test? Expand Scope? • Continue to promote Check My Activity in Portal. – 75 percent of all users login to Bb via my. UMBC – Add “Check My Activity” on Blackboard “flyover” • Conduct focus groups and/or individual usability tests with students about Check My Activity. • Alert students who may be at risk? – Who decides? – How would this not come across as “intrusive advising”?

OTHER EXAMPLES AND POLICY IMPLICATIONS

Examples of other CMS “Data Mining” projects 5/30/08, Chronicle of Higher Education Argosy University Purdue University Slippery Rock University of Pennsylvania South Texas College SUNY Buffalo Tiffin University of Alabama University of Central Florida University System of Georgia Blackboard Greenhouse Grant - Project ASTRO OSCELOT. org, Advanced System Tracking & Reporting tool Hofstra University

“Colleges Mine Data to Predict Dropouts” “At the University System of Georgia, researchers monitored how frequently students viewed discussion posts and content pages on course Web sites for three different courses to find connections between online engagement and academic success. In the graph below, students who were "successful" received an A, B, or C in the class, and students who were "unsuccessful" received a D, F, or an incomplete. ” - 5/30/08 Chronicle of Higher Ed.

John Fritz fritz@umbc. edu | 410. 455. 6596 www. umbc. edu/blackboard/reports MORE INFORMATION

QUESTIONS?

Bb Sysadmin/Developer Perspective • No simple feat to query the system without breaking it. We did (twice). • How do we scale and maintain this process, which is necessary so users can learn to use these “self-service” tools?

INFRASTRUCTURE ISSUES

How We Query Bb: Static Replica Blackboard (Static Replica) Queries (PHP Scripts) Complete copy of database made infrequently, but as needed for reports Blackboard (Production) Cached Reports

Technical Issues to Consider • Pros – Can query current semester data with no impact on production – Data is up-to-date (at time of the static copy) • Cons – Requires a manual process to make the static copy – Need to know in advance when we want to run queries