Student Satisfaction with Online Courses Amber Settle CTI

Student Satisfaction with Online Courses Amber Settle, CTI, De. Paul University joint work with Chad Settle, University of Tulsa CCSC: Southeast Region November 11, 2005

Satisfaction with distance learning n Distance learning is popular with CTI students n n There are 8 M. S. and 1 M. A. degree online (out of 10 M. S. , 1 M. A. , and multiple joint degrees) Distance learning students are 21% of the student population It has been asserted that while outcomes are similar in DL and traditional classes, DL classes are less satisfying to students (Carr 2000) Is DL less satisfying for CTI students? If so, how?

The test cases n The courses n Foundations of Computer Science (CSC 415) n n n Programming in Java I and II (CSC 211 and CSC 212) n n n Graduate discrete mathematics 9 sections between Fall 2001 and Fall 2003 Java I: 8 sections between Fall 2003 and Winter 2005 Java II: 7 sections between Spring 2003 and Fall 2004 The format n n n Traditional Sibling DL: Runs parallel to a traditional class; entire classroom interaction is recorded automatically Pre-recorded DL: High production quality independent of any live class; broken into five modules (CSC 415 only)

Course evaluations n Conducted every quarter for every CTI course n n n Mandatory for all students Online using secure login; anonymous Completed during the 8 th and 9 th week of 10 week quarter Results are withheld from instructor until grades are submitted; results are then published on the CTI web site Consists of 22 multiple choice questions n n 10 questions about course-related factors; 12 questions about instructor-related factors Ratings on a scale from 0 to 10; a higher number indicates greater satisfaction; 0 indicates the question is not applicable

Summary of CSC 415 results n Similarities in evaluations: Overall scores n n None of the coefficients for instructor-related or courserelated questions were significantly different from 0 for DL sections Differences in evaluations: Not applicable response rate n Q-IR 5 (Encourage participation) n n n DL: 60% Traditional: 6% Q-IR 12 (Teaching effectiveness) n n DL: 15% Traditional: 2. 5%

Potential explanations Pre-recorded DL is better organized which compensates for the lack of interaction (Swan 2001) n DL students are not watching the recordings n Small DL sample size n

Summary of Java results n Course-related questions n n Two out of 10 questions (Q-CR 1: Course organization, Q-CR 2: Achieving course objectives) were statistically different from 0 for DL students Both coefficients were negative, indicating less satisfaction Similar results were not seen in an analysis of only the live sections Instructor-related questions n n n Nine out of 12 questions (all but Q-IR 4, Q-IR 7, Q-IR 10) were statistically different from 0 for DL students All coefficients on the 9 questions were negative (from -0. 3 to -0. 8) For live students only two questions were significant over time; one was positive (Q-IR 5: student participation), the other negative (Q-IR 6: availability) Sibling DL students are not as satisfied with the instructor as traditional students or students in live sibling sections Q-IR 12 (comparison to other instructors) is used for T/P and merit Not applicable rate was fairly consistent between sections; only differences on Q-IR 5 (7% for DL, < 1% for live) and Q-IR 7 (6% for DL, < 1% for live)

Potential explanations n n n Larger data set DL students are more sensitive to course organization because of the lack of interaction A more interactive sibling course will highlight the shortcomings of the class for DL students n n n Watching the students in the sibling section participate Listening to instructor questions with no chance of responding The course evaluations do not allow evaluation of the course delivery technology, causing the instructor ratings to suffer

Appendix n Course evaluation questions Course-related questions n Instructor-related questions n n Statistical analysis Least squares regression n Data for CSC 415 n Data for Java I and II n

Course-related questions 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. Was this course well organized? Do you feel the course objectives were accomplished? The amount of work you performed outside of this course was: How difficult was this course material? The textbook for this course was: Supplementary reading for this course was: The assignments for this course were: What is your overall estimate of this course? How valuable was this course in terms in your technical development? Would you recommend this course to another student?

Instructor-related questions 1. 2. 3. 4. 5. 6. How would you characterize the instructor’s knowledge of this subject? How would you characterize the instructor’s ability to present and explain the material? Does the instructor motivate student interest in the subject? How well does the instructor relate the course material to other fields? Did the instructor encourage participation from the students? Was the instructor accessible outside of class?

Instructor-related questions continued 7. 8. 9. 10. 11. 12. What was the instructor’s attitude? How did he/she deal with you? How well did the instructor conduct, plan, and organize classes? Were the instructor’s teaching methods effective? How fair was the grading of the homework and exams of this course? Would you take this instructor for another course? Rate the teaching effectiveness of this instructor as compared to other faculty in the department.

Statistical analysis n Ordinary least squares regression: Qi = 0 + 1 X 1 i + 2 X 2 i + ui Question i Constant n n Time DL Error X 2 i = 0 for traditional, X 2 i = 1 for DL If 2 is statistically different from 0, it indicates a difference in how DL students view the course vs. traditional students

CSC 415: Course-related results Question Time DL Q-CR 1 -0. 164** (0. 075) 0. 200 (0. 393) Q-CR 6 0. 046 (0. 164) 0. 302 (0. 910) Q-CR 2 -0. 120 (0. 079) -0. 023 (0. 416) Q-CR 7 -0. 071 (0. 101) 0. 237 (0. 533) Q-CR 3 0. 064 (0. 086) 0. 221 (0. 460) Q-CR 8 -0. 179** (0. 089) 0. 594 (0. 467) Q-CR 4 0. 305*** (0. 102) -0. 549 (0. 530) Q-CR 9 -0. 199* (0. 110) 0. 620 (0. 573) Q-CR 5 0. 185 (0. 144) -0. 261 (0. 758) Q-CR 10 -0. 202** (0. 092) 0. 067 (0. 486) Coefficient estimates are presented with standard errors in parentheses. *Statistically significant at the 10% level of a two-tailed test. **Statistically significant at the 5% level on a two-tailed test. ***Statistically significant at the 1% level of a two-tailed test.

CSC 415: Instructor-related results Question Time DL Q-IR 1 -0. 010 (0. 072) 0. 065 (0. 379) Q-IR 7 -0. 115 (0. 084) 0. 013 (0. 437) Q-IR 2 -0. 069 (0. 069) -0. 029 (0. 360) Q-IR 8 -0. 110 (0. 084) 0. 042 (0. 442) Q-IR 3 -0. 194** (0. 083) -0. 016 (0. 437) Q-IR 9 -0. 181** (0. 088) 0. 570 (0. 454) Q-IR 4 -0. 189* (0. 098) 0. 043 (0. 515) Q-IR 10 0. 010 (0. 087) 0. 304 (0. 451) Q-IR 5 -0. 267** (0. 102) 0. 481 (0. 654) Q-IR 11 -0. 096 (0. 089) 0. 411 (0. 461) Q-IR 6 -0. 053 (0. 090) -0. 183 (0. 468) Q-IR 12 -0. 114 (0. 082) 0. 268 (0. 443) Coefficient estimates are presented with standard errors in parentheses. *Statistically significant at the 10% level on a two-tailed test. **Statistically significant at the 5% level on a two-tailed test.

Java: Course-related results Question Time DL Q-CR 1 -0. 034 (0. 040) -0. 736*** (0. 226) Q-CR 6 -0. 130** (0. 063) -0. 524 (0. 367) Q-CR 2 -0. 038 (0. 037) -0. 431** (0. 209) Q-CR 7 -0. 047 (0. 044) -0. 037 (0. 252) Q-CR 3 -0. 035 (0. 037) -0. 013 (0. 210) Q-CR 8 -0. 017 (0. 043) 0. 083 (0. 243) Q-CR 4 -0. 096** (0. 046) -0. 149 (0. 261) Q-CR 9 -0. 068 (0. 049) 0. 129 (0. 278) Q-CR 5 -0. 114* (0. 059) 0. 357 (0. 333) Q-CR 10 0. 023 (0. 051) -0. 425 (0. 286) Coefficient estimates are presented with standard errors in parentheses. *Statistically significant at the 10% level of a two-tailed test. **Statistically significant at the 5% level on a two-tailed test. ***Statistically significant at the 1% level of a two-tailed test.

Java: Instructor-related results Question Time DL Q-IR 1 0. 017 (0. 028) -0. 394*** (0. 157) Q-IR 7 -0. 018 (0. 044) -0. 304 (0. 252) Q-IR 2 0. 023 (0. 042) -0. 648*** (0. 239) Q-IR 8 -0. 009 (0. 039) -0. 874*** (0. 222) Q-IR 3 -0. 019 (0. 048) -0. 622** (0. 275) Q-IR 9 0. 051 (0. 048) -0. 754*** (0. 275) Q-IR 4 -0. 026 (0. 057) -0. 143 (0. 317) Q-IR 10 -0. 069 (0. 042) -0. 143 (0. 239) Q-IR 5 0. 075* (0. 045) -0. 718*** (0. 257) Q-IR 11 0. 034 (0. 055) -0. 770** (0. 313) Q-IR 6 -0. 071* (0. 038) -0. 375* (0. 217) Q-IR 12 0. 010 (0. 044) -0. 635** (0. 254) Coefficient estimates are presented with standard errors in parentheses. *Statistically significant at the 10% level of a two-tailed test. **Statistically significant at the 5% level on a two-tailed test. ***Statistically significant at the 1% level of a two-tailed test.