DOES QUALITY IN ONLINE COURSE DESIGN MATTER The

DOES QUALITY IN ONLINE COURSE DESIGN MATTER? The Relationship Between Quality Assurance & Student Grades Alissa Sells - SBCTC Educational Technology & Open Education Darby Kaikkonen - SBCTC Research 10. 29. 19

Session Objectives o List 3 limitations of this study. o Name 2 confounding/explanatory variables that were controlled for in this study. o Identify components required to conduct similar quality assurance research on your campus.

SBCTC QM Implementation Course Reviews Training Subscription System level subscription funded annually for all 34 CTCs in Washington. APPQMR, IYOC and other selected QM training for faculty, and staff. Official QM course reviews are not currently funded.

Project Components 1 6 7 Research Question Permission Editors 2 5 Research Partner Participant Consent 8 Time 3 4 Data Library Access 9 Patience

1 Research Questions Mean student course GPA before & after APPQMR workshop completion? What is the relationship between… Mean student course GPA before & after official QM course certification? Mean student course GPA before APPQMR workshop completion & after official QM course certification?

2 Research Partner SBCTC Educational Technology SBCTC Research Quality Partnershi p

3 Data (2008 – 2018) 138 officially certified QM courses SBCTC Data Warehouse My. QM Excel 46 Excludes 33 faculty without verifiable APPQMR training records, no training, or informal QM training Excludes 52 recertifications, non HE rubric reviews, and additional certified courses from the same faculty Excludes +/-7 faculty who did not have course data for a full year before and after training and before and after course certification

4 Permission o No permission to conduct research was required. o System data collected quarterly for policy research analysis, accountability reporting, and research and evaluation of educational programs

5 Consent o Consent was not required. o Faculty were informed of their possible inclusion in the study by email.

6 Library Access Ralston-Berg (2011) CSU Student Quality Assurance Impact Research (SQu. AIR) Harkness (2014) Hollowell, Brooks, and Anderson (2017) Baranovic et al (2017) • Found no significant differences in student satisfaction, grades, or retention between before and after QM certified online courses. • Studying the impact of QA professional development and course certification on student grades, course completion rates, student satisfaction, and achievement gaps. • Found a 66% percent decrease in failing grades (F), a 20% increase in passing grades (AD), and a 23% decrease in course withdraws in newly designed online courses in relation to faculty QM training. • Found a significant correlation between informal QM course review scores and student final exam performance and final grades. • Found a steady increase in Agrades in courses meeting QM standards with the total GPA for all online courses increasing 1. 25% from 2014 to 2017.

7 Editors o Feedback provided by my supervisor, & my research partner. o Feedback provided by Kay Shattuck & Barbra Burch

8 Time & 9 Patience o This has been a 2 year process and we’re not quite finished. o Still to do…final content draft, journal formatting, 2 page brief, journal submission.

Assumptions o We had the data we needed. o Faculty would be willing to have their course included in the study. o Faculty made changes to their course after completing QM’s APPQMR workshop and continued to make changes to their course until official course certification was achieved.

Limitations Small sample size. Does not measure every possible factor that might influence grades. SBCTC system data may not be generalizable to other community and technical college students. No faculty demographics were collected. Before and after data is compiled from different student No record of specific changes sets. made to courses.

Methodology - MEASUREMENT Compared the average before and average after course GPAs for each of the three measurement periods. Before After 1 yr. prior to the training or course certification date. 1 yr. following the training or course certification date.

Methodology - TESTING Analysis of Variance • ANOVA • Used to test for statistically significant differences in each of the before and after groups. Equity Gap Analysis • Secondary Testing • Used to assess if the observed change in student course GPA from before to after might be due to factors related to characteristics of the student population Explanatory/ Confounding Variables • Age, sex, disability status, low income status, full or part-time status, number of college credits earned at the time of the course, kind of student, and historically underrepresented students of color.

Results Before & After Training Before & After Certification Before Training & After Certification ANOVA: Only the before APPQMR training and after official course certification showed a statistically significant increase. (. 24%) GAP ANALYSIS: Revealed possible areas for additional study related to specific student populations.

Areas for further study… o What is the effect of faculty QM training and the QM course certification process on targeted demographic groups such as students of color, students with disabilities, or by age or gender?

Other things to look at… o Changes in course withdraw rates. o Changes in course completion rates. o Increases in passing grades.

Summary o The quality assurance process of training combined with course certification appears to contribute to increases in student grades more so than just training or course certification alone. o There is room for further study on the effectiveness of the Quality Matters program in general and in relation to specific student populations. 20

Potential Impact o To date our system has focused primarily on the training aspects of QM implementation and our recent findings support the importance of expanding that process to include course certification. o QM subscribers focused solely on the training aspects of quality assurance may need to re-evaluate and consider how to incorporate the course review process into their quality assurance program.

Let’s Review o Who can list 3 limitations of this study? o Who can name 2 confounding/explanatory variables that were controlled for in this study? o Who can identify 3 of the 9 components required to conduct similar quality assurance research on your campus?

Contact Us Alissa Sells Darby Kaikkonen Program Administrator System Lead QMC Policy Research Director asells@sbctc. edu dkaikkonen@sbctc. edu 425. 239. 0456 360. 704. 1019 @WAe. Learning

Thank you! Unless otherwise specified, the content in this presentation is licensed under a Creative Commons Attribution 4. 0 International License.