ONE Scholars Program Reports on the Scholarship of

@ONE Scholars Program — Reports on the Scholarship of Teaching and Learning: Comparing SLOs and Performance Measures in Online and On-Campus Instruction Strengthening Student Success Conference October 4, 2007 Michelle Barton, Palomar College Darla Cooper, Santa Barbara City College Behzad Izadi, Cypress College Roy Mason, Mt. San Jacinto College 1

Scholarship of Teaching and Learning l Accreditation standards require evidence of learning (SLOs) l Requires assessment that is valid and authentic l Ongoing evaluation l Faculty leadership l Research in the classroom (classroom action research) 2

@ONE Scholar Program Overview l Goals of Program 1. Evaluate effectiveness of technology in CCC classrooms 2. Create and share original research on the impact of technology on student learning 3. Support the improvement of faculty members’ research abilities, technology skills, and pedagogical approaches l 10 scholars selected each year through competitive fellowship application 3

Role of the Researchers l Primary role was to provide technical assistance to the scholars throughout the entire research process by helping to: ¡ ¡ ¡ Refine and solidify research question, design and methodology Review and develop survey instruments, databases, and definition files Perform statistical analyses Analyze and interpret data Review monographs and provide feedback to scholars Present data in the monograph 4

Effect of Virtual Machine Technology in Enhancing Student Learning Outcomes Behzad Izadi, @ONE Scholar Cypress College Computer Information Systems Cypress, CA 5

Problem Statement l Most of the people who take computer networking courses are adult learners. l To attract more students must offer 100% online courses. l Must find a substitute for the physical laboratory. l Recent advances in Virtual Machine technology have made it possible to create and maintain virtual laboratories. 6

Research Hypotheses Virtual machine technology is capable of creating and maintaining virtual laboratories in a 100% online classroom environment. l Students will have a similar experience in conducting labs using a virtual laboratory as compared to a physical laboratory. l The performance of the online students will be comparable to the on-campus students. l 7

CIS 189: Microsoft Windows Active Directory l Microsoft Active Directory is a directory service which provides a way to locate, manage, and control objects and resources on the network. l This course is included in vocational certificates and is often taken by professionals to enhance their knowledge of the Microsoft directory service. 8

Class Formats CIS 189 was offered online & on-campus. l The on-campus class consisted of one hour of lecture, and the remaining 3 hours were used for conducting laboratory exercises. l The on-line class only met in person on the first day and last day of instruction. All the material for the on-line students was available on a Blackboard site. 9

Technology l Virtual laboratories were created by running Virtual Machine software on a dedicated server. l The server was equipped with 12 GB of memory, 500 GB of hard drive capacity, and 2 Intel Xeon 3 GHZ processors for faster response time. 10

Subjects l 8 online students had access to virtual laboratory. l 9 on-campus students had access to physical laboratory. l Students in each group were similar in terms of education, computer skills, computer networking background, previous knowledge of the Active Directory, and weekly work hours. 11

Data Collection Student Performance & Background l 12 Quizzes (open book) l 12 Lab Activities (submitting lab worksheet) l 3 Special Lab Projects (pass/fail) l 1 Final Exam (50 multiple choice questions) l Pre & Post Course Survey 12

Student Learning Outcome The student will be able to answer basic concept questions and perform skillsbased activities while proctored by the instructor and score 60% or greater. SLO Assessment l 1 Concept Exam (25 multiple choice questions) l 1 Skills Test (9 hands-on questions) 13

Data Analysis Comparing the 2 Groups l Independent-samples t-test Analysis Comparing quizzes, lab activities, final exam, and SLO assessment. l Frequency Analysis Comparing final exam, SLO assessment, and skills test. 14

Data Categories Online On-Campus Difference* Quizzes 92. 4 95. 5 3. 1 Lab Questionnaires 97. 6 96. 7 -0. 9 Final Exam 68. 8 73. 3 4. 5 SLO Assessment 65. 4 73. 3 7. 9 *For each category, the difference is not statistically significant [ p <. 05]. 15

Results l l l The t-test analysis indicated that the mean performance scores were not significantly different (p<0. 05). Both groups performed exceptionally well on quizzes and laboratory questionnaires. The two groups were comparable in terms of final exam and skills test scores. Majority of students from both groups achieved the SLO. On-campus students performed better on SLO concept exam. 16

Conclusions Virtual machine technology was capable of creating and maintaining virtual laboratories in a 100% online classroom environment. 75% the online students preferred the online environment over the traditional classroom. l 75% felt that the virtual laboratory was very helpful in completing lab activities and special projects, and 25% found them helpful. l 17

Conclusions Students had a similar experience in conducting labs using a virtual laboratory as compared to a physical laboratory. Once the virtual computers were booted up, the online students followed exactly the same steps to complete the laboratory activities as compared to the on-campus students. l Both groups completed special projects, which involved conducting complicated laboratory exercises, successfully without any additional assistance from the instructor. l 18

Conclusions The performance of the online students was comparable to the on-campus students. Students scores on quizzes, exams, homework, and the SLO measure were not statistically significant. l In the final exam, a similar number of students scored in the three different performance categories between the two groups: unsatisfactory (score<=60%), satisfactory (60%<score<80%), and good (score>=80%). l 19

Implications for the Classroom For the online course used in this study, the virtual laboratory required approximately four additional hours per week to set up virtual computers and conduct associated laboratory activities. l Implementing virtual machine technology requires support of the college in providing the necessary hardware, software and connectivity. l A proper virtual laboratory set up can replace the physical laboratory for both on-campus and online students. l 20

Student Learning Outcome Results Virtual machine technology enhanced student learning outcomes (SLOs) in a computer networking course. l Being able to offer an online networking course which includes lab projects. l On-campus and online students performed similarly in lab activities. l Both on-campus and online students achieved the skills-based portion of the SLO assessment. l 21

SLOs, Future Studies l Have the lectures available online to enhance the understanding of basic course concepts. l Revise the skills-based assessment to make it more practical (i. e. simple handson exercises using the virtual server). l Include at least fifteen to twenty students in each group to solidify the statistical analysis. 22

Implementing Recommendations l Cypress College now offers networking courses online and has fewer cancellations l I am offering the course (CIS 189) this semester online only l I am planning to put more resources online l I am planning to put mini lecture movies online (Joe Toto’s @ONE study) 23

Questions & Answers 24

Assessing Student Learning Outcomes: Comparison of online and On-Campus Biological Science Teaching Environments Will You Miss Me When I’m Gone? Confessions of an online Biology Professor Roy Mason, @ONE Scholar Mt San Jacinto College Biology Department Menifee, CA 25

Research Concern Some higher education decision makers are questioning the equivalency of online courses to on-campus courses for degree completion and transferability. 26

Research Hypothesis Online course delivery is equivalent to oncampus course delivery in determining a student’s level of attainment of student learning outcomes in a non-majors introductory biology course. 27

Research Methods and Sample Selection Fall Semester 2006 l Taught two sections of Biology 115 in the traditional on-campus face to face format. l Taught three sections of Biology 115 in the online format using a Blackboard Instructional Delivery system. 142 students: l 71 on-campus format l 71 online format 28

Data Collection Collect data in each of four Assessment Categories: l Examinations (12) l Scientific Article Evaluation (4) l Laboratory Exercises (12) l Field Trip (1) 29

Research Methods Causal-Comparative Quantitative Study l Used equivalent assessment measures of student learning outcomes in both on-campus and online formats. l Identified assessments that measure a student’s level of attainment of the Student Learning Outcomes/Course Objectives identified for successful completion of Biology 115. l Compared online student performance on the identified assessments with the performance of students taking the class in the traditional on-campus format. 30

Biology 115 Course Student Learning Outcomes – Lecture 1. Analyze and solve problems using the methods of scientific inquiry, observation and measurement. 2. Integrate the basic principles of chemistry to construct an understanding of biology at the cellular, organismal, and ecosystem levels. 3. Learn the language of biology and correctly use terms in written and oral communication. 4. Know what defines life and be able to discuss biological organization extending from molecules to ecosystems. 31

Biology 115 Course Student Learning Outcomes – Lecture, continued 5. Construct and illustrate the similarities and differences of prokaryotic and eukaryotic cells with regard to structure and function, illustrate traffic across membranes, and inspect the acquisition and use of energy in metabolic processes. 6. Evaluate and illustrate mitosis, meiosis, and the flow of information from genes to protein. 7. Characterize Mendelian principles of inheritance, and evaluate and appraise the significance of DNA technology. 8. Formulate hypotheses of how structure correlates with function at all levels of hierarchical organization. 32

Biology 115 Course Student Learning Outcomes – Lecture, continued 9. Evaluate and draw conclusions about the interactions between organisms and the interactions organisms have with their environment and analyze the various mechanisms by which organisms maintain homeostasis. 10. Characterize and authenticate the unity and diversity of all organisms within the Kingdoms and Domains of classification. 11. Demonstrate the mechanisms of evolution, and analyze the evidence in support of evolutionary theory. 12. Relate the interdependent nature of science to technology, and evaluate the role of society in shaping the application of scientific knowledge. 33

Biology 115 Course Student Learning Outcomes – Laboratory 1. Engage in the collection of data using various measures, apply scientific methods to investigate biological principles, observe phenomena, test hypotheses, solve problems, and report in scientific format. 2. Using the periodic table, construct diagrams of atoms, molecules, and illustrate the bonding capacities, and design models of macromolecules. 3. Demonstrate proficient understanding of the principles of microscopy in the study of cells and microorganisms. 34

Biology 115 Course Student Learning Outcomes – Laboratory, continued 4. Illustrate and distinguish the differences between the cell types of prokaryotes and eukaryotes, compare and contrast their structures, and critique methods to differentiate organelles. 5. Evaluate the process of osmosis occurring across a semi-permeable membrane, and deduce direction of net water movement under isotonic, hypertonic, and hypotonic conditions. 6. Evaluate and illustrate the phases of mitosis and meiosis through direct observation of mitotically dividing onion root tip cells and other prepared slides. 35

Biology 115 Course Student Learning Outcomes – Laboratory, continued 7. Characterize the complexity of DNA by creating a model, extracting DNA from cells, and/or witnessing DNA technology and its applications. 8. Analyze natural selection including differential survival, reproduction. 9. Identify and critique representative organisms of the five kingdoms: major phyla of Kingdoms Animalia and Protista, classes of vertebrate animals, major divisions of Kingdoms Plantae and Fungi, and bacterial and fungal microorganisms. 36

Biology 115 Course Student Learning Outcomes – Laboratory, continued 10. Locate, describe and identify external anatomical structures and internal organs in a frog and compare their presence and function to that of human anatomy. 11. Identify, contrast and evaluate the chemical processes of photosynthesis and cellular respiration. 12. Evaluate the importance of organic molecules to living systems and their importance to human nutrition. 13. Use the laws of probability and a Punnet square to solve inheritance problems. 37

Student Learning Outcomes, as Assessed by Examinations 38

LEARNING OUTCOME EXAM 1 SCIENTIFIC X CHEMISTRY LANGUAGE X ORGANIZATION X EXAM 2 EXAM 3 EXAM 4 EXAM 5 X X X X EXAM 8 EXAM 9 EXAM 10 EXAM 11 EXAM 12 X X X X X MENDEL X X X X MITOSIS HOMEOSTASIS EXAM 7 X CELLS STRUCTURE EXAM 6 X X DIVERSITY EVOLUTION X X X X 39 KNOWLEDGE X X X

Examination Scores Online On-Campus Difference Exam 1 207 212 5 Exam 2 164 168 4 Exam 3 205 203 -2 Exam 4 171 170 -1 Exam 5 168 167 -1 Exam 6 134 128 -6* Exam 7 179 180 1 Exam 8 167 0 Exam 9 209 197 -12* Exam 10 213 207 -6 Exam 11 171 167 -4 Exam 12 224 209 -15* * Difference is statistically significant [p <. 05]. 40

Exam Data Analysis 9 of the 12 exams test showed no significant differences. 3 did show a difference: l Molecular genetics l Topic 2 l Topic 3 41

Data Analysis Student Learning Outcomes, As Assessed by Laboratories 42

LEARNING OUTCOME LAB LAB LAB 1 2 3 4 5 6 7 8 9 10 LAB 11 LAB 12 SCIENTIFIC DATA COLLECTION CHEMISTRY X X X CELL STRUCTURE X X MITOSIS/MEIOSI S X X GENETICS X NATURAL SELECTION X X DIVERSITY X ANATOMY PHOTOSYNTHES IS NUTRITION INHERITANCE X X MICROSCOPE OSMOSIS X X X x X X 43 X

Lab Scores Online On-Campus Difference Lab 1 97 97 0 Lab 2 94. 8 93. 9 -0. 9 Lab 3 88 98. 9 10. 9* Lab 4 98 94 -4* Lab 5 86 97 11* Lab 6 92 98 6* Lab 7 79 80 1 Lab 8 95 96 1 Lab 9 95 94 -1 Lab 10 90 92 2 Lab 11 84 87. 9 3. 9 Lab 12 97 99 2 * Difference is statistically significant [p <. 05]. 44

Laboratory Data Analysis 8 of the 12 exams test showed no significant differences. 4 did show a difference: l Osmosis and Diffusion l Cellular Respiration l Cells l Genetic Diversity 45

Data Analysis Student Learning Outcomes, As Assessed by Field Trip, Reading Assignments, and Total Points 46

Scientific Article Review Scores Online On-Campus Difference* Article 1 48. 7 49 Article 2 97 97 . 3 0 Article 3 48 49 1 Article 4 49 48 -1 * None of the differences are statistically significant [p <. 05]. 47

Field Trip, Reading & Total Points Data Analysis No statistically significant differences: l Field Trip Assignment l Reading Assignments l Total Points 48

Data Analysis: Significant Differences Examination data showed a significant difference between the performances of students on some examinations, however, other measures of the same student learning outcomes were not significant. 49

LEARNING OUTCOME EXAM 1 SCIENTIFIC X CHEMISTRY LANGUAGE X ORGANIZATION X EXAM 2 EXAM 3 EXAM 4 EXAM 5 X X X X EXAM 8 X MENDEL X X X X X EXAM 11 EXAM 12 X X X X EVOLUTION X EXAM 10 X DIVERSITY KNOWLEDGE EXAM 9 X MITOSIS HOMEOSTASIS EXAM 7 X CELLS STRUCTURE EXAM 6 X X X 50

Data Analysis: Significant Differences l Four of the laboratories with similar student learning objectives do show significant differences. l Molecular and organismal genetics are some of the most difficult of biological study. l Further research may be necessary to evaluate the equivalency of the courses in terms of these specific topics. 51

LEARNING OUTCOME LAB 1 LAB 2 LAB 3 LAB 4 X X LAB 5 LAB 6 LAB 7 X X LAB 8 LAB 9 LAB 10 LAB 11 X X LAB 12 SCIENTIFIC DATA COLLECTION CHEMISTRY X X MICROSCOPE X CELL STRUCTURE X OSMOSIS X MITOSIS/MEIOSIS X X GENETICS X NATURAL SELECTION X X DIVERSITY X ANATOMY PHOTOSYNTHESI S NUTRITION INHERITANCE X X x X X X 52

Conclusions l. A majority of the data analyzed by this research indicate no statistically significant differences in completing student learning outcomes in online and on-campus courses in non-majors biology at Mt. San Jacinto College. l Some SLOs may have been more difficult to achieve online than in person. 53

Implications for the Classroom l New labs being tested on topics that showed significant differences. 54

Implications for SLO Assessment l Letter grades and the performance assessment used to calculate achievement do not necessarily measure individual student learning outcomes. l New assessments that accurately measure performance on individual SLOs need to be developed, using clearly defined, detailed rubrics and a variety of evaluation and assessment techniques. 55

SLO Evaluation: Examinations l 40 - 50 questions on each exam l 10 – 12 SLO categories per exam l Students may take exam up to 3 times l Each time different exam 56

SLO Evaluation: Labs l Experimental procedures l Variety of outcomes l Synergistic results 57

SLO Evaluation: Reading Assignments 4 different volumes l Articles cover different aspects of SLO l Synergistic results l 58

Implications for Future Research assumptions: l no significant differences in the preparation or previous experience of students with biology in both groups. l no significant differences between the two groups in academic ability. l random distribution of academic ability and interest among and between the two sample groups. Analysis of these data was beyond the scope of this research but may be helpful in evaluating the differences observed. 59

Things aren’t always as they seem! 60

Questions & Answers 61

Relationship between Faculty and IR Challenges Opportunities l Many faculty are not l Researchers have trained in research expertise in research process l Institutional Research on learning = Institutional Priorities results in rich and valid evidence l Research is time consuming l Evidence-based decision making 62

For More Information Visit the @ONE website at www. cccone. org for full research monographs, a PDF summary of the nine research projects, and streaming presentations by the researchers. 63

Questions & Answers 64

Thank You!! 65