Pedagogical and Assessment Designs for Online Learning Barbara

Pedagogical and Assessment Designs for Online Learning Barbara Means Center for Technology in Learning SRI International Keynote presentation for the CITE Research Symposium E-Learning Designs & Designs for Learning March 6, 2010 Hong Kong © 2007 SRI International

Outline of Talk § Emergence of online learning at secondary level in U. S. § Variations in online learning designs § Research synthesis on the effectiveness of online learning § Role of formative assessment in enhancing learning outcomes for blended instruction © 2007 SRI International 2

Growing Importance of Online Learning § Long a part of corporate and military training § Online learning has become a big part of higher education and of self-directed adult learning § Now it is becoming part of K-12 education (mostly at the secondary level) © 2007 SRI International

K-12 adoption of online learning rising in U. S. Enrollments up from 45, 000 in 2000 to 1, 000 in 2007 Online Learning Enrollment 0. 08 0. 07 0. 06 0. 05 0. 04 Online Learning Enrollment 0. 03 0. 02 0. 01 0 1999 2001 2003 2005 2007 2009 Source: Estimates of secondary school online enrollments from Picciano & Seaman, 2009 Copyright Clayton M. Christensen

K-12 adoption of online learning rising in U. S. Enrollments up from 45, 000 in 2000 to 1, 000 in 2007 0. 10 0. 09 0. 08 0. 07 0. 06 0. 05 0. 04 0. 03 0. 02 Online Learning Enrollment 0. 01 0 1999 2001 2003 2005 2007 2009 2010 2012 2014 Source: Estimates of secondary school online enrollments from Picciano & Seaman, 2009 Copyright Clayton M. Christensen

K-12 adoption of online learning rising in U. S. 0. 10 0. 09 0. 08 0. 07 0. 06 0. 05 0. 04 0. 03 0. 02 Online Learning Enrollment 0. 01 0 1999 2001 2003 2005 2007 2009 2010 2012 2014 Clayton Christensen predicts that by 2019 half of all high school courses will be taken online. Copyright Clayton M. Christensen

U. S. districts and states are starting to encourage online learning § Most school districts (75% in 2007) provide online learning options for at least some of their students § Several state-funded programs offer online high school courses (Florida, Illinois, Michigan, Kentucky) § 24 of 50 states now have statewide full-time online schools (“cyberschools”) § Two states (Michigan and Alabama) require students to take an online course in order to earn a high school diploma § One state (Florida) requires all of its districts to make online learning options available to their students © 2007 SRI International 7

Typical Reasons for Offering Online Course Options § Provide a course not otherwise available § Provide more college-level (Advanced Placement) course options § Give students a second chance through “credit recovery” programs © 2007 SRI International 8

Different Forms of Online Learning § Fully online v. hybrid or “blended” § Module within a course, supplemental formal course, degree program § Scheduled course (“class-linked”) v. self-paced § Instructor-developed v. professionally developed © 2007 SRI International

Instructional Design Elements in Online Learning § Communication type - Synchronous - Asynchronous § Learner control § Instructional approach - Expository - Active - Interactive © 2007 SRI International

Expository Learning Environments Present content to the learner © 2007 SRI International 11

Active Learning Designs Support the practice of skills and elicit student responses Carnegie Learning’s

Interactive Learning Designs Involve collaboration with other learners or an instructor with the focus of engagement emerging over time Political Science 105 from University of North Carolina Greensboro © 2007 SRI International 13

SRI’s Meta-Analysis of Online Learning Research § Systematic search for studies from 1996 through July 2008 on the effectiveness of online learning § Limited to studies that - compared an e-learning or blended condition to face-to-face instruction measured learning objectively with the same measure for treatment and control groups - used an experimental or quasiexperimental design with control for any pre-existing differences between groups - provided the statistical information needed to compute an effect size - § Out of 1, 132 articles reviewed, 51 study effects met the requirements for meta-analysis © 2007 SRI International

Meta-Analysis Findings § Average effect size of +0. 24 favoring the online condition § Advantage over face-to-face instruction was larger for studies using blended approaches (E. S. = +0. 35) than for studies using pure elearning (E. S. = +0. 14) § Advantage over face-to-face instruction was larger for studies where - online students spent more time learning than did those in the face-toface class - the online and face-to-face conditions varied in terms of content and instructional approach Taken as a whole, the findings suggest that the observed advantage of online learning is a product of redesigning the learning experience, not of the medium per se. © 2007 SRI International

SRI’s Review of Studies Comparing Different Online Learning Designs Online learning usually more effective when § - it stimulates more active engagement with the content - it includes prompts for learner reflection learners have an element of control over their interactions with the software - § Some common practices usually had no effect adding additional media not related to the content to be learned - - adding multiple-choice quizzes Results were inconclusive with respect to different instructional approaches § www 2. ed. gov/rschstat/eval/tech/evidence-based-practices/finalreport. pdf © 2007 SRI International

Conjectures on the Blended Advantage § More learning time § More interactivity § Selective

The Concept of Formative Assessment § Assessment for learning rather than assessment of learning § Occurs during the course of learning while there is still time to improve outcomes § A quality of the way in which assessment information is used, not inherent in the test itself § One of the most powerful levers for improving learning outcomes (Black & Wiliam, 1998) © 2007 SRI International

e-Learning Offers New Options for Formative Assessment § We can do much more than just interspersing sets of multiple-choice questions into instructional materials § Technology enables automated capture of complex learner behaviors § Assessment can be embedded in the learning in ways that feel natural rather than forced © 2007 SRI International 19

Learning in a Multi-User Virtual Environment Chris Dede’s River City MUVE (Science, 2009) Enlargement

River City § Teaches concepts from biology, ecology and epidemiology § Students work in teams of three, moving through the city to run tests in response to the mayors’ challenge § Teams keep online journals, analyze data, form hypotheses, and write up their research in a report for the mayor § Researchers developed measures of science concept knowledge, science inquiry skills, and sense of efficacy as a scientist based on student actions within the River City environment © 2007 SRI International 21

Such online learning environments can be designed to capture. . . § Where a student went § With whom the student communicated and what was said § The artifacts the student activated § Databases the student used § Data that the student gathered using virtual scientific instruments § Screenshots and notations the student entered in a virtual notebook § Hints that the learner requested All of this information can be used for assessment. © 2007 SRI International 22

Examples from Carnegie-Mellon’s Online Learning Initiative (OLI) 12 web-based colleges courses that can be taken entirely online

Examples from Carnegie-Mellon’s Online Learning Initiative (OLI) § Some of the systems contain virtual

Examples from Carnegie-Mellon’s Online Learning Initiative (OLI) § Others incorporate intelligent tutoring and carefully

The Boxplot The boxplot graphically represents the distribution of a quantitative variable by visually displaying The five number summary and any observation that was classified as a suspected outlier using the 15(IQR) criterion. Here is how a boxplot is constructed (this is for the “Best Actress” dataset – to see the dataset click here. ) To see a static version of this movie, click here. Source: Carnegie-Mellon University, OLI Course in Statistics © 2007 SRI International Use the pull-down menu to label the various points on the boxplot. X That is not quite right. I think you might be confusing Q 1 and Q 3. Recall that Q 1, the first quartile of the distribution, is represented By the bottom edge of the box. Q 3, the third quartile of the 28

Multiple Feedback Loops Source: Thille, 2009 © 2007 SRI International 29

Formative Assessment Is Woven into the Online Learning § Interactive simulations that students can manipulate are woven into the courseware. Students interact with the simulation and then respond to probes that get at their understanding of how the simulation works. § “Did I Get This? ” quizzes follow the presentation of new material so that students can check their understanding without fear of hurting their course grade. § Short essay questions that call on students to make connections between different concepts are embedded throughout the online course material. § “Muddiest Point” requests ask students what they thought was confusing. © 2007 SRI International 30

In Blended OLI Courses Instructors Take Advantage of e. Learning Assessments Each student works

In Blended OLI Courses Instructors Take Advantage of e. Learning Assessments Source: Heffernan & Thille, Carnegie Mellon Online Learning Initiative web presentation The e-learning system aggregates information from all the students who used it. © 2007 SRI International 32

In Blended OLI Courses Instructors Take Advantage of e. Learning Assessments The instructor sees

In Blended OLI Courses Instructors Take Advantage of e. Learning Assessments In class, the instructor focuses on concepts that students struggled with when they worked on line. © 2007 SRI International 34

The OLI Statistics Course Study § Students were assigned at random to take college statistics on line or in a conventional class. § Students in the online version of the class finished the material in half a semester or half the time taken by students in the conventional class. § The two classes took the same final examination, and the online students’ scores were significantly better than those of the students in the conventional class. § No difference between the two groups on a long-term retention test. § Study has been replicated in a community college and a large public university. Source: Lovett, Meyer, & Thille, 2008 © 2007 SRI International 35

Principles for e-Learning Design § Incorporate a range of online activities that engage learners with content. § Give learners options for different ways to learn content and with options for review and support (scaffolding). § Embed assessment within the learning activities and use multiple assessment approaches. § Make assessment feedback available to the student as well as to the instructor. § Use information from students’ online activity to shape the content of face-to-face instruction. © 2007 SRI International 36

Thank you! NIH Media and Children May 15,