MEASURING TEACHING PRACTICES Reformed Teaching Observation Protocol 1

MEASURING TEACHING PRACTICES Reformed Teaching Observation Protocol 1 • Describes teaching process on five subscales • • • Lesson Design by instructor Propositional knowledge of instructor Procedural Knowledge (what students do) Student-Student Interactions Student-Teacher Interactions • Five items per subscale • Each item scored 0 (never occurred) to 4 (very descriptive of the class • Maximum Score = 100 (25 x 4) • Typical scores range from 20 -80 1 Sawada et al. , 2002 Observer perspective

MEASURING TEACHING PRACTICES Reformed Teaching Observation Protocol 1 18) There was a high proportion of student talk and a significant amount of it occurred between and among students (quantity of interactions) No student talk 0 Students talk to each other at least once (about lesson content) 1 Student-student talk occurs at least 10% of the time during the course of the class 2 Student-student talk occurs more than 25% of the time during the course of the class 3 In any given moment during the lesson, students are more likely to be talking to each other than the teacher (>50% student to student) 4 • Supplemental rubric created by Budd et al. • Greater inter-rater reliability per item. • Higher scoring classrooms feature more active learning practices. 1 Sawada et al. , 2002; Budd et al. , 2013

Variations in Teaching Practices Teaching Geoscience at US Higher Education Institutions On the Cutting Edge has been using the Reformed Teaching Observation Protocol (RTOP) to measure teaching practices of more than 200 instructors in a variety of undergraduate geoscience classes at a range of US colleges and universities. Here are some of the things we have discovered.

Variations in Teaching Practices Teaching Geoscience at US Higher Education Institutions

RTOP Score vs. Sub-Categories Lesson Design 100 Propositional Knowledge 100 80 80 80 60 60 60 40 40 40 20 R 2 = 0, 8302 20 R 2 = 0, 3078 0 0 0 10 20 20 R 2 = 0, 8479 0 0 10 Student-Student Interactions 100 Procedural Knowledge 100 20 30 80 60 60 40 40 10 20 Student-Teacher Interactions 100 80 0 20 20 R 2 = 0, 8451 0 0 5 10 15 20 R 2 = 0, 8821 0 0 5 10 15 20

Class Size: Total Scores 100 90 80 70 60 50 40 30 20 10 0 89 78 69 38, 5 38 14 13 Small 34 14 Medium Large Small - Medium Large 0. 05 0. 83 0. 10 - Minimum Maximum Median First Quartile Second Quartile Third Quartile Fourth Quartile Medium 14 78 38. 5 26 38. 5 53 78 Large 13 89 38 31 38 47. 5 89 14 69 34 22 34 45 69

Course Level: Total Scores 100 90 80 70 60 50 40 30 20 10 0 89 78 37 16 13 Intro Majors 37 0. 39 Majors Label Minimum Maximum Median First Quartile Second Quartile Third Quartile Fourth Quartile Intro Total Score Majors Total Score 13 89 37 28. 25 37 47 89 16 78 37 26 37 53 78

Institution Type: Total Scores 100 90 80 70 60 50 40 30 20 10 0 89 78 71 43, 5 39, 5 36 20 Associate 17 Baccalaureate 2 yc - bacc - master doctoral 0. 85 0. 77 0. 15 0. 98 0. 49 0. 10 - Minimum Maximum Median First Quartile Second Quartile Third Quartile Fourth Quartile 35 14 13 Masters Associate 2 yc bacc master doctoral 84 Research/Doctoral Baccalaureate 20 71 43. 5 32 43. 5 52 71 17 89 36 31. 25 36 44 89 Masters Research/Doctoral 13 78 39. 5 32. 25 39. 5 49. 5 78 14 84 35 25 35 47 84

OTCE PARTICIPATION VS. RTOP SCORES How did participation in On the Cutting Edge events and use of web resources impact teaching practice? RTOP Scores Cutting Edge Participation: Total RTOP Scores 100 90 80 70 60 50 40 30 20 10 0 89 84 74 47 31 13 Neither 35 14 Website only 17 Both

OBSERVED TEACHING PRACTICES No/few questions asked by instructor No/few questions from students Students are passive/not asked to do anything No student-student interaction/ conversation Student-student interactions or group work Students read graphs, maps, use data Students answer open-ended questions Instructor assesses students (new or prior knowledge) Lesson adjustments based on student work or prior knowledge Most Traditional Lecture n=10 Mean Transitional Lecture n=22 Mean Active Learning n=12 Most Active Learning n=11 50% 60% 27% 36% 0% 9% 0% 0% 70% 36% 4% 0% 0% 70% 80% 32% 0% 0% 0% 9% 59% 100% 91% 20% 0% 27% 4% 67% 17% 45% 10% 18% 33% 45% 0% 0% 9% 33% 54%

EXAMPLE CLASS BREAKDOWNS RTOP = 32, Transitional 0 Minutes 8 55 lecture Activity 1 65 Activity 2 Activity 1: Present LOs, review with 3 conceptest questions Activity 2: Group Venn diagram activity with shout review RTOP = 51, Student-Centered Minutes Activity 1 30 20 7 0 Lecture 1 Lecture 2 Activity 2 37 Lecture 3 70 54 45 Activity 3 Lecture 4 Activity 1: Present LOs and Brainstorming Think-Pair-Share Activity 2: Group interpretive drawing activity with class discussion on implications Activity 3: Activity connecting prior and common knowledge to content Activity 4: Reflection, students summarize 3 most important concepts 75 Act. 4

In. Te. Grate Materials Development Process Teams of authors spent 1 -2 years creating new resources for their courses. They received instruction on pedagogy through workshops and webinars and feedback about their lessons from a team of coaches and assessment specialists. Did it have an impact on their teaching practice?

Teaching Practice Data (Reformed Teaching Observation Protocol; RTOP) 80 Student-centered 70 (83%) RTOP Score 60 56. 2 50 41. 5 40 30 20 10 0 Minerals 1 2 3 4 5 Climate 6 7 8 9 Budd et al. , 2013 Average RTOP score 41. 5 EJ 10 11 Soils 12 13 14 Instructor/Unit Living on the Edge 15 16 17 18 19 20 21 22 23 In. Te. Grate pilot lessons Average RTOP score 56. 2

Conclusions: 1. RTOP data identifies 3 classroom types: Traditional Lecture, Active Lecture (Transitional), Active Learning 2. In. Te. Grate lessons taught by authors correlate with increased student engagement as measured by RTOP 3. Moving from less to more student engagement in class: • Increased use of questioning strategies for student input • • • Increased use of students working together during class Increased discussion between faculty and students Increased assessment (and adjustment) to student learning An Active Learning class is very likely to have some traditional lecturing, along with having students participate in a brief group discussion, think-pair-share or class exercise during every class period.

Position Type: Total Scores 100 90 80 70 60 50 40 30 20 10 0 89 84 78 71 62 42 42, 5 37 34, 5 13 14 14 31 20 17 Adjunct Assoc Full Adjunct Assoc Full Inst Asst 0. 05 Assoc 0. 04 0. 87 Full 0. 21 0. 18 - Inst 0. 98 0. 02 0. 01 - 0. 06 - Minimum Maximum Median First Quartile Second Quartile Third Quartile Fourth Quartile Asst 20 62 31 23. 5 31 44 62 Inst/Lect Assoc 17 84 42 32. 5 42 55. 5 84 Full 13 89 42. 5 31 42. 5 53. 75 89 Inst/Lect 14 78 37 26 37 50 78 14 71 34. 5 24. 5 34. 5 41. 75 71