Briefing session on Online Student Evaluation of Teaching

Briefing session on Online Student Evaluation of Teaching and Learning Prepared by Social Sciences Research Centre of The University of Hong Kong 1 2018

SETL process Ø Online process covers all courses with undergraduate and TPG students (including Common Core courses) on 2 semesters system or summer course with teacher data in SIS. Ø Non-semester courses can be evaluated on-line if the course information and end date have been input into SIS, otherwise use paper. The results will follow the 2 semesters system. (If complete by Dec 7, 2018 results included in the first semester. If complete by May 5, 2019 , results included in the second semester. ) 2

SETL process Ø For RPG only courses, the departments/centres/schools /institutions can choose to use paper or online Ø For the summer courses, the results will be merged into the first Semester of the following academic year Ø The online SETL evaluation conducted for 4 weeks until the start of the assessment period (i. e. for first semester courses, from Nov 10 to Dec 7, 2018) 2018. 3

SETL process Ø Separate SETL forms for courses, teachers, tutors, demonstrators and practicum, practicum common core courses using simpler version. Ø Uses SIS database record of course/teacher/tutor/demonstrator (so these must be input accurately into the SIS) Ø SIS allows joining different courses of the same level in the evaluation, only one overall report will be generated for the parent course to the core department Ø Courses of different level will always be assessed separately, even if using the same course code 4 Ø MBBS and Dentistry online SETL evaluation are handled separately

SETL process Ø Multiple teachers refer to a number of teachers teach the same student in the same course code and in the same classroom Ø Sub-group identifiers refer to a number of teachers teach different students in the same course code in different classrooms Ø Each course will be evaluated only once before the end of the course e. g. a two-semester course will be evaluated before the end of the second semester 5

SETL process Ø Must use standard forms unless Vice-President (TLQC) approves Ø Online yields lower response rates, but better open-ended responses, responses which is good for improvement Ø To ensure good response rates, need to ensure that students are given time at the end of a class to complete the form (although they can complete any time before the deadline) 6

SETL process Ø Cannot currently change teacher names in system, which come from Human Resources Ø All teachers/tutors/demonstrators must have staff ID/University No. (HKU requirement and needed for staff review) Ø For the Department Items, a maximum 5 extra questions for both on-line and paper evaluation, please contact SSRC through email setl@ssrc. hku. hk (on or before 20 October, 2018 for this semester) 7

SETL process Ø For the online SETL, all courses with enrollment of at least 2 students of the same level must undertake SETL, although the teacher/tutor/demonstrator parts are only done if there at least 2 students per teacher/tutor/demonstrator Ø For the paper SETL, all courses with enrollment of at least 5 students must undertake the evaluation, although the teacher/tutor/demonstrator parts are only done if there are such persons for the class 8

SETL process Ø Up to 25 teachers can be evaluated in each course by a student, but only 1 tutor/demonstrator selected by a student from a list of up to 25 names (or evaluated as a group) of each course Ø Tutor means person running tutorials, demonstrator means person responsible for lab or field sessions (role descriptions, not job titles) 9

SETL Workflow Prepared by Social Sciences Research Centre of The University of Hong Kong 10

SETL Workflow of the Online SETL Ø Departments/Centres/Schools/Institutions must enter/update the course enrollment information in the Student Information System (SIS) especially the staff number linked to the name of teacher/tutor/demonstrator in each course and sub-class information 10 weeks before the assessment date (i. e. before end of September for this semester and finish date for nonsemester courses) Ø All information must be updated for the appropriate semester 11 Ø Student enrollment should be updated automatically for courses using SIS for enrollment, but the rest of the information relies on departments/centres/schools to update

SETL Workflow of the Online SETL Ø SIS system exports the course enrollment information of all undergraduate, TPG and RPG courses for which complete SIS data is available including: ü Course code; ü Course title; ü Term (Semester 1/2/full Year) ü Assessment period start date if not semester based ü Student list in each course (student name and university no. ) by course level; ü Teacher name list in each course (teacher name and staff number); ü Tutor and Demonstrator name list in each course (teacher name and staff number); and ü Jointly evaluated courses (course list) 12

SETL Workflow of the Online SETL ü The course/teacher/tutor/demonstrator lists and indication of whether online or paper process will be sent out to departments/centres/schools/institutions on or before 12 October 2018 (for this semester), so departments/centres/schools/institutions can check for errors and missing information. ü Departments/Centres/Schools/Institutions should update any amendments in the SIS system between 12 and 26 October 2018 (for this semester). 13

SETL Workflow of the Online SETL Ø The final course/teacher/tutor/demonstrator lists and indication of whether online or paper process will be sent out to departments/centres/schools/institutions (i. e. on or before 2 November 2018 for this semester). If any errors occurs, please contact SSRC through email setl@ssrc. hku. hk immediately (i. e. on or before 5 November 2018 for this semester). Ø For information, SSRC will send out the invitation emails to all HKU students on 10 November and 2 reminder emails to all HKU students on 30 November and 4 December, 2018 for this semester 14

SETL Workflow of the Online SETL 15 Ø Online process will cover all courses with undergraduate and TPG students (including Common Core courses) on 2 semesters system or summer course with teacher data in SIS. Ø Non-semester courses can be evaluated on-line if the course information with an end date later than the end of first semester and the end date have been input into SIS, otherwise use paper. Ø For RPG only courses, the departments/centres/schools /institutions can choose to use paper or online Ø For the summer courses, the results will be merged into the first Semester of the following academic year

SETL Workflow of the Online SETL Ø Non-semester ü Non-semester courses must input dates through SIS system so they can be evaluated on-line and then the course/teacher/tutor/demonstrator lists will be sent out to departments/centres/schools/institutions 2 weeks before the evaluation date (ONLY ONE circulation will be sent). If any errors are found, departments/centres/schools/institutions must respond within one week after receiving the lists from SSRC and please contact SSRC through email setl@ssrc. hku. hk immediately. 16

Instructions for Online SETL Before administration The teacher concerned should inform the students in advance the scheduled time set aside for the online evaluation (i. e. soon after 10 November 2018 for this semester), and that the students are encouraged to bring a mobile internet device (such as laptop, tablet or smartphone) smartphone to class at the scheduled time. 17

Instructions for Online SETL On the day of administration Similar to the paper exercise, the SETL questionnaire should NOT be administered by the course teacher, but faculty/department staff. The teacher should briefly explain the objectives of the student evaluation and then leave the room leaving it to faculty/department staff to oversee the process. 18

Instructions for Online SETL On the day of administration Faculty/Centre/Institution/Department staff should read out: Please go to the weblink http: //setl. hku. hk where you should see a list of all forms that you need to complete. Your evaluation will be saved anonymously, without any identification. There are separate forms for the course, teacher (and if appropriate) tutor and demonstrator. There are instructions, an FAQ link and a link for you to report any missing courses at the weblink. 19

Instructions for Online SETL During the evaluation period Students should ü ü ü 20 Access the SETL system via the weblink http: //setl. hku. hk/ where there are instructions, FAQ and a list of forms for completion (separate links for the course, teacher, tutor, demonstrator forms) Complete the forms for the class during four weeks until the start of the assessment period (this semester from Nov 10 to Dec 7, 2018) If they cannot find the course that they have enrolled in, send an email to SSRC using the link on the page so SSRC staff can rectify this promptly

Workflow for Paper SETL Ø Departments/Centres/Schools/Institutions are responsible for ensuring accuracy of SIS data before SSRC generates the SETL reports and inform SSRC about the course enrollment information for printing purpose. Ø Departments/Centres/Schools/Institutions should inform SSRC for printing at least 2 weeks before the evaluation date. Departments collects the blank forms one week before the evaluation date and carries out the evaluation. Ø For the paper evaluation forms, departments should deliver to SSRC all completed forms, completed cover sheet on the top of each envelope. 21

SETL Workflow for Reporting 22 Ø The SETL results will be generated upon notice from SSRC after the examination result release date and the following report will be provided to each department/centre/school/faculty and the reports will be saved in MS Excel format (worksheets) : ü an individual report of every teacher for each course; ü an overall teacher report for every course; ü a report for the tutor/demonstrator for every course (if applicable); ü a course evaluation report for every course; ü a soft copy of all open-ended questions’ answer for each course using online SETL (in MS excel format by question number);

SETL Workflow for Reporting Ø The overall SETL report will be provided to each department/centre/school/faculty and the reports will be saved in MS Excel format (worksheets) : ü an overall report on the undergraduate courses; ü an overall report on the TPG courses; and ü an overall report on the RPG courses 23

SETL Workflow for Reporting Ø Additional requests may be entertained at cost Ø SSRC will upload SETL results to the APA for teachers and courses on or before 31 January 2019 for this semester (after the examination result release date 17 January 2019) Ø SSRC will upload ALL SETL results to the SSRC online SETL report system on or before 31 January 2019 for this semester (after the examination result release date 17 January 2019) Ø SSRC will upload SETL yearly course averages to SIS at the end of the year Ø SETL results cannot be changed or removed from the database unless with strong justifications that are approved by VP/T&L, on the recommendation of the Head of Department 24

Instructions for Online SETL ü I will now demonstrate Online SETL, so you can see how the process works in practice: http: //setl. hku. hk ü I will now demonstrate Online SETL report system, so you can see how the process works in practice: http: //setl. hku. hk/report 25

Instructions for Online SETL The SSRC Guide (Third Edition, April 2015) Briefing session on Online Student Evaluation of Teaching and Learning (15 October 2018)and SETL Online DEMO are available: http: //www. ssrc. hku. hk/setl. html 26

Using SETL data effectively SETL data is used to evaluate all courses and can facilitate comparisons across courses and time. It is only one component of the QA process, but it is our focus for today. SETL results can assist: 27 l ensuring courses are well designed and being delivered well l improving course design l improving teaching l reassure students that concerns they have about courses and teaching are being addressed

SETL context (I) Q: How many students in the class? A: If very small, then the results may be less reliable, but the teacher should have been able to communicate better with the class) Q: How many responded (what is the response rate)? A: A low response rate may mean less valid results due to bias. This may reflect any of: a problem with administration of the forms in class (e. g. no time set aside) students did not turn up to class (i. e. serious dropout rate), students do not see this course as important enough to respond, students do not believe the programme team are responsive to their concerns. All of these are problems in different ways. Q: Is this an award bearing course? 28 A: if yes, students will normally have higher expectations

SETL context (II) Q: Part-time or full-time course? A: Part-time students are often more motivated and may be more tolerant of teacher weakness. Q: New course? A: If yes, essential to check that the course design is appropriate, else the evaluation may be poor even if the teacher is relatively effective. Q: New teacher? C: If yes, the teacher may not understand the needs of the students well yet and if inexperienced, may need additional support initially to teach more effectively Q: Service course or required course? 29 C: Students may be more demanding about a required course, because they recognize its importance, but may be negative about a service course that they do not believe addresses a real need.

Interpret results The overall questions are the primary focus for administrators: Overall, the course (or teacher) was effective in helping me achieve the course learning outcomes The report shows the % who strongly disagreed, neutral, agreed or strongly agreed and converts them into mean, standard deviation (SD), standard error (SE) How can we interpret the mean and SE? 30 Responses are scored from 100 (strongly agree) down to 0 (strongly disagree), so a mean of 50 would mean students were equally positive and negative about the course or teacher and a mean of 75 would mean that the average response was agree that the course/teacher was effective. SE gives an indication of how much variation is likely due to chance (taking into account variability across students and the number of responses). With a mean of 50 from 64 students, SE can be anywhere from 6 (half the students rated 0 and half rated 100) down to 0 (all rated 50). SE can be interpreted as that by chance, the mean will vary by about +/- 2 x SE, so we can safely ignore differences of less than 2 x SE.

Reference To decide whether the mean is good or poor, we need a reference, which is normally the average score for that department in that semester, so a course with mean score lower than the average – 2 x SE should perhaps be checked more carefully. Alternatively, we could use a criterion reference. The difficulty is deciding what that criterion reference should be – it could be 50 for tenure track and 70 for others. In any event, it is important to identify remedies for low scores. 31

Improvement My personal experience is that the open-ended comments on aspects that can be improved is the most useful element for identifying things I can improve, even if some of the comments cannot be addressed by me (e. g. scheduling) or reflect unreasonable expectations (teach advanced materials in a short course). It is important to realise that students may not know how your course or teaching can be improved. In my case, my scores (for teaching graduate students) improved most when I started providing screen videos after each class (so they can rewatch if they did not understand what I said) and when I started offering personal feedback on assignments during class breaks (in case they did not understand the written feedback), neither of which had ever been requested by students. 32

Transparency with students is a good thing for course effectiveness, reassuring students that action is taken when problems are identified. HKU now shows the average yearly course effectiveness for each of the last 3 years for all courses, together with feedback about any actions taken to address problems. I believe this will be much more effective than the previous process which, for most departments in HKU, involved sharing only through staff-student consultative committees. However, for teacher effectiveness feedback, I see it best not being made public, especially as some students may write abusive comments about individual teachers, but it is essential for improvement and mentoring. 33 We have an implicit contract with our students in that their role is to provide honest feedback to assist teachers in continuous quality improvement.

Thank you! I am happy to answer questions. Prepared by Social Sciences Research Centre of The University of Hong Kong 34