ONLINE ASSESSMENT Prof Er Hui Meng Content Introduction

ONLINE ASSESSMENT Prof Er Hui Meng

Content • Introduction and Context • Assessments, Online Assessments • Development of an Online Assessment System • Purpose, requirement features • IMU OAS Features • Screenshots of reports • User Feedback

Introduction and Context

The Many Uses of Assessment for Educational Management Grade Quality assurance Feedback ASSESSMENT Improve Student learning Determine competency Formative

Quality assurance of assessment programmes • Internal QA • Process of reviewing evidence within an institution for purpose of assuring stakeholders of quality • Requires evidence of evaluation (design and quality of assessments) • Evidence is used for external QA, audit and accreditation • Be prepared formal challenges or appeals relating to assessment processes or outcomes

• External quality assurance • Process of checking that the provision of teaching, learning and assessment meets agreed standards • Part of accreditation • Requires evidence of evaluation

QA assessment evidences • • QA feedback loop Assessment strategies (tools, timings and rationale) Assessment blueprint Guidance, support, assessment load Validity and reliability Innovation/expertise/training Quality and reliability of marking: standardisation/calibration, moderation • Assessment regulations, reasonable adjustments, mitigating circumstances, resits

Evidence : Item review process Item analysis Item authoring Review /Vetting committee Test administration Info to users Item bank Pre-test review Post-test review

Five criteria for Assessment (Schuwirth and Van der Vleuten, 2003) 1. reliability (the degree to which the measurement is accurate and reproducible) 2. validity (whether the assessment measures what it claims to measure) 3. impact on future learning and practice 4. acceptability to learners and faculty 5. costs (to the individual trainee, the institution, and society at large).

Quality of Assessments Matters Most. Why? • Established to meet healthcare needs of country • Established by educationists • Competitive market: need to attract quality students/ staff • Sustain and grow international partnerships • Recognition of Expertise in HPE • Making a positive difference in the region

All going well with Assessments? • Yes • • Various assessment tools Blueprinting Formative and Summative balance Assessment framework as guideline • No • From one programme to over 15 programmes with increase in student numbers • Workload for exam office and academic staff • Venues for assessment • Unsure • Individual Feedback: timely and personalised • Exam office /Faculty expertise in assessment • Administrative issues: audit trail, turnaround time for result processing • Quality assurance including assessment blueprinting, post-test item analysis • Question bank

Questions for reflection • To what extent are our learning outcomes being assessed? • How well do we know our assessment tools? • Do we have the right resources? • Who are involved ? • How do we provide timely and personalised feedback to enhance performance? • How reliable is our evidence for QA?

AN URGENT NEED TO SUPPORT AND ENHANCE ASSESSMENTS: ONLINE ASSESSMENTS

DEVELOPMENT OF AN ONLINE ASSESSMENT SYSTEM

EXAMPLES OF ONLINE ASSESSMENTS • Admission Test • Psychometric tests for graduates and job applicants • Online job interviews WHAT WOULD SUIT OUR PURPOSE?

Admission test • Eg. GMAT • Use in admission to a graduate management program eg. MBA • Tests verbal, numerical, analytical and written skills • Computer adaptive test, where the difficulty of each question will be determined by the answer given to a previous question • An unofficial score is available straight after the test.

Psychometric test for graduates and job applicants • Used as an initial screening method for employers. • Typically consist of numerical reasoning, verbal reasoning and diagrammatic reasoning tests. • Situational judgement tests are increasing in popularity. • Work with intelligent learning system: • • Developed with the help of occupational psychologists Clear worked solutions provided Review historical test performance and progress Identify weak areas in order to increase preparation efficiency • Review performance against others

Online job interview • On-Your-Own Online Job Interviews: employer emails the candidate a list of interview questions and ask the candidate to film himself answering them. • Use technology to conduct online interviews, simple and effective way to interview candidates for employment • Webcam Job Interviews using video conferencing software like Skype or Google Hangouts. • Online job interview practice using interview simulator • Simulates a mock interview with over 100 questions organized in categories and sets. • Features audio prompts along with text-based tips for responding to each question and examples.

Purpose of Online Assessments • Provide QA data – vetting, feedback processes (purpose) • Identify knowledge gap to prepare candidates before entering a programme (purpose) • Compile data to help decision making on progression point during residency training (purpose) • Study trend and pattern of student performance (accessed for various stakeholders) (purpose) • Conduct objective assessments that are hassle free and regular (purpose) • Standardise assessments across centres (feature) • Programme evaluation (purpose) • Score moderation (feature) • Easy accessibility for all assessees and assessors (feature) • User interface (feature)

Online Assessment Requirements Pre Assessment During Assessment Post Assessment

Online Assessment System (OAS) requirements An online assessment system that is reliable, secure and valid - cater for a range of assessment tools (listed above) , items can include images or video clips. Allows storage of items in a secure question bank that is searchable by keywords- items can be imported and exported easily to other common word processing softwares (eg Microsoft Word, Excel). Allows examinations to be constructed according to learning outcomes, difficulty levels and matrix, and can have filter view A compatible system with the university’s governance system for assessment, which includes submission of questions, vetting processes, examination construction, standard setting and post-item analysis (psychometric analysis) for QA purpose.

OAS requirements Online institution of tests, where questions and options (for MCQs) can be reshuffled Accessible on and off-campus with security features, and allows online grading according to assessment rubrics/checklists and multiple grading (ie double marking), which will reduce manual processing and transcription errors. Generates comprehensive performance reports (including longitudinal performance) for examination coordinators for QA purpose and to inform the T-L strategies Provides personal performance and immediate feedback to students. It has an alert system to identify weak performers via continuous/formative assessments to facilitate early remediation. An efficient and secure data management process that can integrate with Student Lifecycle System

IMU OAS Features

Screen shots of reports generated by IMU OAS

Class Performance Report

OBA/EMQ – Class Responses

Item Analysis Data

Question Metadata Report Performance by learning outcomes Performance by difficulty level

Performance by Bloom’s Classification

Individual Student Performance

SAQ/Essay/OSCE Report

OSCE Marking Rubric

Individual student performance in OSCE - Student view

Implementation of OAS: The lessons we learnt Successful implementation of OAS depends on: 1. System design 2. Functionality of features 3. People Management: Faculty, Professional Staff and Vendor 4. The B&A Implementation Checklist

User feedback

Academic • Reports generated post examinations as well as individual feedback report for students are useful. • Psychometric analysis provides useful data on the quality of assessment • Ease of preparation of examinations: lecturers key in questions to OAS directly, module/exam coordinators prepares exam papers in OAS, and hence OAS acts as a question bank • Potential use of OAS to support personalised learning • Students appreciate the advanced features in the system and its user friendliness (feedback from pharmacy students)

Exam Office • Effective tracking of examination question vetting processes • Adherence to timelines for preparation for examination papers • Automated results processing reduces transcription errors • Manageable record keeping • Reduced results processing time