Guidelines for Effective Student Assessment 7 Effective assessment

Guidelines for Effective Student Assessment 7. Effective assessment requires feedback to students that emphasize strengths of performance and weaknesses to be corrected. n Helping them develop own self-assessment skills

Definitions of Assessment n Assessment is an ongoing process aimed at understanding and improving student learning n n Multiple methods Criteria and standards Evidence Its more than just collecting data

Assessment of Performance and Competence Authentic assessment

The Assessment Process Formulate statements of intended learning objectives Develop or select assessment measures. Discuss and use assessment results to improve learning. Create experiences leading to outcomes.

Formative (diagnostic) Assessment n Summative Assessment n Criterion Reference n Norm Reference n

Qualities of a Test n n n Validity: is the test measures what it is intended to measure. Reliability: To which extend of consistency the instrument measures a given variable. Objectivity: To which extend independent and competent examiners agree on what constitutes a “good” answer for each test item. Practicability: To which extend an instrument drains resources to construct, implement and score an exam and interpret its results; and how simple it is to use by students and teachers. Discrimination: To which extend the test distinguishes between good and poor students

Instruments in Cognitive domain Source: Bloom, B. S et al. Taxonomy of Educational Objectives, Volume 1: Cognitive Domain. New York: Mc. Graw Hill, 1956. Evaluation Synthesis MCQ Essay Oral Exam Analysis Application Comprehension Recall What can be assess in a Test

What is Authentic Assessment? • The instructor observes the student in the process of working on something real • Provides feedback • Monitors the student's use of the feedback • Adjusts instruction and evaluation

What is authentic assessment? Authentic assessment does not encourage rote learning and passive test-taking. Instead, it focuses on students' analytical skills; ability to integrate what they learn; creativity ability to work collaboratively It values the learning process as much as the finished product.

Professional authenticity A simple model of competence Does Shows how Knows Miller GE. The assessment of clinical skills/competence/performance. Academic Medicine (Supplement) 1990; 65: S 63 -S 7. Performance or hands on assessment Written, Oral or Computer based assessment

climbing the pyramid Does Performance assessment in vivo: Masked SPs, Video, Portfolio…. . Shows how Competence assessment in vitro: OSCE, …. . Knows how (Clinical) Context based tests: MCQ, essay type, oral…. . Knows Factual tests: MCQ, essay type, oral…. .

Assessment Tools

Observational 1 - Check list 2 - Rating scale 3 - Anecdotal record (Tutor reports)

Problems in the use of rating scales n n n n Guilford in 'Psychometric Methods' (1974) lists six common errors made by raters when using rating scales Error of leniency. Error of central tendency Halo effect Logical error Contrast error Proximity errors

Advantages of Tutor reports are particularly useful for assessing personal attributes areas which are difficult to test by conventional methods, such as: n attitudes n generic competencies (information mastery) n professional values (for example, reliability and trustworthiness, ability to work with other members of the health-care team)

Advantages of tutor reports n Well-designed tutor reports allow for assessment against an agreed standard and are useful for identifying the underperforming student and for providing feedback to students n Sustained performance is assessed rather than the snapshot obtained during an examination n one of the few assessment methods that assess performance within the working context and thus have high content validity.

Disadvantages of tutor report problems in the use of rating scales n subjectivity n low reliability n potential for adverse influence on the relationship between tutor and student n

Principles of Giving Feedback* n n n n n Work as an ally of the student. . Have mutual agreement on time and place. Solicit feedback of your own performance. Have the student give an assessment of his or her performance (self feedback) before giving one’s opinion. Base feedback on observed incidents and on modifiable behaviors. Give feedback on specific behaviors, not general performance. Give feedback on decisions and actions, not on one’s interpretations of the student’s motives. Give feedback in small, digestible quantities. Use language that is non-evaluative and nonjudgmental

What is a portfolio? A collection of work that demonstrates competence or achieved outcomes of the learner (student/resident/doctor) n Reflection by learner on progression on competencies/outcomes n “Burden of evidence” is returned from teacher to learner n

Portfolios to assess n n n Student course work and its documentation Student attitudes through selection of material or inclusion in their portfolio. What students include in their portfolio highlights what they consider to be important. The portfolio contents are, therefore a statement of what the student value student learning and progression of learning during the course. through the use of commentaries on students’ learning experiences during the course and annotations of submitted work, teacher can explore the learning that has taken place

Portfolios may include: n n n n Case reports A checklist of practical procedures undertaken consultations Descriptions of learning experiences Commentaries on books or journals Research project reports Published work

Construction of the portfolio Identifies how the experience relates to course objectives n Identifies what has been learned n Identifies what further learning required n Undertakes the further learning n Describes the further learning n

Extending the pyramid Competence Does Capability Shows how Knows “Meta” skills

Conclusions n Assessment requires careful planning and monitoring (just like curriculum design) n Assessment requires continuous evaluation and adjustment