Universiteit Maastricht th 11 Ottawa conference on Medical

Universiteit Maastricht th 11 Ottawa conference on Medical Education Barcelona, 6 – 9 July 2004

Where do we go from here? Universiteit Maastricht new challenges in assessment of clinical competence where do we go from here? Lambert Schuwirth

Where do we go from here? Universiteit Maastricht overview of the presentation - why this presentation - developments in competence assessment - developments in performance assessment - implications for research

Where do we go from here? Universiteit Maastricht overview of the presentation - why this presentation - developments in competence assessment - developments in performance assessment - implications for research

Where do we go from here? Universiteit Maastricht experts vs novices: context vs concept Ff g k 5 1, 2 Fz

Where do we go from here? Universiteit Maastricht overview of the presentation - why this presentation - developments in competence assessment - developments in performance assessment - implications for research

Where do we go from here? Universiteit Maastricht simple isn’t it? pass competent fail incompetent test

Where do we go from here? Universiteit Maastricht simple, it isn’t competent pass fail incompetent

Where do we go from here? Universiteit Maastricht modern approaches to assessment 1 re-evaluate the nature of medical competence 2 shift towards more student-centred assessment 3 include general professional competencies 4 broaden the measurement time frame

Where do we go from here? Universiteit Maastricht 1 trait orientated view on medical competence knowledge skills problem solving attitudes instrument instrument instrument

Where do we go from here? Universiteit Maastricht one instrument for one trait instruments A 1 traits 2 3 4 B C D

Where do we go from here? Universiteit Maastricht one instrument for one trait

Where do we go from here? Universiteit Maastricht old model of expertise: problems more variation within than between methods

Where do we go from here? Universiteit Maastricht 1 trait orientated view on medical competence knowledge skills problem solving attitudes instrument instrument instrument

Where do we go from here? Universiteit Maastricht 1 trait orientated view on medical competence knowledge skills problem solving attitudes instrument instrument instrument

Where do we go from here? Universiteit Maastricht old model of expertise: problems more variation within than between methods content determines the assessed competence

Where do we go from here? Universiteit Maastricht Miller's pyramid does shows how knows

Where do we go from here? Universiteit Maastricht 2 shift towards student-centred assessment - self-assessment self-evaluation skills assessing the unmeasurable - peer-assessment peer-evaluation skills peer pressure? assessing the unmeasurable - co-assessment self, peer and co-evaluation skills peer pressure? joint responsibility

Where do we go from here? Universiteit Maastricht 3 include general professional competencies dealing with tasks planning and time management information gathering dealing with others giving feedback and criticising communicating co-operation dealing with oneself-responsibility self-evaluation self-reflection

Where do we go from here? Universiteit Maastricht 4 broaden the measurement time frame specific test occasions - specific preparation of students - inflexibility - unwanted influence on learning - certain aspects unmeasurable longitudinal testing - no specific preparation - congruent with flexible curricula - wanted influence on learning

Where do we go from here? Universiteit Maastricht lessons learnt good assessment draws the whole picture no single method is a panacea the content is important, not the format too much structure does not work sampling strategies are essential

Where do we go from here? Universiteit Maastricht concept and context: subjectivity Mozart experts Lambert

Where do we go from here? Universiteit Maastricht sampling strategies are essential Lambert Neil Dale how do you feel now 7 5 9 how did you feel 5’ ago 7 5 9 how did you feel 10’ ago 7 5 9 how will you feel in 5’ 7 5 9 how will you feel in 10’ what does your partner about think how you feel

Where do we go from here? Universiteit Maastricht lessons learnt good assessment shows the whole picture no single method is a panacea the content is important, not the format too much structure does not work sampling strategies are essential

Where do we go from here? Universiteit Maastricht overview of the presentation - why this presentation - developments in competence assessment - developments in performance assessment - implications for research

Where do we go from here? Universiteit Maastricht definitions competence what professionals can do in controlled representations of professional practice, performing at the maximum of their ability performance what professionals actually do in their professional practice

Where do we go from here? Universiteit Maastricht current approaches: still trait orientated 1 2 3 4 5 6 7 8 clinical expertise communication skills collaboration skills management skills personal development educational skills professional attributes personal health & self care

Where do we go from here? instruments chart review data review rating scales surveys interviews critical incident analyses video portfolios incognito simulated patiens Universiteit Maastricht

Where do we go from here? Universiteit Maastricht but……. no direct link between instrument and trait domains are not discrete entities trait-orientated approach did not work in competence assessment so……. modern approaches define competence and performance in terms of professional roles

Where do we go from here? Universiteit Maastricht professional role-orientated goals health care management patient care person scientist teacher

Where do we go from here? Universiteit Maastricht 1 role – 1 instrument? instruments A 1 roles 2 3 4 B C D

Where do we go from here? Universiteit Maastricht multi-modal assessment sources of information A 1 roles 2 3 4 B C D

Where do we go from here? Universiteit Maastricht how: seeing the whole picture

Where do we go from here? Universiteit Maastricht how: approaches - variation in instruments needed - performance often involves subjective aspects -collation of information based on content-equal bits and not on method-equal bits no flight into structuring, but in sampling

Where do we go from here? Universiteit Maastricht how: organisation plan purposes elements of defensibility quality control measures implementation pathways administration consequences assessee

Where do we go from here? Universiteit Maastricht observations instrument is not synonymous with a competence no single instrument is the panacea, a variety is needed: an assessment programme content is more important than format: it is the concept that counts, not the format assessment is not mainly a measurement problem but more an (educational) design problem.

Where do we go from here? Universiteit Maastricht suggestions assessment is not mainly a measurement problem but more an educational design problem. masterplan of the assessment system - built-in quality control procedures and bodies - sampling in stead of (over) structuring - variation of methods, compensation, regulations - saturation of information approach - training of students, teachers, examiners, board members

Where do we go from here? Universiteit Maastricht overview of the presentation - why this presentation - developments in competence assessment - developments in performance assessment - implications for research

Where do we go from here? Universiteit Maastricht trait orientated assessment as a psychological testing problem - highly quantitative methodologies - reliability of methods - construct validity +++ +++

Where do we go from here? Universiteit Maastricht trait orientated assessment as a instructional design problem - quantitative methodologies - reproducibility of methods - construct validity - qualititative methodologies - direct validity - consequential validity - research into assessment systems + + + ± ++ ++++++

Where do we go from here? Universiteit Maastricht observations methodology is not synonymous to research question the methodological concept is important not the specific analysis or method no single one study/methodology will provide the definitive answer, a variety is needed research of assessment systems is more important now than research into single methods.

Where do we go from here? Universiteit Maastricht research agenda assessment as a programme: what constitutes good assessment programmes? which building blocks should be included? who should do what? how should information be collated: qualitatively / quantitatively?

Where do we go from here? Universiteit Maastricht research agenda reproducibility of (portfolio) judgments? g-analyses other statistical approaches sequential testing? organisational approaches?

Where do we go from here? Universiteit Maastricht Research agenda quality elements of multi-modal assessment how to “score” qualitative judgments? identifiers for profile scores what influences the thinking of judges? fraud/plagiarism (internet) flexible adaptation of teaching to assessment outcomes

Where do we go from here? Universiteit Maastricht research agenda consequential validity / educational impact roles of format, content, scheduling and regulations what influences teacher behaviour

Where do we go from here? Universiteit Maastricht research agenda consequential validity / educational impact

Where do we go from here? Universiteit Maastricht research agenda research into assessment programmes how to change a programme? stakeholder involvement how are the instruments combined? how to use parts in stead of whole instruments? rules of design

Where do we go from here? Universiteit Maastricht overview of the presentation - why this presentation - developments in competence assessment - developments in performance assessment - implications for research SUMMARY

Where do we go from here? Universiteit Maastricht conclusions: in (education), assesment and research the method is not synonymous with the concept translation of concepts into methods is useful no single method is a panacea strength comes from collation of results

Where do we go from here? Thank you Universiteit Maastricht