ESCalate Research Project Using reflective dialogue to assess

ESCalate Research Project ‘Using reflective dialogue to assess professional learning’ website http: //escalate. ac. uk/6333 Ruth Pilkington RMHPilkington@uclan. ac. uk

Non-formal Learning Formal Learning NEW HE STAFF Induction or Toolkit Week Courses I lead, e. g. Standard Descriptor One University HE Teaching Toolkit PG Cert LTHE or another PG Certificate PG Diploma & CPD awards MEd UK PSF Standard Descriptor Two EXPERIENCED HE STAFF Informal, on-the-job development Professional Dialogue Assessment Process Standard Descriptor Three A CPD FRAMEWORK FOR UCLan showing formal and informal learning routes to professional development and accreditation

The Uclan Approach Formal Courses • Toolkit • PGCert LTHE • Certificates/other PGCerts • PGDiploma • MEd • Ed. D Informal Process • Experienced • Discipline-specific • Reflective • Developmental • Discursive • Meaning-making = Professional Dialogue

Informal Learning Route: The ‘Professional Dialogue’ • Normally takes place between an experienced ‘other’ and the ‘student’. • It involves a 3 -stage process: – an initial set-up dialogue, – an exploratory dialogic process, and – the assessed dialogue. • Stage 3 is recorded and involves a second assessor to act as moderator or second marker. • In Stage 3, one assessor comes from the subject or ‘field’ and acts as mentor through the preparation phase of dialogue, the other may have a broader, objective, generic teaching and learning perspective

Assessment and Reward: the Professional Dialogue showing Timeline, Input and Process Mentee Participant Mentor Set up Dialogue: high Mentor input; low Mentee ownership Dialogue 1: high Mentor leadership; increased Mentee ownership Dialogue 2: shared ownership; increased Mentee input and determination Assessed Dialogue: celebration of Mentee role and practice Adapted and adopted by Ruth Pilkington from the process outlined by Brockbank and Mc. Gill (2007)

Benefits Ethos of process Peer-led, supportive, celebratory, affirmative Status to teaching and learning; space Sharing and exchange Communication medium: reliability Rigour of process Can work with teams – culture change Meaning-making, reflective, subject and individual focus • Not ideal for new staff or directed development needs • •

The Professional Dialogue Model • • • Applies a model by Brockbank & Mc. Gill (2007). They outline specific conditions supporting a collaborative approach to process-oriented, active, peer-supported, reflective learning: – Ascertain a reflective dialogue has taken place (ideally with others); – Establish evidence of learner’s participation in dialogue; – Identify evidence of developmental process over time, regardless of the start / end point; – Ascertain evidence that process review has taken place, enabling students to take away understanding of the learning process and replicate it elsewhere. (p 194) Boud & Falchikov (2007) use a cognitive model of coaching particularly relevant to peer assessment of professional learning called ‘cognitive apprenticeship ‘. Cognitive apprenticeship focuses on dialogue over an indefinite period using a structured process of ‘modelling, scaffolding, fading and coaching’ (Collins et al 1991: 2, cited Boud & Falchikov, 2007: 130). Coaching forms a thread that runs throughout ensuring that the collegial relationship is built on trust, confidentiality and empowerment. Meta-cognitive apprenticeship skills are developed in partnership with the experienced ‘other’ who initially models behaviours, scaffolds professional learning using reflective processes, encourages interrogation and engagement with self assessment, and then gradually fades out as ‘expert’.

The UK Professional Standards Framework Values Statements for Professional Standards a) respect for individual learners b) commitment to incorporating the process and outcomes of relevant research, scholarship and/or professional practice c) commitment to development of learning communities d) commitment to encouraging participation in higher education, acknowledging diversity and promoting equality of opportunity e) commitment to continuing professional development and evaluation of practice Areas of Activity 1. Design and planning of learning activities and/or programmes of study 2. Teaching and/or supporting student learning 3. Assessment and giving feedback to learners 4. Developing effective environments and student support and guidance 5. Integration of scholarship, research and professional activities with teaching and supporting learning 6. Evaluation of practice and continuing professional development Core Knowledge 1. The subject material 2. Appropriate methods for teaching and learning in the subject area and at the level of the academic programme 3. How students learn, both generally and in the subject 4. The use of appropriate learning technologies 5. Methods for evaluating the effectiveness of teaching 6. The implications of quality assurance and enhancement for professional practice

Project Process • Four UK Institutions • Professional Dialogues used 1. 2. 3. • • Dialogues recorded and transcribed Analysis of themes emerging (Tag Cloud, thematic identification, reflective cyclical review). attentiveness to issues – a personal response, discussion with colleagues – strengthen and derive codes – – – • • To assess academic leadership (Standard Descriptor 3) To assess new lecturers wanting to meet Standard Descriptor 1 To assess experienced lecturers seeking Standard Descriptor 2 (core lecturer role) Structure of dialogues and management of interaction; facilitation and support; Evidence of reflection and evidence of meeting Standard Descriptors Phatics , use of small talk Rigour of dialogue as assessment tool Judgement and power; performance and control Storytelling, narrative of meaning, cosntruction Reflection by team and review of data, experience Interview data collected from participants and assessors on process

Group Activity • Explore within groups how this might work within your context • Examine examples provided to discuss issues of dialogue within the process • Can you use the model for your assessment? – (Share outlines from institutions and documents) • What would be the issues for you in terms of assessing professional learning?

Issues and Outcomes Power, Judgment Training and Support for Assessors AFTERNOON SESSION

Review of Dialogues • Modelling of dialogue • Emergence of issues of power • Analysis of different types of dialogue; ways that power emerges • Evidence of judgment – a complex thing

Use of Questions • Open Qs / Direct Closed Qs / Open Qs – Redirect issues or focus • Closed questions move things on; focus attention • Open Questions probe and explore • Affirmatory and Summative statements

Dialogue – power and parity • Power (assessor): scene setting, introductions; interruptions, questioning, probing & direction, judgment • Power (assessee): taking ownership by managing exploration, redirecting questions, pursuing own line of thought, own judgments • Reflecting back at assessee; affirmation; noise; comments on issues; sharing own thoughts; empathy – support; meta-cognitory unpacking • Equality: shared exchanges; shared feelings; asides, exploratory detours

Enhancement components • Reflection: description, thinking things through, exploring own experience, reasons behind, recognition of own values • Storytelling: evidence, authenticity, entry to reflection, role and identity • Relationship building: sharing, humour, use of phatics, empathy • Meaning making: exploration, explanation, interpretation, reflecting back, meta-analysis, positioning within outcomes

Dialogue - Types • Parity, Equality • Assessor control monologue dialogue presentation interview Incr. Input by assessor supported monologue Incr. Input by assessee Incr. Control by assessee interview interrogation Incr. Control from assessor

QUESTION NEW QUESTION 1 PROBE Dialogue Cycles SHARE REVIEW SUMMARISE REFLECT REPEAT CYCLES FINAL STAGE REVIEW GAP CHECK FEED BACK 2 INTRODUCTION SET CONTEXT CONCLUSION TOPIC AFFIRM RESPONSE PROMPT / PROBE NEW TOPIC JUDGMENT EXAMPLE LINK MADE DESCRIBE

Institution A • Preparatory dialogue and final dialogue • 2 nd final dialogue • Peer dialogue • Video of final dialogue by deaf studies Mentoring process Co-construction Story telling Turn-taking Power and management • Open v. closed questions • Judgment • • •

Assessment, Reflection and Professional Judgement within Dialogue – an exciting outcome Dialogue ICEBERG An Interesting ‘Sub-text’ • Assessment of professional knowledge • Judgment • Assessment of values • Use of reflective models to assess professional learning

Judgment • Expertise and experience of assessor • Comfort and familiarity with outcomes (UK PSF) and field (L&T), and appreciation of subject (of assessee) • Familiarity with setting: evidence, locus, complexity, authenticity – areas of activity • Reading body language, examples, stories, language – truth, reliability, values • Contextualising: alignment to outcomes; tacit v. explicit; hidden text • Gap analysis: Is it enough? What more is needed?

Use of Dialogue • • Purpose of Dialogue Position of Dialogue Process Training of Assessor, Participant

Brockbank A & Mc. Gill I 2007 Facilitating Reflective Learning in HE SRHE/OUP 2 nd Ed. Bowen Clewley L ‘Assessing against competency standards in the workplace’ in 207 -227 in Arguelles & Gonczi (2000) Eraut M 1997 Professional Learning and Competence Ghaye T & Lillyman S 2006 Learning Journals and Critical Incidents RP for HE Professionals 2 nd Ed Quay Books London O’Donovan B, Price M and Rust C (2004) ‘Know what I mean? Enhancing Students understanding of assessment standards and criteria’ in Teaching in HE Vol 9, No 3, July 2004 Knight P & Yorke M (2003) Assessment Learning and Employability SRHE/OUP Boud D and Falchikov N Eds Rethinking assessment in HE 2007 Routledge Bryan C & Clegg K 2006 Innovative Assessment in HE Routledge Goodfellow R & Lea MR (2007) Challenging E-learning in the University: a literacies perspective SRHE / OUP Larrivee, Barbara(2008)'Development of a tool to assess teachers' level of reflective practice', Reflective Practice, 9: 3, 341 — 360 Moon J (2004) A Handbook of Reflective and Experiential Learning: Theory & Practice Routledge Falmer Pedler M (1996), Action Learning for Managers, The Learning Company Project Shulman LS ‘Knowledge and Teaching’ pp 61 -77 in Leach J, Moon B (eds) (1999) Learners and Pedagogy Sage/ Paul Chapman pubs