What you see is what you get or

What you see is what you get – or is it? Improving the quality and consistency of Direct Observations of Practice Mary Keating and Cathie Jasper NOPT 7 th July 2016

Skills for Care Pilot - sites Partnerships Area Trafford Council Cheshire East Manchester Metropolitan University North West Kent County Council Canterbury Christ Church University Medway Council. London and the South East Leeds City Council Wakefield Leeds Beckett University of Leeds Yorkshire and Humber and the North East London Borough of Enfield Middlesex University London and the South East

Why are we doing this? § Direct observation - integral part of social work education in UK – pre and post qualifying § Conduct and assessment of practice is variable (Ruch, 2015; Bogo et al. , 2014) § Major issue is assessor capability and bias (Humphrey, 2007; Finch and Taylor, 2013) § PEPS/PDE have no specific learning outcomes for observation § Desirable to moderate/standardise assessment of practice

What is different? Pilot project to test the following: Tightened assessment criteria § Two stage assessment 1. Direct observation. 2. Post observation structured written review by NQSW. Assessor training § Development of learning outcomes for assessors, focus on observation skills. § Development of training curriculum. Video recording of live practice to support learning and internal moderation

Outputs from the project What are we doing / what have we done / what do we expect to achieve? • Development of improved criteria for the assessment of Direct Observations based on the Knowledge and Skills Statement (adult) 2015 and the PCF at ASYE level • Identification of Learning Standards for Assessors with a focus on observation skills • Development of a training programme for Assessors • A video solution for recording and secure storage of live practice observations

Adaptation 1…. ’keep/change’ rule (Doel et al. , 1996) Marion Bogo work on OSCEs adapted for social work (Bogo et al. 2014; Bogo, et al. , 2011) Objective: Skills directly observed Structured: A set scenario and format Clinical: Assesses practice skill implementation. Exam: Performance is evaluated on a set scale.

Understanding meta competency Holistic competence – involves two dimensions – procedural and meta competence (Bogo et al. , 2014) Procedural competence – “refers to performance and the ability to use procedures in various stages of the helping process and includes the ability…in direct practice, to form a collaborative relationship, to carry out an assessment, and to implement interventions with clients and systems” (Bogo et al. , 2014, p. 6). Meta competence – “refers to higher order, overarching qualities and abilities of a conceptual, interpersonal and person/professional nature. This includes students’ cognitive, critical and self-reflective capacities” (Bogo et al. , 2014, p. 6).

Simulation and simulated practice scenarios Miller (1990) Pyramid of professional competence Does…. doing and being Knowledge demonstrated / assessment methods: Exams, portfolios, written reflections, work products, supervision discussion, OSCE (shows how) Direct observation – allows for demonstration of doing and being Shows how Knows Sources: Miller, (1990); Ruch, (2015); Rushforth, (2007)

Adaptation 2…Sf. C project - what have we kept /changed? • Observation of live practice (home visit; assessment, review interview etc) “Direct observation is the optimal method for assessing professional skills because it is closest to ‘real practice’ “(Ruch, 2015, p. 38) • Development of structured assessment criteria –focus on Adults KSS / PCF level descriptor • Observation and reflective activity that: - captures procedural and meta competency - assesses NQSW cognitive understandings of process, content, conceptualisation of practice and affective domains

Adaptation 2…Sf. C project - what have we kept /changed? • NQSW – completes Post Observation Structured Written Review, immediately after observation, max 45 minutes • Both – post observation reflective discussion • Assessor – overarching holistic assessment

Example: n . Observation - Assessor criteria Assessment skills Focusing the interview, employing holistic assessment skills Responding flexibly to user or carer’s needs and concerns Adaptability (regarding use of methods, theories and responsiveness to unpredicted situations) Assessment of capacity

Example: NQSW Post observation – structured written review Post observation structured written review Summarise the key elements of the interview and the key issues/ challenges identified Identify and critically reflect upon areas of challenge and complexity in relation to KSS/PCF (e. g this might be in relation to the holistic identification and assessment of needs, risks and /or safeguarding issues; person centred approaches to decision making ; the impact of key factors such as trauma, loss, cognitive and emotional development etc. ) What were your emotions during the interview and how did they affect your practice?

Assessor training- Learning objectives Development of new learning objectives to supplement the existing Practice Development Educator learning objectives Observation Demonstrate observation skills of looking, listening and meta-competence in the identification and assessment of practice dynamics and the assessment of the intervention and behaviour of the NQSW. NEW Confidently evaluate observed evidence for its relevance, validity, reliability, sufficiency and authenticity according to the agreed standard PDE Confidently demonstrate the ability to record clearly and concisely evidence and comments on the practice of the learner. PDE Demonstrate an awareness of and ability to counter professional bias in the role NEW of the Observer by maintaining objectivity. Be skilled in maintaining a distance from the intervention unless forced to intervene

Assessor training- Programme and resources • Skills audit, pre and post training • Focus on observation skills • Major vehicle for the training is the use of video material • Training video - social worker and service user. Practice and discussion of observation assessment • Linked example of structured written review – commented on and discussed

Preliminary reflections on the training • • Positive response to the project Like the standardised assessment criteria Positive if nervous about the learning potential of use of video Difficulties in finding observation opportunities where Service User has capacity • Value of Assessor reviewing the video before finalising the observed element of Assessment • Training programme worked well

Video : Storage Secure Library All recordings will be transferred into a centrally held storage library accessible via an Internet / Wi. Fi connection. The videos will be transferred from the Surface device, once there is an adequate internet connection. They will then be automatically wiped from the surface device. The videos will be accessible from the Surface or any PC / Laptop providing the user has the login credentials to the secure library.

Who views the video?

Evaluation and Conclusions • Evaluating the project • Use of video • New criteria for observations • Moderation • Guidance – aim to be available January 2017 • Informing the Chief Social Workers thinking on ASYE

What is observer bias? • A form of cognitive bias, this is something hard wired into us. • The process by which we interpret what we observe in a way which corroborates our existing biases, either positive or negative. • Evidence suggests that once we reach a conclusion about something it is difficult for us to change our minds even when evidence to the contrary is presented.

Cognitive biases… • Confirmation bias Become attached to a judgement (e. g student not very. good); do notice disconfirming evidence; accept only evidence that fits with original beliefs (‘anchor’ hypothesis) • Bandwagon effect, “group think” Effect of others opinions on your assessment/judgements; pressure not to disagree or stand out • Availability heuristic /recall bias / relative judgements How many previous examples can be brought to mind (“this student is like another one we had where something similar happened…”); more recent examples predominate. Also, student being judged ‘relatively’ against another current or recent student • Decision avoidance Making a decision invokes action ; can be time consuming and emotionally challenging

Direct observations In undertaking direct observations observer bias can manifest in different ways 1) Seeing only what we expect to see 2) Looking for only the things that confirm our bias 3) Dismissing or minimising things that do not confirm our bias

Exercise Consider in pairs for 5 minutes, please be ready to reflect back to the whole group if required: • What things in life do you take for granted as ‘facts’? • How often are these ‘facts’ challenged? • How easy do you find it to change your mind and what needs to happen for you to do this?

How aware we? Please watch the clip below and then discuss what you think influenced what you saw and how might these things impact on your direct observation of practice? http: //www. youtube. com/watch? v=Ahg 6 qcgoay 4

Addressing observer bias • • • Where possible, try not to form bias in the first place. Objective observation Structure approach to recording observations Reflection and awareness Challenging bias- taking the opposite position. Moderation

Discussion Please take some to reflect on direct observations of practice you have undertaken in the past. Do you think observer bias played a role in this, were you aware of this at the time and if so what steps did you take to address it? What strategies will you employ to minimise the impact of observer bias on your Direct Observations?

References Ruch, G (2015), Evidence scope regarding the use of practice observation methods as part of the assessment of social work practice. RIP: Dartington. https: //www. rip. org. uk/resources/publications/evidence-scopes/regarding-the-use-of-practiceobservation-methods-as-part-of-the-assessment-of-social-work-practice-evidence-scope-2015/ Humphrey, C. (2007) Observing Students Practice( Through the Looking Glass and Beyond. Social work education Vol. 26, No 7, October 2007 pp. 723 - 736 Kowproska, J. , Hicks, L. , Mc. Cluskey, U. , Fisher, T. , and Wishart, J. (1999) Facts, Feelings and Feedback: A Collaborative Model for Direct Observations. Video. Heslington: University of York Department of Social Policy and Social Work. Doel, M, Shardlow, S, Sawdon, C and Sawdon, D (1996) Teaching Social Work Practice, Arena: Aldershot Finch, J & Taylor , I (2013) Failure to Fail? Practice Educators' Emotional Experiences of Assessing Failing Social Work Students, Social Work Education, 32: 2, 244 -258 Bogo, M; Rawlings, M; Katz, E and Logie, C (2014) Using Simulation in Assessment and Teaching, CSWE Press: Virginia Rushforth, H. E(2007) Objective structured clinical examination (OSCE): review of literature and implications for nursing education, Nurse Education Today, 27, 481 -490. Miller, G (1990) The assessment of clinical skills /competence/performance. Academic Medicine, 65 (Suppl. 9), s 63 -s 67 Bogo, M. , Regehr, C. , Logie, C. , Katz, E. , Mylopoulos, M. , & Regehr, G. (2011). Adapting objective structured clinical examinations to assess social work students’ performance and reflections. Journal of Social Work Education (47)1, 5 -18. Logie, C. , Bogo, M. , Regehr, C. , & Regehr, G. (2013). A critical appraisal of the use of standardized client simulations in social work education. Journal of Social Work Education, 49(1): 66 -80.

References Bog, M; Regehr, C; Logie, C, Katz, E, Mylopooulos, M and Regehr, G (2011) Adapting Objective Structured Clinical Examinations to assess social work students performance and reflections. Journal of Social Work Education, 14, 1, p 5 -18 Irwin, M and Mc. Glade, A (2011) You talkin’ to me? (2011) Journal of Practice Teaching and Learning, 11, 2, p. 92 -108. Irwin, M (2014) Back to Basics: A skill based approach to Assessing Social Work Students within Directly-observed Practice. Field Educator, vol 4. 2

Contact Mary. keating@skillsforcare. org. uk Cathie Jasper C. Jasper@mmu. ac. uk