Evaluation of service userled role play feedback for

Evaluation of service user-led role play feedback for social work students Dr Eleni Skoura-Kirk, Lecturer in Social Work Sarah Brown, Senior Lecturer in Social Work Dr Rasa Mikelyte, Research Assistant - CHSS

Introduction × Evaluation of service-user led role play interviews as part of the Readiness for Direct Practice module (BA year 1 students) × Examine the development of professional skills (procedural and meta-competencies; Bogo, 2006) for social work students through assessment feedback from service users, practice educators and student self-assessments. × Do role-plays with service users lead to better communication, interpersonal and reflective skills? 2

Skills development through role-plays × Skill development through experiential learning (Kolb, 1984); importance of reflective observation (Cheung and Dalavega 2014; Schon, 1987) × Service user and carer involvement: × involvement in role-plays creates a more ‘authentic’ experience × self-reported improvements in developing empathy, active listening skills, selfawareness and challenge to preconceptions (Skilton, 2011; Duffy, Das and Davidson, 2013; Hitchin, 2016) 3

Role play interviews: design & implementation § OSCE model adapted for social work (Bogo et al, 2006): observed standardized role-play interviews, assessed for development of professional skills § Procedural skills (ability to undertake set tasks to a required level; in this case relationshipforming and professional communication skills) § Meta-competencies (cognitive, critical and selfreflective capacities of the learner) § Formative assessment based on: PI feedback, Student self reflection and PE group feedback 4

Role play interviews: design & implementation Role-play based on scenario written by Partnership Initiative members (PIs), given to students in advance. • Reflecting PCF Readiness for Direct Practice levels, assessing building rapport, engaging with the individual and identifying some initial assessment information. • Learning and Teaching Enhancement Fund (University of Kent) Roleplay 1 - 15 minute roleplay between 1 Student and 1 PI member - Feedback collected from Students and PIs soon after - PE-facilitated reflective discussion 1 week later, PE feedback collected Roleplay 2 March 2018 November 2017 • - 15 minute roleplay between 1 Student and 1 PI member - Feedback collected from Students and PIs soon after - PE-facilitated reflective discussion 1 week later, PE feedback collected

Methodology Participants: Students: 3% <20 yos 7 PIs 70% 20 -30 yos 4 PEs 30 Students 17% 30 -40 yos 10% 40+yos Category 1 97% Female 6

Methodology Qualitative section: Quantitative section: - - Thematic analysis of: - PI qualitative feedback on assessment forms - Student selfassessment forms completed soon after the role plays (qualitative feedback) - PE reflective discussion feedback form (qualitative) PE Survey on roleplay performance - Student Self-Rating Survey on roleplay performance - 5 items; 10 -point semantic differential scale PI Survey on roleplay performance - 3 items; 10 -point semantic differential scale 15 items; 5 -point Likert scale Repeat measures 7

Qualitative Findings 8

1. Overall improvement Students “Since the first role play I feel that I have gained many new communication skills, I am a lot more confident when speaking to service users” (S 18) PIs From: “Student was so nervous that it is very difficult to answer these two questions. She shows signs of future promise but will have to work on her approach to service users and lack of confidence” To: “Student goes from strength to strength. She is confident and capable and knows how to ask the right questions” (P 03 for S 22) PEs “[The students] were more confident regarding asking openended questions compared to when they completed the initial role plays” (E 02) 9

2. Diverging emphases of what improved Students Procedural & Explicit Improvement PEs Cognitive & Theory-to-Practice Improvement PIs Relational & Embodied Improvement 10

2. Diverging emphases of what improved Students: Procedural & Explicit Improvement “I ensured that I was: summarising, reflecting, using minimal encouragers, showing empathy, used empowering speech and my body language was open” (S 24) PEs: Cognitive & Theory-to-Practice Improvement “More discussion about theories used – strengths-based, Active listening and paraphrasing, Narrative theory, Ecological Theory, Systems Theory and even Solution-focussed” (E 01) PIs: Relational & Embodied Improvement “Student was very engaged and completely interested in and focused on what I was saying” (P 03) 11

3. Competing ideas on what improvement is desired Problem-Solving Students PEs “I probably attempted to problem solve too early, rather than establishing a relationship/rappo rt with the service user” (S 27) “We discussed the temptation to problemsolve and offer solutions before the assessment stage has been completed. It was explored that there could be a risk of not only raising unrealistic expectations, but also not listening to the service user” (E 02) PIs “Willingness to find out what can be done. ” (P 07) “Very quick to think of solutions” (P 05) 12

Quantitative Findings 13

Change in Student Self-Ratings Change in PE Ratings 10 10 9 9 7. 97 8 7 7. 80 8. 93 8. 23 8. 10 7. 93 8 6. 87 6. 63 7 6 6 5 5 4 4 3 3 2 2 1 9. 18 7. 93 7. 48 6. 53 6. 47 6. 40 6. 17 4. 44 1 Cognitive Development Reflective Development Time 1 Time 2 Affective Development Performance 14 Knowledge of theory Confidence Time 1 Time 2 Helpfulness of Need to Reflection develop skills

Time 1 Time 2 en fid ce g 4. 1 on 4. 6 C 4. 1 tin g 4. 7 ee tin 4. 3 m e th g ee r m on rg ja g 4. 5 in ss fo in ns 4. 3 ne ed id io 4. 7 En d ar ep Pr 4. 1 st ue gs 4. 8 4. 7 Av o q lin t ec sp ee I f r P nt va le re fo e 4. 5 ng ki As re 3. 7 ar ith fo rt om 4. 6 C w PI I c s w 4. 8 4. 5 g tin ea r P fo ut 4. 1 Tr o ok s 4. 4 P I v ie w ie I v I 4. 8 Lo ng di an st er r P P 4. 4 nd fo to g in 4. 6 U g in w e 4. 8 lo en m a 4. 0 st ti to nd Very well 4. 4 Li ng ki ic Fairly well 4 ge g Somewhat 4. 1 St a tin ee r m of n io at an n io ct Slightly Not at all well 5 Al pl du tro 1 fo n io at an In 2 Ex pl Ex Change in PI ratings 4. 4 4. 2 4. 4 3. 8 3

Results PI Ratings PE Student Self-Rating Cognitive Development Reflective Development Affective Development Performance Knowledge of theory Confidence Helpfulness of Reflection Need to develop skills Introduction Explanation for meeting Explanation of agenda Sticking to time Listening to PI Allowing for PI views Understanding PI views Look out for PI comfort Treating PI with respect Care for PI feelings Asking relevant questions Avoiding jargon Preparedness for meeting Ending the meeting Confidence Time 1 M SD 6. 63 1. 19 7. 80. 41 8. 10. 86 6. 70 1. 44 6. 29 1. 78 6. 46 1. 80 8. 89 1. 83 6. 15 2. 07 4. 07 1. 16 4. 07 1. 08 3. 83 1. 12 4. 20. 93 4. 40. 68 4. 50. 63 4. 14. 83 3. 89. 85 4. 66. 61 4. 30. 84 4. 17. 87 4. 43. 57 4. 24. 83 4. 17. 75 3. 90. 96 Time 2 M SD 7. 97. 67 7. 80. 41 6. 87 1. 14 7. 48 1. 55 7. 93 1. 33 7. 93 1. 54 9. 18 1. 52 4. 44 2. 08 4. 76. 44 4. 40. 72 4. 37. 77 4. 60. 68 4. 80. 41 4. 77. 50 4. 55. 57 4. 48. 58 4. 83. 38 4. 73. 52 4. 50. 68 4. 73. 52 4. 55. 51 4. 43. 57 4. 40. 56 df 29 29 29 26 27 27 27 26 28 29 29 29 28 29 29 t-test -7. 62*** -1. 43 -4. 85*** -2. 01 -4. 65*** -4. 27*** -. 90 3. 23** -3. 10** -1. 44 -2. 44* -2. 56* -3. 03* -2. 28* -2. 35* -3. 65** -1. 54 -2. 90** -1. 84 -2. 19* -1. 88 -1. 86 -2. 35* 16

Results PI Ratings PE Student Self-Rating Cognitive Development Reflective Development Affective Development Performance Knowledge of theory Confidence Helpfulness of Reflection Need to develop skills Introduction Explanation for meeting Explanation of agenda Sticking to time Listening to PI Allowing for PI views Understanding PI views Look out for PI comfort Treating PI with respect Care for PI feelings Asking relevant questions Avoiding jargon Preparedness for meeting Ending the meeting Confidence Time 1 M SD 6. 63 1. 19 7. 80. 41 8. 10. 86 6. 70 1. 44 6. 29 1. 78 6. 46 1. 80 8. 89 1. 83 6. 15 2. 07 4. 07 1. 16 4. 07 1. 08 3. 83 1. 12 4. 20. 93 4. 40. 68 4. 50. 63 4. 14. 83 3. 89. 85 4. 66. 61 4. 30. 84 4. 17. 87 4. 43. 57 4. 24. 83 4. 17. 75 3. 90. 96 Time 2 M SD 7. 97. 67 7. 80. 41 6. 87 1. 14 7. 48 1. 55 7. 93 1. 33 7. 93 1. 54 9. 18 1. 52 4. 44 2. 08 4. 76. 44 4. 40. 72 4. 37. 77 4. 60. 68 4. 80. 41 4. 77. 50 4. 55. 57 4. 48. 58 4. 83. 38 4. 73. 52 4. 50. 68 4. 73. 52 4. 55. 51 4. 43. 57 4. 40. 56 df 29 29 29 26 27 27 27 26 28 29 29 29 28 29 29 t-test -7. 62*** -1. 43 -4. 85*** -2. 01 -4. 65*** -4. 27*** -. 90 3. 23** -3. 10** -1. 44 -2. 44* -2. 56* -3. 03* -2. 28* -2. 35* -3. 65** -1. 54 -2. 90** -1. 84 -2. 19* -1. 88 -1. 86 -2. 35* 17

Results Change between Time 1 and Time 2 1. Change in PE Ratings 2. Change in Student Rating 3. Change in PI Rating N 30 27 26 M 1. 00. 50. 42 SD 1. 06 1. 22. 63 Correlations 1 2 3 1 2. 55 1 -. 12 -. 02 1 × Most ratings showed improvement at Roleplay 2, whether they were made by PIs, PEs or the students (based on t-test results) × However, PI, PE and Student self-ratings did not go together × Student self-perceived improvement or deterioration in roleplay performance was not linked to that seen by the PIs or the PEs 18

So… Do role-plays with service users lead to better communication, interpersonal and reflective skills?

Key messages × Evidence suggests overall improvement in students’ professional skills. × The repetition of the task contributed to increased confidence, allowed exercise and refinement of procedural skills and linking theory to practice. × Three-fold assessment promotes a holistic approach to the task (procedural/cognitive/relational). × Competing ideas: importance of service user and carer involvement in social work education, avoiding exclusive reliance on professional narratives.

References Bogo, M. , Regehr, C. , Woodford, M. , Hughes, J. , Power, R. & Regehr, G. (2006) ‘Beyond competencies: field instructors’ descriptions of student performance’, Journal of Social Work Education, vol. 42, no. 3, pp. 579– 593. Cheung, M. and Dalavega, E. (2014) Five-way Experiential Learning Model for Social Work Education, vol 33, no. 8, pp 1070 -1087. Duffy, J. , Das, C. and Davidson, D. (2013) Service User and Carer Involvement in Role-plays to Assess Readiness for Practice, Social Work Education, 32(1), pp. 39 -54. Hitchin, S. (2016) Role-played interviews with service users in preparation for social work practice: exploring students’ and service users’ experience of coproduced workshops, Social Work Education, 35: 8, 970 -981 Kolb, D. A. (1984) Experiential Learning. Englewood Cliffs: Prentice Hall. Schon, D. (1987) Educating the Reflective Practitioner, Jossey Bass, San Francisco. Skilton, K. J. (2011) Involving Experts by Experience in Assessing Students’ Readiness to Practise: The Value of Experiential Learning in Student Reflection and Preparation for Practice, Social Work Education, 30(3), pp. 299 -311 21

Thanks! Any questions? You can contact us at: E. Skoura-Kirk@kent. ac. uk 22