Demonstrating outcomes and impact in the context of

Demonstrating outcomes and impact in the context of professional learning and development

Aims To identify and explore some key questions regarding evaluation, impact and outcomes To introduce and exemplify a 'level model' of evaluation of professional development outcomes

Impact evaluation should focus on what participants learn, how they use what they have learned and the effect on the learning of children and young people (Training and Development Agency for Schools 2007, p. 2).

Kirkpatrick (1998), identified four levels of outcome for training interventions: (i) participants’ reactions; (ii)participants’ learning; (iii) changes in participants’ behaviour; and (iv) desired results.

Measuring impact and outcomes: key questions Which aspect(s) of impact are we measuring? Are these the right ones? For what purpose are we measuring them? How do we go about measuring them? Are our measurements valid? What do we learn from measuring them?

A levels model of evaluation of professional development outcomes We have used the term ‘level models’ to describe a family of evaluation approaches that share the characteristic of tracing the effects of training and development interventions through a series of ‘levels’, each of which more closely approaches the ‘ultimate’ intentions or outcomes of the intervention. Coldwell, Mike and Simkins, Tim (2010) page 2

Level model Coldwell and Simkins 2010 Antecedents Participants' expectations and motivation Moderating factors Setting/organisational culture Participants' role/ organisational structure Capacity/resourcing to support change Intervention support - coaching, supervision Time to reflect/plan/act Final Outcomes 3 Capacity individual/team capability Education setting systems/practices/culture Interventions CPD input Intermediate Outcomes 1 Participant reactions Intermediate Outcomes 2 Learning & personal development Intermediate Outcomes 3 Behaviour Final Outcomes 1 Short/longer term CYP outcomes Final Outcomes 2 Career development

Interventions: the CPD activities themselves. Antecedents: those factors associated with individual participants that affect their ability to benefit from the opportunities offered to them. Intermediate outcomes: those outcomes of the CPD activities that are steps towards (pre-conditions for) the achievement of the final outcomes, Final outcomes: the intended effects of the CPD activities, on organisations, teachers and children and young people.

Moderating factors: variables in the school and wider environment that influence whether, and how, the interventions lead, via the achievement of intermediate outcomes to producing final outcomes. These factors help to explain why apparently similar activities have different consequences for different individuals, teams and schools. Understanding these factors helps in preparing/ supporting individuals and settings to achieve intermediate and final outcomes

Narrowing the Gap: Improving Learning for Vulnerable Children and Young People using Lesson Study • TSA supported and funded project - context school improvement, raising attainment for vulnerable groups of children • Representative schools identified in clusters by TSA lead group (primary secondary special)

Antecedents Individual school representation - a team must include SLT and an inclusion specialist e. g. SENCO Participants' expectations and motivation - briefing paper sent outlining lesson study and the purpose of the project Interventions Session 1 Introduction to lesson study Identify a pupil who puzzles you and is not making adequate progress Identify lesson study team in school Session 2 Planning the lesson study Supported planning addressing main issues of implementation Inter session offer of additional support for planning Intermediate Outcomes 1 Participant reactions Identifying learners Commitment to undertake lesson study Sharing practice agreeing outcomes analysis and moderating factors Active initial planning as a team Session 3 Outcomes presentations

Moderating factors Setting/organisational culture - is this a priority for us? Moderating factors Setting/organisational culture - are we focused on understanding learners learning or implementing an intervention Capacity/resourcing - this is resource intensive, do we have the resource and are we willing to commit it to this work? Participants' role - SLT support? Parental support for the lesson study Time to reflect/plan/act Capacity/resourcing - staffing and attendance issues Time to reflect/plan/act - has the team applied the features of lesson study effectively? Did they need/ access more support to do so? Intermediate Outcomes 1 Participant reactions Intermediate Outcomes 2 Learning & personal development Identifying learners Understanding the key features of the lesson study model Commitment to undertake lesson study Developing the practice of working collaboratively with colleagues Active initial planning as a team Developing the practice of gaining learner's perspectives on their classroom experiences Intermediate Outcomes 3 Behaviour Teachers /TAs apply knowledge of individual learners and insight into learner perceptions of learning Focussed collaboration between teaching assistants, teachers, SENCO and specialists Teachers /TAs shift emphasis from techniques re task completion/ curriculum delivery to barriers to /enablers of learning Active learner engagement in goal-setting and understanding their own learning and influencing their learning context

Moderating factors Culture- preconceptions regarding learners' role, autonomy and insights Final Outcomes 3 Capacity individual/team capability Education setting systems/practices/culture Capacity/resourcing to enable collaboration Enhances teachers’ /TAs’ awareness of the importance of knowledge of individual learners and insight into learner perceptions of learning Time to reflect/plan/act re interventions/ adjustments etc Develops skills in gaining learners' perceptions regarding their learning Challenges teachers’ /TAs’ perceptions of learners Intermediate Outcomes 3 Behaviour Teachers /TAs apply knowledge of individual learners and insight into learner perceptions of learning Focussed collaboration between teaching assistants, teachers, SENCO and specialists Teachers /TAs shift emphasis from techniques re task completion/ curriculum delivery to barriers to /enablers of learning Active learner engagement in goalsetting and understanding their own learning and influencing their learning context Models structured collaboration between teaching assistants, teachers, SENCO and external specialists Final Outcomes 1 Short/longer term outcomes for learners Identifies barriers to learning that impact on a range of learners Informs design of interventions, adjustments and associated support Enables progress towards short and medium-term targets Final Outcomes 2 Career development Implement and articulate: leadership of collaborative practice evidence informed responses to differentiation, personalisation, intervention and adjustment

References Coldwell, Mike and Simkins, Tim (2010) 'Level models of continuing professional development evaluation: a grounded review and critique', Professional Development in Education pp 1 -15 Kirkpatrick, D. , (1998) Evaluating training programmes: the four levels. 2 nd ed. San Francisco, CA: Berrett-Koehler Sachs, Judyth (2007) Learning to improve or improving to learn: the dilemma of teacher continuing professional development, Proceedings of the International Congress Effectiveness and Improvement. Training and Development Agency for Schools, 2007. Impact evaluation of CPD. London: TDA.

Professional development/learning Sachs 2007 Driver Retooling Accountability and control by government Purpose Upgrading of skills Focus Professional development Passive recipient of knowledge Learning process Remodelling Compliance with government change agenda Modifying existing practices Revitalising Professional renewal Re-imagining Professional reinvention Rethink and renew practices Transformative practices Professional development Uncritical consumer Professional learning Collaboration Mutual engagement and knowledge creation Approaches One-off seminars outside Programmes devised by Collaborative learning expert programmes expert over an extended circles, networks, action time research Practitioner enquiry or action research, enquiry as stance Professional outcomes Improving teaching skills Updated discipline knowledge or pedagogical skills New approaches to pedagogy and learning Production of new knowledge View of teaching/ professionalism Teacher as technician Controlled Teacher as reflective learner Collaborative Autonomous professional Activist Teacher as craft worker Compliant