Performance Assessment for Leaders the Massachusetts PAL Assessments

  • Slides: 22
Download presentation
Performance Assessment for Leaders: the Massachusetts PAL Assessments and Field Trial Results New York

Performance Assessment for Leaders: the Massachusetts PAL Assessments and Field Trial Results New York State Principal Project Advisory Team Meeting New York State Education Department Albany, New York January 25, 2017

Credits Authors Funding and support • Margaret Terry Orr, Bank Street College, (morr@bankstreet. edu)

Credits Authors Funding and support • Margaret Terry Orr, Bank Street College, (morr@bankstreet. edu) • Ray Pecheone, Stanford University • Liz Hollingworth, University of Iowa • Rabia Karatoprak, University of Iowa • Barbara Beaudin, Bank Street College • Jon Snyder, Stanford University • Joe Murphy, Vanderbilt University • Work made possible by the Massachusetts Department of Elementary and Secondary Education (ESE) • Funded through federal Race to the Top grants • Bank Street College, lead agency for this work • Show. Evidence (now Tumble. Feed, on-line assessment management

What are the considerations to determine leadership candidate skill development and readiness? What is

What are the considerations to determine leadership candidate skill development and readiness? What is ascertained? • Leadership skills and indicators • Skill level of initial leaders How candidates • Authentic work demonstrate skills? • Scoreable work products How skills are evaluated? • Rubrics and scoring systems • Trained, reliable skilled scorers

Conceptual foundation • Performance assessment as a means of authentic learning (Kolb, Mezirow) •

Conceptual foundation • Performance assessment as a means of authentic learning (Kolb, Mezirow) • Essential leadership practices • • Inquiry on practice (Leithwood, Anderson) Professional learning culture (Du. Four, Seashore, Wenger) Supervision and feedback (Danielson, Supovitz) Family and community engagement (Bryk, Epstein, Henderson) • Reflective practice in leadership development (Schon; Kottkamp and Osterman)

Theory of performance assessments for professional readiness • Attributes: • Authentic • Constructivist, educative

Theory of performance assessments for professional readiness • Attributes: • Authentic • Constructivist, educative • Scalable to reflect views of professional readiness • Use in teacher preparation and medical preparation • Efficacy depends upon the validity and reliability of the assessment system: • Components • Candidate supports • scoring

Massachusetts policy context for assessment development • Policy reform in educational leadership made possible

Massachusetts policy context for assessment development • Policy reform in educational leadership made possible by RTTT • Preparation program redesign • Multiple pathways to preparation and licensure • Preparation programs (university and non-university) • Administrative internship/apprenticeship • Panel review • Earlier commitment to performance assessment • 800 -1200 candidates licensed annually • Contracted with Bank Street College and their expert team to develop performance assessments for leaders

Ensuring valid and reliable Assessments: Development Process Design Implementation and standards setting Field trial

Ensuring valid and reliable Assessments: Development Process Design Implementation and standards setting Field trial Development and readiness Fall and spring pilot

Components of the Performance Assessment for School Leaders • Four assessment tasks • Represent

Components of the Performance Assessment for School Leaders • Four assessment tasks • Represent the primary educational improvement levers of school leaders • Based on MA Leadership Standards and Indicators • 3 -4 artifacts and commentary produced per task • 3 -4 rubrics with 6 -8 indicators for scoring each task

PAL Assessment Task Components • Four parts to the work for each task •

PAL Assessment Task Components • Four parts to the work for each task • Three types of products • Artifacts • Other documents • Commentary Investigate Assess Prepare • Four-point rating scales • • Beginning Developing Meeting Exceeding Act

Task 1: Leadership through a vision of high student achievement Steps Work products •

Task 1: Leadership through a vision of high student achievement Steps Work products • Collect and analyze 3 -5 years of quantitative student performance indicators, qualitative indicators of school culture and student learning. • Artifact #1—Priority area and its context • Artifact #2—Plan for action strategies • Artifact #3—Findings, feedback and recommendations • Commentary—leadership skills developed and learning experienced • Identify a priority student learning area of interest with attention to federally-designated priority student groups • Collect additional qualitative and quantitative information for the priority student learning area, • Document existing school programs, services, and practices and ascertain the gaps and opportunities for improvement. • Develop action strategies to improve student learning in the priority student learning area. • Seek leadership feedback and support

Task 2: Leadership for a Professional Learning Culture Steps • Document how teacher teams

Task 2: Leadership for a Professional Learning Culture Steps • Document how teacher teams and groups are used in the school. • Form a professional learning group of teachers • Select a priority academic area and focus for improvement • Engage group in meetings and support teachers collectively and individually on improving practice • Collect evidence on new or improved practices and analyze • Solicit team feedback on their learning and improvement Work products • Artifact #1—identify priority area, professional group and professional learning plan • Artifact #2—describe the learning process and work accomplished by the group, emphasizing candidate’s role • Artifact #3—present group member feedback on the process, learning and benefits • Commentary—leadership skills developed and learning experienced

Task 3: Leadership in observing, assessing and supporting individual teacher effectiveness Steps • Conduct

Task 3: Leadership in observing, assessing and supporting individual teacher effectiveness Steps • Conduct a pre-observation conference. • Document a teacher observation using a district guide on effective teaching practices • Describe teacher performance using the rubric indicator performance levels. • Conduct a post-observation conference that facilitates teacher rapport and learning • Provide constructive feedback and strategies for improvement in a post-observation conference • Collect and analyze teacher feedback on the effectiveness of the observation, feedback and support Work products • Artifact #1—Preobservation template • Artifact #2—Teacher observation video recording • Artifact #3—Post-observation meeting video recording • Artifact #4—Analysis of observed teaching • Artifact#5—Teacher feedback on observation and post-conference • Commentary—leadership skills developed and learning experienced

Task 4: Leadership for family engagement and community involvement Steps Work products • Identify

Task 4: Leadership for family engagement and community involvement Steps Work products • Identify a priority area for improving family and community engagement that would directly or indirectly enhance student learning in a priority area • Engage staff, leaders and family and community leaders as a planning group to enhance family and community engagement • Create a multi-strategy plan on how to improve family and community engagement. • Implement one planned strategy • Gather and analyze feedback and other evidence on the plan and strategy’s effectiveness for improving family and community engagement • Artifact #1—Analysis of data, priority area and plan • Artifact #2—Implementation of one strategy • Artifact #3—Feedback on plan and strategy implementation • Commentary—leadership skills developed and learning experienced

Field trial questions • Is each task a valid and reliable measure of a

Field trial questions • Is each task a valid and reliable measure of a key aspect of leadership practices? • Do the four tasks measure different aspects of leadership? • Are the four tasks measuring complementary parts of a comprehensive leadership construct? • Is scoring reliable? • Are the tasks free of bias in scoring?

The field trial process • One year (2014 -15) • 100% of eligible candidates

The field trial process • One year (2014 -15) • 100% of eligible candidates seeking initial school leader licensure • Complete all four tasks; No cut scores, only completion requirements • Yielded: 422 candidates who performed all four tasks • 82% from preparation programs • 65% female; 74% indicated white • 30 scorers • 25% of submissions were double scored

Are the tasks valid? Type of Score Task 1 Task 2 Task 3 Task

Are the tasks valid? Type of Score Task 1 Task 2 Task 3 Task 4 Total average score Number of Minimu of m domains indicators 6 3 1. 00 6 3 1. 17 8 4 1. 00 6 3 1. 00 26 13 1. 35 Maxim um Mean Std. Deviation 4. 00 2. 76 2. 97 2. 86 2. 66 . 553. 529. 472. 685 3. 58 2. 83 . 379

Task 1 Task 2 Task 3 Task 4 • Does each task have measurement

Task 1 Task 2 Task 3 Task 4 • Does each task have measurement integrity? When factor analyzed, the indicators loaded strongly and discretely as four independent factors—the tasks. • Are the tasks independent measures? (. 27 -. 35) • Do the task measures have coherence as an overall leadership score? (. 64 -. 74) Average total score Are the measures independent? Do they have coherence as one overall measure? Average Total Score Task 1 1. 000 -- -- . 696 1. 000 -- -- -- Task 2 . 710 . 350 1. 000 -- -- Task 3 . 642 . 320 . 315 1. 000 -- Task 4 . 736 . 284 . 340 . 267 1. 000 Measure

Are the measures reliable? Are they bias free? • Estimated reliability coefficients for each

Are the measures reliable? Are they bias free? • Estimated reliability coefficients for each task shows strong reliability for Tasks 1, 2 and 4 and weaker reliability for Task 3 (due primarily to low variance among candidates). • Initial evaluation of results show no different by program type but some differences by gender. There was insufficient information to compare results by race/ethnicity. Number of scorers Task 1 2 1 0. 728 0. 842 2 0. 656 0. 792 3 0. 208 0. 345 4 0. 581 0. 735

Considerations and related effects Assessment development Candidate and program benefits • Strengthen instructions, scorer

Considerations and related effects Assessment development Candidate and program benefits • Strengthen instructions, scorer training and supervision • Testing out use of cut-scores and fee payment on submission quality • Early work shows greater quality, narrowing the performance range • Fewer candidates are seeking licensure • Irregularity problem • Learning and readiness benefits reported by most candidates • Program alignment and improvement reported by many programs • Gradual reshaping school and district expectations for leadership preparation • Other states are interested in replication

Implementation Year Outcomes Implementation changes • Added a fee ($500 for initial registration) •

Implementation Year Outcomes Implementation changes • Added a fee ($500 for initial registration) • Added cut-scores (on 4 -point scale) • 2. 1 threshold for each task • 2. 5 total average cut-score for 2016 -17 • 2. 75 total average cut score for 2017 -18 Implementation outcomes • Fewer candidates completed all four tasks during the Program Year (n=153) • Tasks continue to be rated by candidates and programs as relevant, feasible, educative and beneficial. • Total average scores were slightly higher (12% scored below 2. 5 and very few below 2. 1 threshold) • Task 4 remained most challenging • Program candidates performed somewhat better than nonprogram candidates, especially for Task 1 and 4.

Conclusions and implications • The four tasks are valid and reliable assessments of initial

Conclusions and implications • The four tasks are valid and reliable assessments of initial leadership readiness • PAL has had positive intended outcomes: • Educative (yielded expected learning benefits for candidates) • Evaluative (identified candidates who are ready for initial leadership) • Beneficial for school settings (lead to improvements based on authentic work completed) • Informative (lead to positive changes in preparation programs content, organization and internships)

Further information • For information on the research, contact Margaret Terry Orr at morr@bankstreet.

Further information • For information on the research, contact Margaret Terry Orr at morr@bankstreet. edu