Principal Evaluation August 2012 Iowa Joseph Murphy Vanderbilt

Principal Evaluation August 2012 Iowa Joseph Murphy Vanderbilt University

• IOWA • August 2012

te & licy a t S po t c tri dis Standards Dis tric evaluation Conditions of Work t& sta te p olic y governance incentives professional development preparation internship accreditation/program approval l i c e n s u r e Career Line induction (residency) professional development teacher leadership preparation relicensure mentoring Education Distri c state t & policy State Standards & licy t po distric

LEVERAGE POINT MODEL Leverage Point National data State of Affairs on Leverage Point Potential Improvement Strategies Iowa data

Think, Pair, Share 1. Describe the current evaluation system in your district. 2. List two things that you don’t like about it. 3. On a scale of 1 (not useful) to 10 (very useful), how useful is it in: showing you who your really strong principals are ensuring principal accountability providing targeted professional growth for principals promoting improvement at each school promoting district-level improvement initiatives

Principal Evaluation: The National Story

PRINCIPAL EVALUATION: The National Story PART A: Problems with Current Evaluation Designs PART B: “Construction Crew” Work in Other States

PART A: PROBLEMS WITH CURRENT EVALUATION SYSTEMS

Understudied Leverage Point

Content • Little evidence that systems evaluate what is important, i. e. not valid • Insufficient attention to leadership for learning, especially curriculum and instruction

Process • Limited architecture – focus on one approach (e. g. goals) • Perfunctory – not a deep process

Impact • Principals not receiving useful feedback • Not promoting professional growth of principals • Not promoting organizational improvement

PART B: CONSTRUCTION CREW WORK

Principal Evaluation “Construction Crew” Work STATES Components Goals Ohio Delaware New York Illinois Kentucky X X X School Improvement X Student Achievement X X X 360° Assessment X X X X ISLLC Domain Customer Satisfaction Professional Growth X X X

Learning-Centered Leadership

Leadership 1. Leadership matters. 2. In difficult times, leadership matters even more. 3. In periods of significant organizational transition, leadership is the major controllable factor in explaining organizational performance.

Leadership 4. Instructionally-focused and change- oriented leadership are especially effective frames for education. 5. Team leadership seems to offer promise for enhancing organizational performance. 6. Assessment provides a strategic leverage point for strengthening leadership.

• External conditions • Curriculum • Instruction • Culture Personal Characteristics Knowledge Leadership Behaviors Value-added • Standards Distribution • Accountability Level School Experience Student Success • Achievement • Graduation • College attendance • Post graduation success Values & Beliefs Classroom “Context” School • Type • Level • Nature • Student composition • Staff composition “precursors” Figure 1. Learning Centered Leadership Framework “’behaviors” District & State “influence pathway” “’outcomes”

Table 1– Knowledge Base for the Assessment System I. Vision for Learning II. Instructional Program III. Curricular Program IV. Assessment Program V. Communities of Learning VI. Resource Acquisition and Use VII. Organizational Culture VIII. Social Advocacy

I. Vision for Learning A. Developing vision B. Articulating vision C. Implementing vision D. Stewarding vision

II. Instructional Program A. Knowledge and involvement B. Hiring and allocating staff C. Supporting staff D. Instructional time

III. Curricular Program A. Knowledge and involvement B. Expectations, standards C. Opportunity to learn D. Curriculum alignment

IV. Assessment Program A. Knowledge and involvement B. Assessment procedures C. Monitoring instruction and curriculum D. Communication and use of data

V. Communities of Learning A. Professional development B. Communities of professional practice C. Community-anchored schools

VI. Resource Acquisition and Use A. Acquiring resources B. Allocating resources C. Using resources

VII. Organizational Culture A. Production emphasis B. Learning environment C. Personalized environment D. Continuous improvement

VIII. Social Advocacy A. Stakeholder engagement B. Diversity C. Environmental context D. Ethics

ISLLC Educational Leadership Policy Standards: 2008 As Adopted by the National Policy Board for Educational Administration

STANDARD 1: An education leader promotes the success of every student by facilitating the development, articulation, implementation and stewardship of a vision of learning that is shared and supported by all stakeholders Functions: • A. Collaboratively develop and implement a shared vision and mission • B. Collect and use data to identify goals, assess organizational effectiveness, and promote • organizational learning • C. Create and implement plans to achieve goals • D. Promote continuous and sustainable improvement • E. Monitor and evaluate progress and revise plans

STANDARD 2: An education leader promotes the success of every student by advocating, nurturing, and sustaining a school culture and instructional program conducive to student learning and staff professional growth. • Functions: • A. Nurture and sustain a culture of collaboration, trust, learning, and high expectations • B. Create a comprehensive, rigorous, and coherent curricular program • C. Create a personalized and motivating learning environment for students • D. Supervise instruction • E. Develop assessment and accountability systems to monitor student progress • F. Develop the instructional and leadership capacity of staff • G. Maximize time spent on quality instruction • H. Promote the use of the most effective and appropriate technologies to support teaching • and learning • I. Monitor and evaluate the impact of the instructional program

STANDARD 3: An education leader promotes the success of every student by ensuring management of the organization, operation, and resources for a safe, efficient, and effective learning environment. Functions: • A. Monitor and evaluate the management and operational systems • B. Obtain, allocate, align, and efficiently utilize human, fiscal, and technological resources • C. Promote and protect the welfare and safety of students and staff • D. Develop the capacity for distributed leadership • E. Ensure teacher and organizational time is focused to support quality instruction and • student learning

STANDARD 4: An education leader promotes the success of every student by collaborating with faculty and community members, responding to diverse community interests and needs, and mobilizing community resources. Functions: • Collect and analyze data and information pertinent to the educational environment • B. Promote understanding, appreciation, and use of the community’s diverse cultural, social, • and intellectual resources • C. Build and sustain positive relationships with families and caregivers • D. Build and sustain productive relationships with community partners •

STANDARD 5: An education leader promotes the success of every student by acting with integrity, fairness, and in an ethical manner. Functions: • A. Ensure a system of accountability for every student’s academic and social success • B. Model principles of self-awareness, reflective practice, transparency, and ethical behavior • C. Safeguard the values of democracy, equity, and diversity • D. Consider and evaluate the potential moral and legal consequences of decision-making • E. Promote social justice and ensure that individual student needs inform all aspects of schooling

STANDARD 6: An education leader promotes the success of every student by understanding, responding to, and influencing the political, social, economic, legal, and cultural context. Functions: • A. Advocate for children, families, and caregivers • B. Act to influence local, district, state, and national decisions affecting student learning • C. Assess, analyze, and anticipate emerging trends and initiatives in order to adapt • leadership strategies

Principal Performance-Based Evaluation Prototype 2011 -2012 Metropolitan Institute for Leadership in Education (MILE) and the GSU Teacher Quality Partnership (TQP) Grant Developed by Southland Administrators in Consultation with Dr. Joseph Murphy, Vanderbilt University and aligned with state of Illinois PERA 2010 Initiative Coordinator, Dr. Donna Joy

Guiding Principles of the Evaluation System Foundations • Highlight teaching and learning • Include student growth as a significant factor • Be standards based: ISLLC and Illinois Performance Standards for School Leaders • Underscore learning-centered leadership

Guiding Principles of the Evaluation System Process ● Be evidence based ● Have set benchmarks agreed upon in advance ● Be transparent ● Foster a culture of collaboration between the principal and the supervisor ● Be valid and reliable ● Be comprehensive but not overly complex ● Be both formative and summative ● Include multiple measures ● Tap into multiple constituents ● Have well-defined timelines ● Provide ongoing feedback to the principal ● Be site specific, connected to the needs of the specific school ● Be flexible enough to allow for adjustments

Guiding Principles of the Evaluation System Outcomes • Motivate principals to improve • Promote targeted professional growth opportunities • Promote school improvement • Enhance academic and social learning of students • Result in positive or negative consequences

Components of the Evaluation System • VAL-ED – 25% Assessment on ISLLC Standards/Illinois Performance Standards for School Leadership • Organizational and Professional Goals (25%) Assessment on meeting an organizational goal (520%) Assessment on meeting a professional development goal (5 -20%) • Student Growth (50%) Assessment on growth in student achievement

Components of the Evaluation System Component 1 (25%): • 360 degree assessment of principal’s leadership on instructional leadership • Provides nationally normed measures of principal performance • In-depth discussion of VAL-ED later this morning

Components of the Evaluation System Components 2 & 3 Organizational and Growth Goals – One each – 25% total Source of goals: • Student outcomes • District and school improvement plans • Board of education and superintendent objectives • Surveys (e. g. , of school climate, parent satisfaction, and so forth) • Previous evaluations (i. e. , data from assessments of components in the evaluation system) • Accreditation reports

Components of the Evaluation System Components 2 & 3 Organizational and Growth Goals – One each – 25% total Characteristics of Effective Goals: • Be linked to the Illinois Performance Standards for Schools Leaders and ISLLC Standards. • Be organizationally grounded and emphasize the direct contributions of the leader. • Be anchored in an analysis of multiple sources around relevant data. • Be specific, measurable, attainable, realistic, timely, and challenging. • Have a longitudinal focus. • Be mutually determined through collaborative dialogue. • Be collaboratively reviewed with frequent and specific feedback.

Components of the Evaluation System Growth In Student Achievement – 50% Source of Goals: • State norm-referenced and criterion-referenced tests • Promotion and high school completion • RTI/progress monitoring data • College entrance examinations • End-of-course examinations • Advanced placement examinations • Other local assessments

The Evaluation Process Summative Self. Assessment February 1 Summative Assessment March 1 VAL-ED Assessment December Goal Setting June 30 Formative Conference November 30

The Evaluation Process Goal Setting (by June 30) Steps: 1. Review data sources prior to meeting 2. Bring relevant data to the meeting 3. Discuss and agree on: Ø Goals Ø Sources of evidence Ø Performance measures

The Evaluation Process Formative Conference: (by November 30) Steps: 1. Examine evidence and discuss the progress on target measures. 2. Review any new data available. 3. Make adjustments to the goals as necessary. 4. Add resources and supports to make goals attainable.

The Evaluation Process VAL-ED Assessment (In December) Completion of the VAL-ED assessment by the supervisor, principal and teachers in December.

The Evaluation Process Principal’s Pre-Summative Self-Evaluation (by February 1) Steps: 1. Gather evidence 2. Reflect on progress 3. Provide written self-assessment

The Evaluation Process Summative Evaluation (by March 1) Steps: 1. Meet and discuss self-reflection of principal 2. Evaluator shares her/his perspective 3. Sign off, using scoring system

SCORING SYSTEM Components VAL-ED Performance (25%) Below Basic 1 Does Not Meet Expectations Basic Proficient Distinguished 2 Basic 3 Proficient 4 Exceeds Expectations

SCORING SYSTEM Components Organizational Performance Goal (5% to 20%) Misses the target by a considerable margin Approaches the target 1 Does Not Meet Expectations 2 Basic Meets the target 3 Proficient Significantly exceeds the target 4 Exceeds Expectations

SCORING SYSTEM Components Personal Growth Goal (5% to 20%) Misses the target by a considerable margin 1 Does Not Meet Expectations Approaches the target 2 Basic Meets the target Significantly exceeds the target 3 Proficient 4 Exceeds Expectations

SCORING SYSTEM Components Student Achievement Goal (value-added) (50% ) Misses the target by a considerable margin Approaches the target Meets the target 1 Does Not Meet Expectations 2 Basic 3 Proficient Significantly exceeds the target 4 Exceeds Expectations

Weights (sample scoring) VAL-ED Score Weight Sub-total 2 25% 50 15% 30 Organizational 2 Composite Rating 250/100 Goal 2. 5 Professional Growth Goal 2 10% 20 Student Achievement Goal 3 50% 150 100% 250 Total

Final Evaluation Rating Unsatisfactory Needs Improvement Proficient Excellent 1. 0 -1. 6 1. 7 -2. 4 2. 5 -3. 2 3. 3 -4. 0 The categories above are required by PERA The numbers above are an initial cut; they may need refinement.

I. Vanderbilt Assessment of Leadership in Education VAL-ED

The VAL-ED Instrument • The instrument consists of 72 items defining six core component subscales and six key process subscales. • Principal, Teachers, & Supervisor provide a 360 degree, evidenced-based assessment of leadership behaviors. • Respondents rate effectiveness of 72 behaviors on scale 1=Ineffective to 5=Outstandingly effective. • Each respondent rates the principal’s effectiveness after indicating the sources of evidence on which the effectiveness is rated. • Two parallel forms of the assessment facilitate measuring growth over time. • The instrument is available in both paper and online versions.

Purpose & Uses • The VAL-ED can be used as part of a comprehensive assessment of the effectiveness of a leader's behaviors. • The VAL-ED reports principal performance through ▫ Norm-referenced scores and ▫ Criterion-reference scores. • VAL-ED can be used annually or more frequently to: ▫ Facilitate a data-based performance evaluation, ▫ Measure performance growth, and ▫ Guide professional development.

Implementation • • Identify respondents and invite participation. Discuss use of results & confidentiality. Decide paper or online version. Time and Timing ▫ Average respondent requires 20 to 25 minutes. ▫ Schedule completion after respondents have had a reasonable time to observe/experience the leader’s work and its effects on the school. • Designate person(s) to manage collection and submission of response forms, if paper version used. • Ensure teacher confidentiality.

Directions for Completing Rating Scale

An Example Set of Responses

Instrument Development and Early Studies of Psychometric Properties

Assessing Learning-Centered Leadership: The VAL-ED vision… A leadership assessment system that has the following properties: • • Works well in a variety of settings and circumstances, Is construct valid, Is reliable, If feasible for widespread use, Provides accurate and useful reporting of results. Is unbiased, Yields a diagnostic profile for summative and formative purposes. Can be used to measure progress over time in the development of leadership, and • Predicts important outcomes.

Psychometric Evidence • Item and response scale development ▫ Based on review of learning-centered leadership literature and alignment to ISLLC standards. ▫ Critiqued by education leaders and leadership researchers. • Item sorting study ▫ Established content validity by asking education leaders to sort the items into 36 cells. • Cognitive interviews of paper/pencil version ▫ Two rounds of cognitive interviews in three districts each. ▫ Three respondents evaluated the format and items. • Nine-school pilot test (320/440 teachers = 73%) ▫ Estimated reliability of each of 12 scales. ▫ Established construct validity through factor analysis. ▫ Established face validity through questions to respondents.

Psychometric Evidence • Cognitive interviews of online instrument including revisions based on 9 -school pilot • Bias review ▫ Submitted to urban districts to evaluate language. • 11 -school pilot test (283 teachers for 57% response rate) ▫ Confirmed changes made after 9 -school pilot test.

National Field Trial

Design of the Sample • Target of 300 Schools ▫ 100 elementary, 100 middle, 100 high schools ▫ 150 urban schools, 100 suburban schools, 50 rural schools ▫ Of the 150 urban schools, 50 from Wallace grantee districts and 50 from Wallace grantee states ▫ 75 schools from each of four geographic areas • The Obtained Sample ▫ 60 districts (9 Wallace districts) ▫ 235 schools for principal data, 253 schools for supervisor data, 245 schools for teachers data ▫ 8, 863 teachers ▫ 218 schools for which there was data from all three respondent groups ▫ 50% of the schools had a 68 percent teacher response or better

Setting Performance Standards

Performance Level Descriptors • Distinguished ▫ A distinguished leader exhibits leadership behaviors of core components and key processes at levels of effectiveness that over time are virtually certain to influence teachers to bring the school to a point that results in strong valueadded to student achievement and social learning for all students. • Proficient ▫ A proficient leader exhibits leadership behaviors of core components and key processes at levels of effectiveness that over time are likely to influence teachers to bring the school to a point that results in acceptable value-added to student achievement and social learning for all students. • Basic ▫ A leader at the basic level of proficiency exhibits leadership behaviors of core components and key processes at levels of effectiveness that over time are likely to influence teachers to bring the school to a point that results in acceptable value-added to student achievement and social learning for some sub-groups of students, but not all. • Below basic ▫ A leader at the below basic level of proficiency exhibits leadership behaviors of core components and key processes at levels of effectiveness that over time are unlikely to influence teachers to bring the school to a point that results in acceptable value-added to student achievement and social learning for students.

The Bookmark Procedure • Twenty-two panelists: Ten principals, four teachers, four supervisors of principals, two researchers of school leadership, and two education policymakers • Item-ordered booklet based on the National Field Trial (items from Form A) • Panel met August 12 and 13, 2008

The Results • The cut between proficient and basic was set at 3. 60. • The cut between distinguished and proficient was set at 3. 77. • The cut between basic and below basic was set at 3. 42. • The impact data are ▫ 30% of the principals fall below basic ▫ 50% fall below proficient ▫ 70% fall below distinguished • While the panel was highly positive about the experience and felt comfortable with where they had set the proficient cut, some expressed concern about having set the distinguished cut too low and the below basic cut too high.

Post-Standard Setting Reconsideration • In a follow-up email, panelists were given three options to consider ▫ ▫ ▫ Option 1 – Leave the cuts where they are. Option 2 – Move the below basic cut to be consistent with the table that set the lowest value (3. 29) and the cut for distinguished consistent with the table that set the highest value (3. 87). Option 3 – The same as Option 2 except the distinguished cut is moved to 4. 0, requiring that a principal on average get a score of 4 on the 5 point scale. • Panelists were given impact data for each possibility across the three options. • All 22 panelists responded. ▫ 21 favored moving below basic to 3. 29 ▫ 18 favored moving distinguished to 4. 00 • The decision was to go with Option 3, thus the cut scores are 3. 29, resulting in 17% of principals below basic, 3. 60, resulting in 50% of the principals below proficient, and 4. 0 for distinguished, resulting in 14. 2% of principals designated as distinguished.

Score Reports

Interpretation of Rating Scale Results • Descriptive Analysis ▫ Total Score ▫ Core Components Subscale Scores ▫ Key Process Subscale Scores • Norm-Referenced Profiles ▫ Principal ▫ Teacher ▫ Supervisor ▫ Total respondent composite • Criterion-Referenced Profiles ▫ Distinguished ▫ Proficient ▫ Basic ▫ Below basic

Aggregated Effectiveness Ratings

Comparisons Across Respondent Groups

Comparisons Across Respondent Groups

VAL-ED and Professional Development

Cell-by-Cell Feedback

VAL-ED and Professional Growth • Cell-by-cell feedback highlights up to 6 potential areas of growth. • Behaviors from these 6 domains are listed. • Areas of growth provide principals with information about key targets for professional development.

Leadership Behaviors for Possible Improvement Example of a potential area of growth: