AEA conference Belfast 10 November 2011 J Scheerensutwente

AEA conference Belfast, 10 November, 2011 J. Scheerens@utwente. nl Assessment for Effective Schooling

Contents Theories on effective school improvement Evaluation and feedback as an improvement mechanism What is the actual impact of evaluation and assessment? results from meta-analyses Tentative explanations of non-effectiveness; remedies

Control theories Synoptic planning Market mechanisms “retroactive planning” Alternative “non control” theories Self-organization Professional bureaucracy and loose coupling

Synoptic planning System level variables relevant from the perspective of the synoptic planning model: national curriculum planning infrastructure for program evaluation in education centralized structures, limited school autonomy school inspection

Market mechanisms System level variables relevant from the perspective of creating market mechanisms free school choice financial and managerial school autonomy privatization vouchers “high stakes” accountability arrangements

Retroactive Planning System level variables relevant from the perspective of retroactive planning a broad range of accountability provisions national assessments school inspection new public management: free processes control outcomes (institutionalization of) school self-evaluation (institutionalization of) school development planning Facilities for continuous professional development of teachers

Self-organization school autonomy recognition of educational cultures recognition of the importance of composition effects

Professional Bureaucracy and Loose Coupling Professionals resist rational techniques as planning and evaluation Little management required Teacher autonomy; initial training and enculturation in the profession most important coordination mechanism Goals and means are loosely coupled

Zooming in on retroactive planning Central place for evaluation and assessment Tests as operational goals Test results as basis for learning and improvement Test results as a basis for task related cooperation and “professional learning communities” and instructional leadership Teaching to the test

Retroactive and Synoptic Planning characteristics synoptic planning Retroactive planning initial activity formulate encompassing goals Assess organization’s functioning choice of means and methods deduce from scientific knowledge Induce as improvement of weaknesses in current functioning scope a broad scope encompassing all major aspects of the organization a partial “piecemeal” approach time-frame long term short term organizational structure bureaucracy Learning organizational participation top-down Participative

Key expectations retroactive planning Output control Motivational role; judgmental feedback Informative role: diagnosis, instrumental feedback, improvement

Definitions Evaluation as a general term for evidence based judgment of programs, organizations and people Monitoring registration and evaluation associated with administrative data Assessment associated with performance measures (tests) of students Evaluation and Monitoring may include assessment

Feedback A feedback loop to control human behaviour involves four distinct stages. Firstly - Evidence. A behaviour must be measured, captured, and data stored. Secondly - Relevance. The information must be relayed to the individual, not in the raw-data form in which it was captured but in a context that makes it emotionally resonant. Thirdly - Consequence. The information must illuminate one or more paths ahead. Fourthly - Action. There must be a clear moment when the individual can recalibrate a behavior, make a choice, and act. Then that action is measured, and the feedback loop can run once more, every action stimulating new behaviors that inch the individual closer to their goals. (Source Wikipedia)

Results from meta-analyses Results earlier meta-analyses J. S. and co- authors Results other Meta-analyses Results recent meta-analysis, Hendriks, Scheerens and Steen, 2011

Scheerens et al. 2007 (monitoring & Evaluation at school level) N publications N replications Fisher’s Z 43 194 . 06 School self evaluation School wide performance assessment School performance feedback Evaluation of school policies

Scheerens et al. 2007 (evaluation & monitoring at classroom level) N Publications N replications Fischer’s Z 46 . 086 Teachers monitoring student work Monitoring and feedback Use of evaluative results Progress monitoring system

Results other meta-analyses Seidel and Shavelson, 2007, report an effect size of . 01 (N 84)for evaluation of learning and. 02 (N 143)for regulation and monitoring (classroom level) Effect size Fisher’s Z Creemers and Kyriakides, 2008, report an effect size of. 13 for evaluation of teaching policies at school level and. 18 for assessment Hattie, 2009, reports much higher effect sizes, see subsequent slides

Results John Hattie, 2009 FEEDBACK 23 meta-analyses ES=. 73 FREQUENT TESTING 8 meta-analyses ES =. 34 FORMATIVE EVALUATION 2 meta-analyses ES =. 90

Possible explanations of high effect sizes reported by Hattie Several studies have quite divergent dependent and independent variables Suggestion In Hattie, 2009, p. 202 that studies did not adjust for student prerequisites Regional differences, Anglo Saxon versus European studies

Results Hattie, minus divergent metaanalyses FEEDBACK 5 meta-analyses ES =. 42 FREQUENT TESTING ES =. 33 2 meta-analyses FORMATIVE EVALUATION 1 meta-analysis ES =. 70

Results meta-analysis Hendriks at al. , 2011(school level) N studies N replications Fisher’s Z 7 61 . 076 School self evaluation School wide performance assessment School performance feedback Evaluation of school policies

Results meta-analysis Hendriks at al. , 2011(evaluation class level) N studies 14 N replications 46 Teachers monitoring student work Monitoring and feedback Use of evaluative results Progress monitoring system Fisher’s Z. 070

Results meta-analysis Hendriks at al. , 2011(Assessment class level) N studies N replications Fisher’s Z 7 38 . 026 Use of tests Frequency of testing Assessment op pupils Review of achievement results

Discussion of results Small effects of school and classroom evaluation on student achievement Somewhat higher effects when feedback is explicitly referred to Impact of study characteristics; ( higher effect sizes for experimental studies, for language and reading as compared to other subjects and for studies carried out in the USA)

Why relatively low impact? (substantive explanations): Mixed quality of evaluation/assessment instruments Mixed “completeness” in following PDCA cycle Unfamiliarity of schools in using tests Low priority and Resistance

How to improve use by schools? Indicate the ideal situation: characteristics of an evaluation based learning organizations Optimize supply side, instruments, information provision, professional development and training Analyze resistance and reasons for non-use by teachers and schools

An evaluation based learning organization certain desirable characteristics of the information (valid, reliable, useable); certain desirable characteristics concerning the cognitive capacities of the receivers of the information; facilitating organizational structural conditions, for instance opportunities to discuss the evaluation results, support from the school leader, technical support services; supportive cultural conditions, i. e. a majority of staff that have favorable attitudes and dispositions and dynamics that maintain and support an overall positive, learning oriented climate.

Theory Type of constraint to organizational learning Micro economic theory Individual members have personal objectives, some of them egoistic and opportunistic Bounded rationality Limited information processing capacity of individuals Images of the school as an organization Fragmented structures, loose coupling, professional bureaucracy Personality characteristics relevant Personal and collective resistance, to the application of rational anxiety and defensiveness techniques Unconscious self-referential processes in the latent organizational culture Inhibitory loops to learning (Argyris & Schön, 1978)

Relevant personality traits Personality trait Related behavioral disposition Anxiety Evaluation apprehension; ambiguity tolerance Egoism Opportunism Introversion Degree of openness Locus of control Sense of efficacy General Intelligence Information processing capacity

Hypotheses: to the degree that a critical percentage of teachers are externally oriented, and have a low sense of efficacy evaluation results are more likely to be underutilized to the degree that a critical percentage of teachers show high levels of anxiety, evaluation apprehension and low ambiguity tolerance defensive reactions towards evaluation findings are more likely to the degree that a critical percentage of teachers is oriented to personal objectives evaluation results are likely to be used as part of power games to the degree that a critical percentage of teachers is high on introversion unilateral actions are more likely to inhibit collective learning and decision making on the basis of evaluation findings

Critical issues for “positive” and “negative” cycles the “task related” vs the “judgmental” purpose of the school self-evaluation, and providing clear information on this real or perceived inequalities in the amount of information that is made available at various levels in the hierarchy the transparency of the evaluative information and the sense of ownership that teachers have with respect to the data

Cognitively induced paths to underutilization information is global and vague – does not make sense to teachers – is delegated upwards or is caught in a loop of enduring vagueness feedback lacks an instrumental component – insufficient handles to use at classroom level – non use feedback is judicial rather than task oriented – defensive reactions –non use/political use

Conclusion: important principles for evaluation management Clarification of the purposes of the (self) evaluative activities, concerning task related and judgmental aspects. b) Scaling down the level of ambition with respect to linear and instrumental use of evaluative information. c) Breaking inhibiting loops on the basis of higher levels of reflection a)

Overall conclusion Meta analytic results partly influenced by imperfections of individual studies Potential of practical applications can be enhanced “Assessment for School Effectiveness” strong mechanism, no silver bullet, importance of being embedded in other effectiveness enhancing conditions, achievement orientation, instructional leadership and others.