A radically different approach to evaluator competencies AES

A radically different approach to evaluator competencies AES 2010 Kate Mc. Kegg, Nan Wehipeihana, Jane Davidson.

Purpose of the competency framework �Inform and guide high quality, sound and ethical evaluation practice in Aotearoa/NZ �Provide broad guidance about evaluation standards �Enhance the professional accountability of evaluators and commissioners �Support the development of employment criteria for evaluator roles �Increase public awareness about what makes ‘good’ evaluation practice in Aotearoa/NZ �Ultimately about better quality evaluation

A broad quality framework – three pillars – much like others Competencies Framework Ethical Conduct and Guidelines • Evaluator competencies for evaluators in Aotearoa • Evaluator practice standards • Designation / accreditation • Assessment and Review • Ethical Guidelines and principles for the conduct and practice of evaluation in Aotearoa Evaluation Standards • What constitutes a high quality evaluation in Aotearoa? E. g. , utility, credibility, feasibility, validity, propriety, accuracy?

A broad quality framework – three pillars – with a unique underpinning Competencies Framework Ethical Conduct and Guidelines • Evaluator competencies for evaluators in Aotearoa • Evaluator practice standards • Designation / accreditation • Assessment and Review • Ethical Guidelines and principles for the conduct and practice of evaluation in Aotearoa Evaluation Standards • What constitutes a high quality evaluation in Aotearoa? E. g. , utility, credibility, feasibility, validity, propriety, accuracy? Treaty of Waitangi – Partnership, Protection, Participation

Process of development to date �Working Group formed (Māori and Pacific Portfolio and anzea Convenor) �Literature precis commissioned �Expert Caucus convened – Co-Chairs appointed (Jane Davidson and Nan Wehipeihana) �Working group development of draft framework �Consultation hui / workshops around the country �Feedback collated �Caucus to reconvene in December this year to consider feedback from consultation process �Redrafting and plan for release and implementation in first half of 2011

What’s radically different? �It’s all about the values! �Cultural values, particularly the relevance of the Treaty of Waitangi as our country’s founding document �Values as an integral part of evaluation-specific methodology �And, how these two intersect to drive the quality and value of evaluation in our unique context

It’s not called e. VALUation for nothing! �Clarity about how and why this is different from research, audit, monitoring, management consulting: �Evaluation-specific methodologies are key �i. e. relevant values Outcomes of value Dimensions of programme quality �Evaluative interpretation against the values

And more specifically… �A demonstrated understanding of the knowledge base informing the discipline and practice of evaluation �The knowledge and skill to: � frame evaluation questions which are explicitly ‘evaluative’ �Design evaluation that actually answer these �Evaluatively interpret information �Evidence & values-driven analysis, synthesis and interpretation is credible and valid to all

And finally…. �Evaluation reporting that: �Gets to the point - answers their questions �Is clear and transparent about methodological choices and evaluative interpretations made �credible and useful for the commissioner and others (stakeholders) involved in and affected by the evaluation

The centrality of cultural values in evaluation in Aotearoa / New Zealand �It’s about knowing ourselves (individually and collectively) - our roots, histories, biases, prejudices, and assumptions about race, culture, and ethnicity �It’s about personal responsibility and commitment to personal development and education about culture. �becoming culturally competent requires a willingness and ability to draw on the values of different cultures in order to appropriately and effectively meet key stakeholder needs �It’s as much about who we are, and where we position ourselves in relation to others, as it is about what we do

The draft framework �Actively tries to demonstrate a commitment to: �Ensuring that what differentiates evaluation from other professional activity is clearly articulated �ensuring the inclusion and participation of indigenous perspectives and worldviews in the development of evaluation competencies, standards and practices, and the integration of these perspectives into the profession. �Demonstrates a clear imperative to centralise the position of cultural competence

More importantly for evaluation and evaluators… �The common thread between culture and evaluation is the concept of values �Culture shapes values, beliefs, and worldviews �Evaluation is fundamentally an endeavour of determining values, merit, and worth �Therefore, in order to be a competent evaluator – one must also have a willingness and commitment to being culturally competent

The four domains Contextual analysis and engagement Reflective practice and professional development Cultural Systematic competency evaluative enquiry Evaluation project management and professional practice

Some tensions that have emerged… On the one hand… And on the other…. 1. What’s the problem? What are the benefits, who benefits and who is marginalized? 2. ‘The good or competent evaluator’ 3. Freedom to practice ‘The good or competent commissioner’ Professionalization 4. The role of the Treaty of Waitangi / indigenous rights The meaning of culture 5. Skills, attributes and abilities Personal and professional commitment / dispositions

Final thoughts �Cultural values and evaluation specific competencies are at the heart of evaluation quality in our view �Treaty relationships are central to our evaluation ‘culture’ �Values embedded in these relationships are integral to all other evaluation competencies �It’s an emergent strategy – review and reflect