What Makes a Competent Evaluator What Makes an

What Makes a Competent Evaluator: What Makes an Evaluator Competent? DONNA PODEMS, PHD STELLENBOSCH UNIVERSITY

Key question What is the concern of not having standards and competencies, and what hope does a clear list of each provide?

The Joint Committee Program Evaluation Standards/AFREA The five categories of the PES: ◦ Utility ◦ Feasibility ◦ Propriety ◦ Accuracy ◦ Evaluation Accountability

CES/US/Degeval Utility Feasibility Accuracy Propriety Accountability (meta-evaluation) 5 standards with 30 rules/5 standards 25 rules AFREA Utility principle: for produced information and expected and provided results. Feasibility: for realism, cautiousness and efficiency. Precision and quality: for a relevant methodology related to the goal and the subject matter of the evaluation Respect of ethics: respect of legal and ethical rules. 4 standards with 34 rules

DAC Overarching consideration Purpose, planning and design Implementation and reporting Follow up, use and learning SEVAL Group standards into function categories Recognise not all have same weight (others imply this)

Evaluator Competencies

What is a competency? Knowledge, skills and abilities

Competencies All reviewed lists had some use for the three areas –most focus on evaluator Some challenges ◦ Clarifying the role of government—Conduct? Contract? Participate in the evaluation? ◦ Related to this: role of the M&E advisor ◦ Evaluators not likely to use this list ◦ Separating out monitoring from evaluation competencies (and their own related challenges) ◦ Note: IDEAS does not separate these

Key underpinning challenges In an emerging field that does not always agree on basic concepts, such as the best way to collect or analyse data, it is challenging to establish minimum competencies There is no empirical research that shows that if a person brings particular competencies, that this results in a useful evaluation or one that people from multiple different perspectives would consider acceptable.

Discussion Points Hiring an evaluator is not like buying a It is not always clear what will happen, there is no certainty, and there is no ‘perfect’ evaluation design – there is no perfect evaluator…

Potential Dangers of the List ◦ Control into the field and related political issues ◦ Limit evaluation practice ◦ Needs to work hand in hand with government wide M&E system ---competencies may bureaucratise evaluations

Potential Advantages ◦ Transparency on who is hired ◦ Guidance on what is, and how to be, an evaluator (African? )

The Essential Competencies for Program Evaluators

And an additional challenge. . . Public sector knowledge and skills Qual Evaluation knowledge and skills “Pick your subject Research area” knowledge and knowledge skills and skills Quant

Evaluation Competency Framework 3 areas Those that manage, those that advise and those that do 4 dimensions (1) overarching considerations, (2) leadership, (3) evaluation craft, and (4) the implementation of evaluations Each dimension is then divided into descriptive areas ◦ EXAMPLE: Overarching considerations are divided into three areas that focus on contextual understanding, knowledge, ethical conduct, and interpersonal skills

In South Africa. . .

Where We are Now with Competencies