Evaluation 2010 Quality and Evaluation American Evaluation Association

Evaluation’ 2010 “Quality and Evaluation” American Evaluation Association San Antonio, Texas Use of Training in Evaluation Capacity Building Alexey Kuzmin, Process Consulting Company Moscow, Russia alexey@processconsulting. ru A. Kuzmin, Use of training in ECB

Training in ECB • Important • Not identical • Not enough A. Kuzmin, Use of training in ECB 2

What factors affect the use of evaluation training in evaluation capacity building? 1. Factors which cause organizations to become involved in evaluation training 2. Factors related to the utilization focus of the training 3. Factors related to the training itself 4. Factors related to the complementary learning strategies that reinforce use of the training 5. Factors related to the sustainability of training outcomes A. Kuzmin, Use of training in ECB 3

Factors which cause organizations to become involved in evaluation training include: • External pressure that creates a demand for conducting evaluation, as in a request from a client or donor; • Internal pressure, as in the need to better control field operations, monitor programmes, or inform strategic decisions; • Needs related to the current state of the organization’s development, as in “start-up", “renewal", or “formalization"; • The motivation of the top managers caused by dis -confirmation, lack of confirmation; feelings of anxiety, personal interest, curiosity, and so forth. A. Kuzmin, Use of training in ECB 4

Factors related to the utilization focus of the training include: • Clarity prior to the training, about the intended use of the training and the primary intended users; • Involvement of the primary intended users in the training design and/or the training Utilization-focused evaluation training (as opposed to evaluation training in general) is conducted for and with specific, intended primary users for specific, intended uses A. Kuzmin, Use of training in ECB 5

Factors related to the training itself include: • Consideration of the actual needs of organizations, especially their existing and required evaluation capacities, in the training design • Duration of the program • Incorporation of practice (doing a real evaluation) in the training design; • Use of a variety of training methods, including a combination of lectures and interactive methods. • The personal involvement, competence and enthusiasm of the trainer or trainers A. Kuzmin, Use of training in ECB 6

Factors related to the complementary learning strategies that reinforce use of the training include: • • Learning by doing and mentoring Peer learning Self-education Learning by sharing A. Kuzmin, Use of training in ECB 7

Factors related to the sustainability of training outcomes include: • Support from the top managers • The regularity and intensity of evaluation practice • Research background of both employees involved in evaluation and the leaders of organizations • The degree of localization of evaluation capacity in organization • The inclusion of evaluation functions in the responsibilities of a person or a group • The availability of financial resources is an important condition for all five groups of factors • The availability of professional resources is an important condition for training and complementary learning A. Kuzmin, Use of training in ECB 8

Evaluation Capacity Formula “If you don’t use it, you don’t have it” EC=C*P EC is evaluation capacity; C is evaluation competence; and P is ongoing evaluation practice. A. Kuzmin, Use of training in ECB 9

Toward a theoretical framework for an ECB intervention that includes a training component Stages of an ECB intervention Major contributions of activities undertaken Make thoughtful decisions about evaluation training Focus training on the intended use Design and conduct a quality training Provide complementary learning opportunities Contributes to the development of evaluation competence Contributes to the use of evaluation competence in the ongoing practice Sustain training outcomes Outcomes Increased evaluation competence . A. Kuzmin, Use of training in ECB Effective use of the evaluation competence in the ongoing practice 10

Checklist: Understand the needs • Does the organization experience any external or internal pressures to develop evaluation capacity? If so, what specifically are the pressures? • What kind of evaluation capacity does the organization need to respond adequately, and in a timely manner, to these external and/or internal pressures? How will evaluation knowledge and skill help the organization to respond? • Where is the organization in its lifecycle and what are its developmental needs in consequence of being at that stage? How can evaluation training meet those needs? • What are the motivations of the top manager(s)? Why are they interested in using evaluation training in their organization? How can training meet their individual interests? A. Kuzmin, Use of training in ECB 11

Checklist: Make sure that the training is utilization-focused • What is the specific intended use of the training? • Who are the primary intended users? • How can these users be involved in the training design? • How can they be involved in the training? A. Kuzmin, Use of training in ECB 12

Checklist: Design and conduct a high-quality training • Are the organization’s needs reflected in the training design? • Does the training design fit the existing and required levels of the organization's evaluation capacity? Is it neither too basic nor too advanced? • Is the training long enough to achieve its goals? • Does the training include practicing evaluation in real life situations? Does it include reflection on the participants' experience? • Are various training methods used in the training? Is the balance between lectures and interactive methods appropriate? • Are the trainers competent, committed and enthusiastic? A. Kuzmin, Use of training in ECB 13

Checklist: Provide complementary learning opportunities • Will the training participants have complementary learning opportunities in the course of the training? After the training? • Will these opportunities include learning by doing? Peer learning? Mentoring? Selfeducation? Learning by sharing? • Is there a quality control system in place to assure the adequacy of the participants' performance when they are learning by doing and/or by sharing? A. Kuzmin, Use of training in ECB 14

Checklist: Make the training outcomes sustainable • Do the top managers support use of the training results? Did they make specific decisions regarding this use? Did they involve the evaluation training participants in these decisions? • Does the organization practice evaluation on a regular basis? Does it provide opportunities for trainees to do evaluation work on a regular basis? • How is evaluation capacity localized in the organization? Is this capacity distributed widely enough to prevent its loss if people competent in evaluation leave the organization? • Are the evaluation functions included in the formal job descriptions or informal responsibilities of a person or a group? A. Kuzmin, Use of training in ECB 15

Checklist: Use financial resources effectively • Are sufficient funds available for a useful evaluation training? • What is the most appropriate allocation of available funds between the training and related activities, i. e. , follow up training, complementary learning, evaluation practice, information dissemination, etc. )? A. Kuzmin, Use of training in ECB 16

Short version: Kuzmin, A. (2009). Use of evaluation training in evaluation capacity building. In M. Segone (Ed. ), From policies to results: Developing capacities for country monitoring and evaluation systems. New York: UNICEF. http: //www. mymande. org/? q=content/policies-results Full version: Kuzmin, A. (2004). Exploration of factors that affect the use of evaluation training in evaluation capacity development (Doctorate Dissertation). Cincinnati, Ohio Union Institute and University. http: //gradworks. umi. com/31/91/3191344. html A. Kuzmin, Use of training in ECB 17