The Use of Evaluation in Agricultural Policy Making

The Use of Evaluation in Agricultural Policy Making: The Case of Mexico November 11, 2010 San Antonio, TX Alfredo Gonzalez Cambero FAO Salomón Salcedo Baca FAO Renato Olvera Nevarez Mexican Ministry of Agriculture

Importance of evaluation � Program evaluation exists because of the expectation its results will be used for: § Program improvement (by means of feedback): § Valuation of program adequacy § Measurement of impacts and performance § Detection of areas of improvement § Program termination § Accountability � If evaluations are not used, their funding should be used for other purposes.

Uses of evaluation � Conceptual: To better program or related issues. � Strategic/symbolic: understand a To persuade others or to use evaluation findings to gain a particular outcome.

Uses of evaluation � Process: Engagement in the process of evaluation (deliberated use during the evaluation process). § Creating and enhancing shared understandings § Increasing participants´ engagement, sense of ownership § Learning to learn. � Instrumental: Change in decision-makers´ understanding of a situation to provide information for instrumental reasons. To make overt decision making.

Uses of evaluation � Bottom-line: The use of the evaluation is measured by the impact of the evaluation itself: ◦ What did the evaluation help to change? ◦ To what extent changes are due to the evaluation findings and recommendations? � This subsumes evaluation. into: Influence of the Achilles' heel of program evaluation: The use of evaluation.

Influence of evaluation �Process based influence of evaluation: § Cognitive: Changes in understandings stimulated by discussion, reflection and problem analysis. § Affective: Individual and collective feelings of worth resulting from involvement. § Political: Creation of new dialogs to draw attention to social problems or to influence power relationships.

Factors affecting evaluation use and influence: Relevance of evaluation Credibility of evaluation Communication of evaluation Translation of evaluation results User involvement Timeliness of the report Political factors Quality of evaluation

Quality of evaluation Methodological quality Usefulness of findings Usefulness of recommendations: what and how Quality is a necessary condition for evaluation to be used.

For the evaluation to be useful … It should be: ü Participatory ü Impartial and independent, preserving the freedom and criteria of the evaluator. üImpartiality gives credibility to evaluation findings. üIndependence gives legitimacy to the evaluation. üObjective: Avoid unsolicited unsubstantiated criticism. flattering and ü Proposal oriented: Concrete and viable proposals related to the substance.

Context of the FAO-SAGARPA collaboration The Mexican Government instruments public policies that look for the wellbeing of the rural population. UNDAF PRIORITY AREAS Strengthened public policies and capacities in order to: a) Promote new employment opportunities and quality of labor. b) Bolster equal opportunity, sustained and competitive economic growth. Principles of sustainable development incorporated into national and regional programs. EVALUATION AND POLICY ANALYSIS To contribute to the sustainable development of the rural and fishing sector through greater efficacy and efficiency of the rural public policy. Judgment elements for program decision making, which are pertinent and timely. COINCIDENCE IN DEVELOPMENT OBJECTIVES 10

Evolution of evaluations: 1998 -2006 Evaluation Objectives To identify areas for improvement and to To contribute to capacity propose changes in the building for the evaluation design and operation of and analysis of rural Alianza para el Campo to public policy. increase its operating effectiveness and results. Approach ü ü Continuous analysis Prospective vision Practical usefulness Feedback for decision-making

• Attention to critical aspects identified by programs and by previous evaluations – continuous analysis • Better feedback and participation of program officials • Impact evaluation sets in • Technical support for program implementation extends to more states (Tab, N. L, Jal, Gto, Sin, Son, Zac, Chih, Nay, Tam and Tlax) Advancement towards impact and process evaluation. Mandated by Budget Law 2002 2003 – 2006 1998 -2000 • Verification of norms • Verification of financial and physical goals • Based on beneficiaries’ perception. Mandated by the Ministry of Agriculture 2001 Evolution of evaluations: 1998 -2006 • Advancement towards measurement of realized impacts • Incorporation of capacity building activities • Technical support for program implementation in trial states (Jalisco y Tabasco)

Topic Evolution of evaluations from Role of SAGARPA Little ownership of and feedback on evaluation reports by program officials. Technical support and capacity building Technical supports kicks in at a basic level (there is insufficient capacity for agricultural program evaluation) Follow up on recommendations No follow up on evaluations´ recommendation. to Greater interest by program officials to participate in the drafting of evaluation terms of reference and to discuss evaluation reports. • Workshops to share conceptual, methodological, and organizational guidelines. • Formal distance education to train evaluators. Implementation of recommendations´ follow up. Achievements Evaluation ownership and a better match between expectations and evaluation products. • Minimum quality standards and homogeneity in evaluations. • Network of trained evaluators available. Use of a system for recommendations as a tool to support the improvement of program operations.

New approach to evaluations Objective Timely and reliable information for program decision-making in order to improve public policy application and accountability: § To promote results based operation § To evaluate results § To promote the systematic use of evaluation findings § To improve accountability § Monitoring and evaluation of useful programs and government entities.

For the future … Current situation Program operating areas are not prepared. There is no information or monitoring systems for program operation. Lack of coherence in programs and alignment with superior policy objectives. Lack of coordination between agency suppliers and users of information. Challenges To foster a culture of evaluation in government entities which contributes to decision-making. To implement a Monitoring and Evaluation System. To adequately direct resources and to make programs more efficient and effective. To adequately direct activities on planning, budgeting, execution, and public expenditure control.

A virtual cycle through program life: Continuous feedback Evaluation Intermediate evaluations Design RESULTS BASED PROGRAMS Execution Planning Implementation Monitoring

Thanks Alfredo Gonzalez Cambero agonzalez@fao-evaluacion. org. mx Salomón Salcedo Baca Salomon. Salcedo@fao. org Renato Olvera Nevarez rolvera@sagarpa. gob. mx