DIRECCIN DE PLANEACIN Y NORMATIVIDAD The Importance of

DIRECCIÓN DE PLANEACIÓN Y NORMATIVIDAD The Importance of the Institutionalization of M&E: The Mexican Case DE LA POLÍTICA DE EVALUACIÓN Gonzalo Hernández Licona 2013

Institutional milestones Mexico 1997 -2009 q 2000 Congress’ Decree: annual external evaluations to all federal programs q 2001 Evaluation Units within ministries q 2005 -6 Social Development Law q CONEVAL. National Council for the Evaluation of Social Policy q Multidimensional Poverty Methodology q 2006 Budget Law: Performance Evaluation System q 2007 Evaluation Guidelines q CONEVAL, Ministry of Finance, National Audit Office q 2009 Government Accountability Law

Challenges for constructing Evaluation system: an q Institutional: It’s almost impossible to have a public Evaluation system without a proper institutional arrangement: evaluation mandate, evaluation unit, feedback procedures for policy improvements, norms about transparency. q Technical: Suitable methodologies, trained evaluators, administrative records, information. Constructing an M&E system is a political and institutional task with technical elements…. not the other way round….

q Mandate: CONEVAL § Measurement of Poverty at the National, State and Municipality level § Evaluation of social programs and policies q Governance § CONEVAL is part of the Executive, but § The Board has 8 seats. The majority of Board members (6) are academic researchers elected by all the States, representatives from Municipalities, Congress and the Executive (44 votes) § Other two seats: M. of Social Development and Finance. § Technical and managerial independence

THE M&E SYSTEM

M&E: Budget decisions based on Results. ü Identify the main social problems to prioritize strategies and resources. • Measuring Poverty (various dimensions) ü Public Policy Actions § § Which programs can be related to the social priorities? Improvement of Programs and Policies • Evaluation of Programs and Policies

Measuring poverty: Social Development Law Dimensions for poverty measurement • Current income per capita • Educational gap • Access to health services • Access to social security • Quality of living spaces • Housing access to basic services • Access to food

Evaluation Guidelines for all institutions, together with the Ministry of Finance: The Demand for evidence was clearer now Planning Evaluation Annual Evaluation Plan National Development Plan Ministries’ Strategic objectives Results Consistency & Results Evaluations (SYS) Logical Framework: All Programs (SYS) Impact Evaluations Process Evaluations Policy Evaluations Recommendations’ follow-up (SYS) Annual Performance Reports (SYS)

Uses of the Information from the Performance Evaluation System 1 • Congress • M Finance • Presidency • States 2 Information for the budget allocation process Handing in to relevant actors + Meetings with Congressmen + Explanatory meetings 3 Release to press and civil society • Congress: Budget Comission • M Finance Website Press Civil society Work documents 4 Recommendations´ follow-up

Products and facts q q q q q There are poverty figures at a national, state and municipality level Almost 600 programs have Log Frameworks. All social programs plus others… We hired ECLAC to help us with the capacity building for the Log models. We have offered an evaluation course every year for policy makers and every 2 for researchers. 25% of all indicators are oriented to measure Results 140 programs are evaluated every year (now every two). There around 273 social programs and actions. This is 90% of the total budget. 258 one-page summaries every year. Between 2 -4 impact evaluations are done every year. CONEVAL budget is around $23 millions on even years (household survey); $16 millions on odd years. But there is more budget for evaluation within ministries. We can find on the internet: § Poverty estimates § All the evaluations § The program’s point of view about its evaluation § Each program’s Work Plan

INFORMATION: EXAMPLES

Evolution of multidimesional poverty, Mexico 2008 -2010 Millions of persons Social Deprivations 2008 10. 6 % 11. 7 millions Population whose income is below the wellbeing threshold 6 Access to 4. 1 food 4 4. 8 3. 5 2010 3. 2 10. 4% 11. 7 millions 2 0. 0 0 -0. 8 -2 Population whose income is below the minimum wellbeing treshold Poverty Educational gap -2. 9 -4 -2. 5 -2. 3 Quality and spaces of the dwelling Access to basic services in the dwelling -6 2008 44. 5 % 48. 8 millions Access to social security -8 -10 Extreme Poverty -9. 0 Access to health services Sorce: estimations of CONEVAL based on information of MCS-ENIGH 2008 y 2010 46. 2 % 52. 0 millions

Percentage of population living in poverty by municipality. Mexico, 2010 Sorce: estimations of CONEVAL based on information of MCS-ENIGH 2010 and the Census 2010

Performance Assessment of Social Programs The Traffic Lights System Each program’s evaluations results are rated according to the following categories: • Results • Design • Results Oriented Planning • Coverage and targeting • Operation • Perception of targeted population • Follow-up of recommendations • Logical framework • Budget compliance • Distributional impact

PROCAMPO’s Performance Assessment

Performance Assessment of Social Programs

USING THE RESULTS FROM M&E IN SOCIAL POLICY

Changes in social policy due to the M&E System q The government launched in January 2013 an q CONEVAL published that poverty and food aggressive strategy to reduce hunger and extreme (multidimensional) poverty. q In September 2013 the President sent a bill to Congress to reshape the way social security has been working for almost 50 years. q The fiscal reform of 2013 changed the way resources will be allocated between States and Municipalities. q Since 2009, the Ministry of Finance uses information from the evaluation process to elaborate the annual budget. q There is a constant public and in-house debate and analysis (Presidency, ministries, programs, Congress, media) about the public evaluations of social programs and policies. deprivation increased between 2008 and 2012, and the Government acknowledged it. q Multidimensional poverty estimations showed that the larger deprivation is the access to social security (61%) q The reform has the goal to reduce extreme poverty according to the new multidimensional poverty methodology. q There are now annual information for almost 270 social programs. For some of them there are impact, process, design evaluations. q All evaluations are discussed with policy makers and all of them are public.

Changes in social policy due to the M&E System q The budget for the Cement Floor program q The impact evaluation demonstrated was increased nearly 400% between 2007 and 2012. reductions of gastrointestinal diseases for children do to the program q The program Employment for the Youth was q The design evaluation of the program showed cancelled in 2009. q The food supplement of the Progresa- Oportunidades program was modified in order to have a better iron formulae in 2003. q The rural program Procampo is now less regressive, due to a design change in 2009. program were changes due to the evaluation. q Due to the design evaluation of the National Crusade Against Hunger, the strategy improved definitions, diagnosis and coordination tools various design and operational flaws at a considerable cost. q The impact evaluaton showed the iron had not been absorbed by children. q The distributional analysis of various programs illustrated that the program was very regressive. q In July 2013 CONEVAL launched and evaluation to the CNCH, which was discussed with the Ministry of Social Development and also was public. The goal: On going improvement of the strategy.

Various social programs and their budget have changed due to the “Evaluation. Improvement Mechanism”: Type of Improvement 2010/1 2011 -2012/2 Programs Relative Participation Programs/3 Relative Participation Improvement in programs´ activities or processes 19 16. 8% 35 47% Improvement in the services offered by programs 22 19. 5% 10 14% Reorientation of programs 71 62. 8% 26 35% Programs were merged - - 3 4% The program was cancelled 1 0. 9% 0 0% 113 100. 0% 74 100% TOTAL Information from the Diagnostics of the Indicators Matrixes for Results 2008 and 2010. Information from the Follow-up of Aspects Susceptibles of Improvement System. The estimation considers the aspects susceptibles of improvement attenden by a 100% to march 2012. /3 The estimations includes two programas that finished their obligations but are not in operation since 2011. /1 /2

CHALLENGES

Challenges for Mexico § Better coordination between evaluation institutions and budget allocation decisions based on results: Planning vs Budget; CONEVAL, Ministry of Finance and Ministry of Audit § More capacities on evaluation and measurement tecniques § Improve indicators of various programs § Indicators of quality of educational and health services § List of beneficiaries § Evaluation of legislation and norms’ changes § Evaluation of Sectors/Ministries. Strategic objectives. § Rigorous evaluations and transparency in states and municipalities. § Take more into account the information from evaluations in budgetary, operative and strategic decisions. § A closer approach with the Executive

SOME LESSONS

Lessons q Countries should find their own institutional arrangement: Chile, Colombia, South Africa, USA, Canada, Mexico, China, etc. q The balance of power between Congress and the Executive is important q Public pressure helps q Credibility is at the center of the institutional arrangement q Champions are always a key factor q Internacional help is important…but try to build your own path q Impact evaluations are not always the first step, more basic evidence is sometimes more important q South-South exchanges of knowledge and innovation q Keep the fine balance between Transparency and Policy Improvement. q On of the main users: Ministry of Finance

ANNEX

Why all these changes in a country where citizens usually mistrust government information and where political parties fight for resources? q Since 1997 there is a balance of power between Congress and the President. q Congress demanded the creation of an independent institution, CONEVAL, for the measurement of poverty and for evaluation. q Information on poverty and evaluation is public and transparent. q The Finance Ministry has been an important ally in the process. q CONEVAL and the Ministry of Finance produced Guidelines for the Evaluation of Programs. q The center of the Guidelines: Results q The Guidelines also created an “Evaluation-Improvement Mechanism” q Most ministries have cooperated in the design of the M&E system. q Fine balance between: Accountability and Policy Improvement. q The (autonomous) Statistical Office has invested a lot on data collection. q Good academic skills and suitable evaluation methodologies have been increasingly disposable for the past 15 years.

Evidence must be part of a planning system § Identify and measure social challenges § Analysis § What works? § Program design § Budget § Implementation § Monitoring & Evaluation Why do we need evidence? § Improve social policy § Make better decisions (management, design, budget…) § Accountability

Main Products Social Programs Evaluation Poverty Measurement and Analysis • Multidimensional Poverty Measurement (National and State Estimations) o o o o • • Economic wellbeing Education lag Access to health services Access to social security Quality and spaces of the dwelling Access to basic services in the dwelling Access to food Social cohesion Social Gap Index Poverty Labor Trend Index (ITLP) Poverty Maps Income Poverty Measurement • Annual Evaluation Plan • • Log Framework: Programs Consistency & Results Evaluations Impact Evaluations Complementary Evaluations Performance Evaluations Programs’ Performance Summary Social Development Programs’ Inventory Thematic Evaluations Policy Evaluations Recommendations’ Follow-up Document for Budgetary Considerations • •