Class6 Tool for Policy Making and Policy Evaluation

Class-6: Tool for Policy Making and Policy Evaluation Dr. M. K. Satish

POLICY EVALUATION

Policy Evaluation � Does the Government really know what it is doing? � THEY KNOW How much money is spent � How many beneficiaries � Program organization, management � Interest groups � � WHAT THEY DON’T KNOW Beneficial effects � Effects immediate or long range � Relationship between costs and benefits �

What needs to be looked into � Measuring Impact, Not Output � Target Groups: Impact on target group- poor, sick � Non-target groups: Spillover effects � Short term and long term effects � Indirect costs and benefits

How Government evaluates Policy � Hearing and Reports � Site Visits � Program Measures: e. g. no of recipients, no of hospital beds � Comparison with professional standards: e. g. . Water supply- 144 lpcd � Evaluation of citizen complaints

What Government can do to improve evaluation � Before versus after comparison � Projected trend line versus post program comparisons � Comparison between jurisdiction with and without programs � Comparisons between Control groups before and after program implementation

Policy Research by Governments raise many important questions � BIAS TOWARDS POSITIVE RESULTS: Product results supportive of policy � � THE HAWTHRONE EFFECT � � People behave differently when they know they are being watched. GENERALIZING RESULTS TO THE NATION � � Fudge the data, reinterpret the data Results of small-scale experiments may differ substantially from those occur at a national scale POLITICAL INTERPRETATION OF RESULTS

Program Evaluation: Why it fails so often? � Lack of clarity in goals, target groups and desired effects � Many programs and policies have primarily symbolic value � They do not actually change the conditions of target groups but merely make these groups feel that the government cares � Government agencies have a strong vested interest in proving that their programs have a positive impact � The pressure of day-to-day business generally takes priority over study and evaluation � Program evaluation requires funds, facilities, time and personnel � Govt agencies do not want to sacrifice the above

How Bureaucrats explain negative findings � The effects of the program are long range and cannot be measured at the present time � The effects of the program are diffused and general in nature; hence no single criterion or index adequately measures what is being accomplished � The effects of the program are subtle and cannot be identified by crude measures or statistics � The fact that no difference was found between persons receiving the services and those not receiving them means that the program is not sufficiently intensive and indicates the need to spend more money/resources on the program � The failure to identify any positive effects of a program is attributable to inadequacy or bias in the research itself and not in the program

Why government programs are seldom terminated? � Government programs are rarely terminated � Even policymakers are fully aware of fraud, waste and inefficiency Few General Reasons � � � CONCENTRATED BENEFITS, DISPERSED COSTS � Limited beneficiaries are concentrated in a small, well-organized constituency � Costs are dispersed over a large, unorganized and uninformed public: E. g. Tax Payer LEGISLATIVE AND BUREAUCRATIC INTERESTS � Bureaucratic jobs depend on programs continuation � Closing a department is not easy INCREMENTALISM AT WORK � Government hardly considers program as a whole � E. g. : Poverty is not seen from a holistic perspective

The Limits of Public Policy � Some societal problems are incapable of solution because of the way in which they are defined � Expectations might outweigh the capabilities of the government � Policies that solve the problems of one group may create problems for other groups � Solutions to some problems may require policies that are more costly than the problem. E. g. status quo is much better than intervention � The political system is not structured for completely rational decision making process

Logical Framework Analysis

Problem Tree

Logical Framework � Logical Framework provides a � � Systematic structure for identification, planning and management of projects � Developed in a workshop setting � With principal interest groups ADVANTAGES OF THE LOGICAL FRAMEWORK � � Problems are analysed systematically The objectives are clear, logical and measurable The risks and conditions for success of a project are taken into account There is an objective basis for monitoring and evaluation

Project Planning Matrix � 4 x 4 matrix Goal Indicators MOV (Means of (Objectively verification) verifiable Indicators (OVIs) Purpose Indicators Mo. Vs Outputs Indicators Mo. Vs Activities (Inputs) Indicators- “Milestones” Mo. Vs Assumptions/Risks

Logic Model � The Core of Logical Framework is the “Logic Model” � This takes the form of a series of statements If these activities are done and these assumptions are true, then these outputs will be delivered � If these outputs are delivered and these assumptions are true, then this purpose will be achieved � If this purpose is achieved and these assumptions are true, then this goal will be achieved �

Logical Framework Logic modeling is based on mapping and defining linkages between what we do and why we do it. Series of If-Then Relationships IF I Work Out for One Hour Each Day INPUTS THEN IF I Will Burn More Calories Than I Consume THEN Lose Fat and Build Muscle OUTPUTS IF THEN IF Improve My Looks and Health Outcomes Assumptions: improving looks = better self image Jasani Center for Social Entrepreneurship and Sustainability, NMIMS Mumbai THEN Have Better Image, Feel Better & Live Longer Impact

Logical Framework What is your goal at your annual dental check-up? IF THEN üToothpaste üFloss üTooth brush IF üBrush twice a day üFloss once a day THEN IF Remove plaque (decrease plaque in my mouth) THEN Have fewer (ideally zero) cavities INPUTS OUTCOMES Assumptions: Plaque causes tooth decay Jasani Center for Social Entrepreneurship and Sustainability, NMIMS Mumbai Impact

Logical Framework Impact Outcom e Output Input Related to Goal Malnutrition rates amongst young children reduced % of young children Related to Objectives (or getting appropriate Purposes) complementary food Related to Outputs Related to Activities/Resources X number of mothers know about good complementary food and how to prepare that Nutritional education to mothers on complementary food Jasani Center for Social Entrepreneurship and Sustainability, NMIMS Mumbai

Log Frame- Goal TITLE Orissa Modernising Economy, Government and Administration (OMEGA) Programme GOAL Indicator Enhanced ability of Government of Orissa to reduce poverty and manage environment responsibly (or enhance environment sustainability) Rate of poverty reduction in Baseline Milestone The state Backward districts Source: NSSO/ State BPL Survey Indicator Women work force participation rate Source: NSSO Target

Log Frame- Purpose PURPOSE Indicator Enhanced Go. O capacity to generate higher and pro-poor economic growth and employment % revenue e. Growth To be of VAT/ GST revenue developed relative to growth of state domestic product Source: State Budget/ Annual Financial Statement Indicator Baseline Milestone Target Reduction in number of poor Baseline Milestone (Year 1) Milestone (Year 2) Target (Year 3) To be developed Source: NSSO/Surveys Indicator Industrial performance in the state in terms of- Cost of doing business; manufacturing growth Baseline To be developed Milestone Source: Business Survey by Various Agencies Target Assumptions 1. Impact of Global economic slowdown is well tackled 2. Go. O’s commitment to fiscal and budgetary reforms continue

Log Frame- Output-PDS OUTPUT 2 Indicator Baseline Milestone Target Assumptions Poor people’s access to selected poverty alleviation programs increased Delivery of targeted schemes – National Rural Employment Guarantee Scheme (NREGS) and Public Distribution System (PDS) PIP (Performance Improvement Plan) - including gender and convergence milestones prepared. 40% of the PIP milestones achieved 60% 80% 1. Government remains committed to reforms and transparency. 2. Civil society organisations are able to support the citizen participation activities. Citizen participation: (a) inadequate (a) % of citizen aware of their entitlements/ rights; 25% 50% improvement from baseline 80% b) No. of grievances (b) weak; recorded, and (c) % of grievances redressed (all information gender disaggregated). Environmental improvement plan (EIP) in relation to NREGA (c) low (baselines to be established) Environment study on NREGA initiated for selected districts. Environment 40% of the EIP 75% study milestones completed; achieved. EIP agreed, based on the study.

Log Frame-Inputs INPUT PDS Indicator Baseline Funds to develop To be IT Systems developed Funds for monitoring Citizens awareness campaigns Process Improvement • Procurement • Human Resource Supporting policy to prevent corruption. Ombudsmen Milestone Target Assumptions

Value for money � Value for money is a term used as a synonym for cost-effectiveness � Four Key Terms � ECONOMY : Inputs have been procured at the least cost for the relevant level of quality � EFFICIENCY : Value of outputs in relation to the total cost of inputs � EFFECTIVENESS: Achieving program outcomes in relation to the total cost of inputs � EQUITY : Ensuring that the benefits are distributed fairly

Vf. M-DFID Approach

Vf. M- Framework

Methods to determine Vf. M � COST EFFECTIVENESS ANALYSIS � � The technique compares the relative costs to the outcomes (effects) Cost: in INR Effectiveness: in physical units CE Ratio= C 1/E 1 – Cost per Unit of Effectiveness (ranking from lowest to highest) � or EC Ratio= E 1/C 1– Effectiveness per unit of cost (ranking from highest to lowest) � C 1= Cost of Option 1, E 1= Effectiveness of Option 1

Methods to determine Vf. M COST BENEFIT ANALYSIS � Cost-benefit analysis (CBA) is a technique used to compare the total costs of a programme/project with its benefits, using a common metric (most commonly monetary units) � This enables the calculation of the net cost or benefit associated with the programme � To evaluate the overall impact of a programme in quantifiable and monetised terms � Example: � Cost: Rs 10 lakh spend on providing enhanced skill for 10 people over a period of 2 years � Benefit: Rs 2, 000 per month is the additional income earned by each individual over the 2 year period

Methods to determine Vf. M SOCIAL RETURN ON INVESTMENT � Social return on investment (SROI) is a principles-based method for measuring extra-financial value (i. e. , environmental and social value not currently reflected in conventional financial accounts) relative to resources invested SROI = (Tangible + Intangible) / (Time + Money) SROI can be used for a range of evaluation purposes � Assess Projects (Forecast) � Demonstrate achievements (Evaluative)

Methods to determine Vf. M 7 principles of SROI � Involve stakeholders � Understand what changes � Value things that matter � Only include what is material � Do not over-claim � Be transparent � Verify the result

Thank You