Using Indicators to Measure Progress and Performance Dominique

Using Indicators to Measure Progress and Performance Dominique Senn, seecon gmbh Using Indicators to Measure Progress and Performance

Find this presentation and more on www. sswm. info Copyright & Disclaimer Copy it, adapt it, use it – but acknowledge the source! Copyright Included in the SSWM Toolbox are materials from various organisations and sources. Those materials are open source. Following the opensource concept for capacity building and non-profit use, copying and adapting is allowed provided proper acknowledgement of the source is made (see below). The publication of these materials in the SSWM Toolbox does not alter any existing copyrights. Material published in the SSWM Toolbox for the first time follows the same open-source concept, with all rights remaining with the original authors or producing organisations. To view an official copy of the Creative Commons Attribution Works 3. 0 Unported License we build upon, visit http: //creativecommons. org/licenses/by/3. 0. This agreement officially states that: You are free to: • Share - to copy, distribute and transmit this document • Remix - to adapt this document. We would appreciate receiving a copy of any changes that you have made to improve this document. Under the following conditions: • Attribution: You must always give the original authors or publishing agencies credit for the document or picture you are using. Disclaimer The contents of the SSWM Toolbox reflect the opinions of the respective authors and not necessarily the official opinion of the funding or supporting partner organisations. Depending on the initial situations and respective local circumstances, there is no guarantee that single measures described in the toolbox will make the local water and sanitation system more sustainable. The main aim of the SSWM Toolbox is to be a reference tool to provide ideas for improving the local water and sanitation situation in a sustainable manner. Results depend largely on the respective situation and the implementation and combination of the measures described. An in-depth analysis of respective advantages and disadvantages and the suitability of the measure is necessary in every single case. We do not assume any responsibility for and make no warranty with respect to the results that may be obtained from the use of the information provided. Using Indicators to Measure Progress and Performance

Find this presentation and more on www. sswm. info Contents 1. Introduction 2. What Are Indicators 3. The Selection of Indicators 4. Why Use Indicators 5. What to Use Indicators for 6. The Development of Indicators 7. Typical Indicators in SSWM 8. Advantages and Disadvantages 9. References Using Indicators to Measure Progress and Performance 3

Find this presentation and more on www. sswm. info 1. Introduction Every project and programme has specific objectives. In order to assess the progress and performance of the project (e. g. to analyse whether targets are met, to give account of the progress to donor agencies or to show accountability to the public, funders, donors and stakeholder groups), monitoring and evaluations need to be carried out. Indicators provide an effective tool for measuring progress and performance. Source: http: //www. clearhorizon. com. au/wp-content/uploads/2009/02/theproject 25 percent. jpg [Accessed: 23. 10. 2013] Using Indicators to Measure Progress and Performance 4

Find this presentation and more on www. sswm. info 2. What Are Indicators Definitions • OECD/DAC: “A quantitative or qualitative factor or variable that provides a simple and reliable means to measure achievement, to reflect changes connected to an intervention, or to help assess the performance of a development actor. ” • USAID: “A variable, which purpose it is to measure change in a phenomena or process. ” • UKAID & United States Institute of Peace: An Indicator “refers to a measure used to demonstrate change in a situation, or the progress in, or results of, an activity, project, or program. ” • European Commission: “A description of the project’s objectives in terms of quantity, quality, target group(s), time and place. ” Source: MDF 2005, UKAID & UNITED STATES INSTITUTE FOR PEACE n. y. Using Indicators to Measure Progress and Performance 5

Find this presentation and more on www. sswm. info 2. What Are Indicators Example Specific indicators have been developed in order to measure progress towards the Millennium Development Goals and Targets Indicators Goal 1: Eradicate extreme poverty and hunger Target 1. A: Halve, between 1990 and 2015, the proportion of people whose income is less than one dollar a day Target 1. B: Achieve full and productive employment and decent work for all, including women and young people Target 1. C: Halve, between 1990 and 2015, the proportion of people who suffer from hunger 1. 2. 3. 4. 1. 2. Proportion of population below $1 (PPP) per day Poverty gap ratio Share of poorest quintile in national consumption Growth rate of GDP person employed Employment-to-population ratio Proportion of employed people living below $1 (PPP) per day Proportion of own-account and contributing family workers in total employment Prevalence of underweight children under-five years of age Proportion of population below minimum level of dietary energy consumption Goal 2: Achieve universal primary education Target 2. A: Ensure that, by 2015, children everywhere, boys and 1. girls alike, will be able to complete a full course of primary 2. schooling 3. Net enrolment ratio in primary education Proportion of pupils starting grade 1 who reach last grade of primary Literacy rate of 15 -24 year-olds, women and men For the full list, see: http: //mdgs. un. org/unsd/mdg/Host. aspx? Content=Indicators/Official. List. htm Using Indicators to Measure Progress and Performance 6

Find this presentation and more on www. sswm. info 3. The Selection of Indicators (1/2) Generally an indicator focuses on a small, manageable set of information that gives a sense of the bigger picture. àThe choice of indicators is important as to whether it gives sufficient “sense of the bigger picture”. à Only well selected indicators can assist in maintaining a focus on the important work areas and take strategic decisions to address problem areas. àThe careful selection of indicators makes a project sustainable and allows responsible persons to act. Using Indicators to Measure Progress and Performance 7

Find this presentation and more on www. sswm. info 3. The Selection of Indicators (2/2) The term “minimum indicators” is used in recognition that: • It is better to start with a small set of indicators that are feasible to monitor and to improve over time. • There are many other indicators that could be used to measure progress beyond this basic level. Using Indicators to Measure Progress and Performance 8

Find this presentation and more on www. sswm. info 4. Why Use Indicators An indicator is used to answer the question: How do we know if and how much we have achieved our objective? Indicators are useful to: • Measure progress and performance over time against various water and sanitation objectives to evaluate the effect of policy actions and plans. • Provide information on the status, trend or performance of a given system to inform the project team (responsible organisation), stakeholders, the public and policy makers in a simplified way. • Translate data into policy relevant information (describe, show trends, communicate results of implementing objectives). • Identify areas for increased attention by an organisation. Using Indicators to Measure Progress and Performance 9

Find this presentation and more on www. sswm. info 6. The Development of Indicators Criteria for Developing Indicators Good indicators should meet certain criteria. Three different concepts are presented on the following slides: SMART indicators, SPICED indicators and CREAM indicators. While SMART describes the properties of the indicators themselves, SPICED relates more to how indicators should be used. ! But keep in mind: the most important point in developing indicators is: Don’t be too ambitious! Start with what can be realistically done or else failure is guaranteed. Using Indicators to Measure Progress and Performance 10

Find this presentation and more on www. sswm. info 6. The Development of Indicators SMART Indicators SPECIFIC The indicators need to be clearly articulated, well defined and focused. They should be clear to people with a basic knowledge of the issue, programme or initiative. M MEASURABLE The indicator should be measurable, that is, it has the capacity to be counted, observed, analysed, tested, or challenged. The indicators should be able to determine the degree of completion or attainment. Using the same methodology and information, findings should be repeatable and comparable. A ATTAINABLE The target attached to the indicator should be achievable within the scopes of the project/ programme. R RELEVANT The indicators should be able to detect change and be related to the specific situation they are “indicating” information about. They should be measurable at an appropriate scale. T TIME-BOUND The indicator should be attached to a time frame. Term dates for measurement should be included. S Source: Save the Children, https: //sites. google. com/site/savethechildrendme/Home/smart-indicators [Accessed: 17. 12. 2013] Using Indicators to Measure Progress and Performance 11

Find this presentation and more on www. sswm. info 6. The Development of Indicators SPICED Indicators S P I C E D SUBJECTIVE Informants have a special position or experience that gives them unique insights which may yield a very high return on the investigators time. In this sense, what may be seen by others as 'anecdotal' becomes critical data because of the source’s value. PARTICIPATORY Indicators should be developed together with those best placed to assess them. This means involving a project's ultimate beneficiaries, but it can also mean involving local staff and other stakeholders. INTERPRETED & COMMUNICABLE Locally defined indicators may not mean much to other stakeholders, so they often need to be explained. CROSS-CHECKED & COMPARED The validity of assessment needs to be cross-checked, by comparing different indicators and progress, and by using different informants, methods, and researchers. EMPOWERING The process of setting and assessing indicators should be empowering in itself and allow groups and individuals to reflect critically on their changing situation. DIVERSE & DISAGGREGATED There should be a deliberate effort to seek out different indicators from a range of groups, especially men and women. This information needs to be recorded in such a way that these differences can be assessed over time. Source: MDF 2005 Using Indicators to Measure Progress and Performance 12

Find this presentation and more on www. sswm. info 6. The Development of Indicators CREAM Indicators should be. . . C R E A M CLEAR . . . Precise and unambiguous RELEVANT . . . Appropriate to the subject at hand ECONOMIC . . . Available at reasonable cost ADEQUATE . . . Able to provide sufficient basis to assess performance MONITORABLE . . . Amenable to independent validation Source: UKAID & UNITED STATES INSTITUTE FOR PEACE n. y. Using Indicators to Measure Progress and Performance 13

Find this presentation and more on www. sswm. info 7. Typical Indicators in SSWM (Adapted from MUGA and MIHELCIC 2008) • Functional indicators (e. g. adaptability, durability and reliability of the system) • Economic indicators (e. g. capital costs, operation and maintenance costs and user costs) • Environmental indicators covering resource use (e. g. nutrient reuse) and emissions (e. g. emissions of carbon dioxide) • Social-cultural indicators (e. g. public participation, acceptance, and stimulation of sustainable behaviour) Using Indicators to Measure Progress and Performance 14

Find this presentation and more on www. sswm. info 7. Typical Indicators in SSWM Examples of Indicators for Water Sources Management • Water withdrawal: absolute or per-person value of yearly water withdrawal gives a measure of the importance of water in the country’s economy. • Ground water level development (rise or fall of groundwater levels, pollution). • Water sources quality (e. g. turbidity, minerals content, pathogens, chemical quality). Water quality assessment with different indicators (biodiversity, chemical substances, nutrients, toxics, sediments, etc. ). Source: http: //www. epa. gov/region 1/lab/reportcardintro. html [Accessed: 10. 12. 2013] Using Indicators to Measure Progress and Performance 15

Find this presentation and more on www. sswm. info 7. Typical Indicators in SSWM Examples of Indicators for Water Purification • Treatment efficiency indicator (e. g. removal of indicator bacteria) Removal efficiency of organic nitrogen in different treatment methods (1 -nitrification, 2 oxidation pond, 3 -chemical coagulation, 4 -chlorination, 5 ammonia removal, 6 -filtration, 7 -reverse osmosis). Source: http: //www. intechopen. com/books/watertreatment/relationship-of-algae-to-water-pollution-andwaste-water-treatment [Accessed: 10. 12. 2013] Using Indicators to Measure Progress and Performance 16

Find this presentation and more on www. sswm. info 7. Typical Indicators in SSWM Examples of Indicators for Water Use • Percentage or number of people not served with improved drinking water and extension of piped water supply. Number of household connections per 100 residents in urban sub-Saharan Africa. Source: WSP & WUP 2003 • Domestic water use efficiency (per capita use of water). • Agricultural water use efficiency and efficacy (water demand per irrigated area). • Industrial water use. • Awareness and actual use of water-saving appliances. Using Indicators to Measure Progress and Performance 17

Find this presentation and more on www. sswm. info 7. Typical Indicators in SSWM Examples of Indicators for Wastewater Collection • Percentage of wastewater collected to total wastewater production. Particularly in peri-urban and rural areas in developing countries, wastewater is rarely collected, representing high health risks to the people living in these aras. Source: http: //www. ens-newswire. com/ens/aug 2007/200708 -07 -01. asp Using Indicators to Measure Progress and Performance 18

Find this presentation and more on www. sswm. info 7. Typical Indicators in SSWM Examples of Indicators for Wastewater Treatment • Percentage or number of people not served with basic sanitation. Percentage of households having a sewerage connection in sub-Saharan African cities. Source: WSP & WUP 2003 • Number of people affected by water borne diseases (e. g. diarrhoea, cholera). • Investment in drinking water supply and sanitation. • Removal of pollutants. • Awareness and actual use of safe sanitation systems. Using Indicators to Measure Progress and Performance 19

Find this presentation and more on www. sswm. info 7. Typical Indicators in SSWM Examples of Indicators for Reuse and Recharge of Nutrients and Water • Percentage of reclaimed water used in agriculture to total agricultural water use. • Percentage of nutrients used from recycled sources. Proportion of recycled water to withdrawn water. Source: http: //www. potashcorp. com/annual_reports/2012/gr i/environment/#EN 8 [Accessed: 10. 12. 2013] Using Indicators to Measure Progress and Performance 20

Find this presentation and more on www. sswm. info 8. Advantages and Disadvantages • Failures can be detected and eliminated. • Indicators can be adapted to the circumstances. • It is not necessary to measure every single parameter to get a “bigger picture” over the whole progress. Using Indicators to Measure Progress and Performance • A set with too many indicators will tend to clutter the overview. • Measuring indicators is a longterm exercise. If the indicators are not defined properly, and if measuring is not done correctly and conscientiously, it will not work. 21

Find this presentation and more on www. sswm. info 9. References Business Dictionary. URL: http: //www. businessdictionary. com/definition/indicator. html#ixzz 2 m. Vtid. EG 6 [Accessed: 04. 12. 2013] MDF (Editor) (2005): MDF Tool: Indicators. Ede: MDF Training and Consultancy. URL: http: //www. undg. org/docs/11652/MDF-Indicators-Brief-%282005%29. pdf [Accessed: 04. 12. 2013] MUGA, H. E. ; MIHELCIC, J. R. (2008): Sustainability of Wastewater Treatment Technologies. In: Journal of Environmental Management 88, 437 - 447. URL: http: //www. ncbi. nlm. nih. gov/pubmed/17467148 [Accessed: 23. 10. 2013] TAYLOR, P. ; LIDEN, R. ; NDIRANGU, W. ; JIN, L. (2008): Integrated Water Resources Management for River Basin Organisations – Training Manual. Pretoria: International Network for Capacity Building in Integrated Water Resources Management (Cap-Net). URL: http: //www. cap-net. org/sites/cap-net. org/files/RBO%20 Manual. pdf [Accessed: 23. 10. 2013] UKAID & UNITED STATES INSTITUTE FOR PEACE (n. y. ): Indicator Module. URL: http: //dmeforpeace. org/sites/default/files/3. 9%20 Indicators. pdf [Accessed: 10. 12. 2013] WUP and WSP (2003): Better Water and Sanitation for the Urban Poor. Good Practice from Sub-Saharan Africa. Kenya: Water Utility Partnership (WUP) and The World Bank Water and Sanitation Program (WSP). URL: http: //www. wsp. org/sites/wsp. org/files/publications/330200725049_af. Better. Waterand. Sanitation. For. The. Urban. Poor. G ood. Practice. From. SSA. pdf [Accessed: 13. 09. 2013] ZEWO (Editor) (n. y. ): Outcome and Impact Assessment in International Development. Zewo Guidelines for Projects and Programmes. Zurich: Schweizerische Zertifizierungsstelle fuer gemeinnuetzige, Spenden sammelnde Organisationen (ZEWO). URL: http: //www. zewo. ch/impact/en/impact [Accessed: 23. 10. 2013] Using Indicators to Measure Progress and Performance 22

“Linking up Sustainable Sanitation, Water Management & Agriculture” SSWM is an initiative supported by: Created by: Using Indicators to Measure Progress and Performance 23