Session 5 Selecting and Operationalizing Indicators Session Overview

Session 5: Selecting and Operationalizing Indicators

Session Overview § Selecting indicators § Common difficulties with identifying indicators § Operationalizing indicators § Tools to use to operationalize indicators

Session Learning Objectives By the end of the session, the participant will be able to: § understand the process for selecting indicators; § be aware of the common difficulties with identifying indicators; and § understand how to operationalize indicators.

Selecting Indicators

Steps in Selecting Indicators Step 1: Review the program framework Identify what needs to be measured. Good indicators start with good program framework. Start with the overall goals and objectives for the program.

Steps in Selecting Indicators Step 2: Develop a list of possible indicators Identify indicators for each level of results. You can use: § indicators from past years of the program § experience of similar programs § global/regional/national indicators § indicator guides

Steps in Selecting Indicators Step 3: Assess each indicator Each indicator should be: § § § § measurable practical reliable relevant useful for decision-making precise sensitive capable of being disaggregated

Steps in Selecting Indicators Step 4: Select the “best” indicators § Narrow your list to the final indicators. § Aim for an optimum set that meets management needs at a reasonable cost. § Limit the number of indicators for each objective/result to 2 -3. § Remember your target audiences, both external and internal.

Common Difficulties with Selecting Indicators

COMMON PITFALLS IN INDICATOR SELECTION § Indicators not linked to program activities. § Using outputs as outcomes. § Poorly defined indicators. § Data needed for indicator is unavailable. § Too many indicators!!!

Pitfalls with Selecting Indicators Indicator not linked to program activities IR: Expanded access to diarrhea treatment services Activities: Train providers in treating acute diarrhea Inappropriate Indicator: Percent of facilities with adequate conditions to provide care Better indicators: Number of clinicians trained, # of facilities with a trained provider, percent of clinicians with 100% on post-test The program is not aiming to affect facility conditions, only provider skills.

Pitfalls with Selecting Indicators Using outputs to measure outcomes Problem: routine monitoring data (outputs) are available from households but outcomes are needed for reporting. You collect: Number of farms which practice recommended biosecurity measures You need: Percent of farms that practice recommended biosecurity measures Routine monitoring data should not be reported as outcome data. Compile your routine data as well as information about your target area to turn outputs into outcome measures.

Pitfalls with Selecting Indicators Indicator poorly defined Activity: A radio campaign to provide information about AI prevention methods Inappropriate indicator: Percent population with AI knowledge Better indicator: Percent population that can cite top 3 ways to protect their families from AI “Knowledge of AI” is vague. What knowledge is critical for the prevention of transmission of AI?

Pitfalls with Selecting Indicators Data needed for indicator not available Data issue: Information on stock-outs may not be collected daily Inappropriate indicator: Percent of days per quarter that service delivery points have stock-out of drugs Better indicator: Percent of service delivery points that had a stock out of drugs at some time during the last quarter If relying on routine data, indicator definition will depend on how data are collected.

Pitfalls with Selecting Indicators Too many indicators!!! Rules of thumb: • one or two indicators per key activity or result (ideally, from different data sources) • not more than 8 -10 indicators per programmatic area • at least one indicator for every core activity (e. g. infection control, laboratory improvement, outbreak response) • use a mix of different data collection strategies and sources There is no set formula for how many indicators to use. Consider your budget, stakeholders, and program plans. Then consider: Is it reasonable? Feasible?

Adding Indicators to the Program Framework

Adding Indicators to Program Framework Input Activity/ Process Quantifiable 1. What you resources do to going in to your accomplish activities – the your things you objectives? budget for. Indicators (example) Output Outcomes Impact Immediate results from your activity – people trained, services provided Longer-term change in knowledge, attitude, behavior, etc. related to program goal Long-term, population level change. Can relate to a program or organization’s vision/mission statement How do you measure this?

Logical Framework: Training Activity INPUT PROCESS Time and skills to Conduct develop training new events biosecurity training curriculum for market owners 18 OUTPUT OUTCOME IMPACT Market Improvemen Reduction owners t in market in trained in conditions HPAI/H 5 N biosecurity 1 virus methods circulation

Adding Indicators Input Time and funding to develop new biosecurity training curriculum for market owners • Indicators (example) • Amount of funding for curriculum development Percent of staff for curriculum development Activity/ Process Conduct training events Number of trainings conducted Output Outcomes Market owners Improvement in training in market biosecurity conditions methods Percent of markets in Percent of which proper market owners biosecurity trained in methods have biosecurity been methods implemented Impact Reduction in HPAI/H 5 N 1 virus circulation in targeted areas Percent of poultry in targeted markets found to be infected with H 5 N 1

Small Group Activity Return to small groups and return to the logical frameworks from Session 3. As in the examples, fill in the indicators for your logical framework: § You must have indicators for every column. § REMEMBER: In general, output indicators are counts while outcome indicators measure longer term changes in percentages or proportions in knowledge, attitudes, or practices. After group work, share logic models with the larger group. You have about 30 minutes for this activity.

Small Group Projects Present your findings to the group.

Operationalizing Indicators

Operationalizing Indicators What does operationalizing indicators mean? Identify exactly how a given concept, result, or behavior will be measured. Challenges include: § subjective judgment § local conditions § unclear yardsticks

Importance of Operationalizing Indicators Operationalizing indicators helps ensure that: § everyone is using the same definitions; § units of measurement are the same; § everyone understands the indicators; § only the most significant program elements or achievements are being tracked; and § the chosen indicators will assist in appropriate decisionmaking and/or action-planning.

Exercise: Operationalizing Indicators In groups of 3 -4 people, review and discuss how these indicators were operationalized : § § § § § Percent of provinces and districts with at least 5 or more surveillance officers trained on disease surveillance. Number of poultry disease reports to central level per year. Percent of provinces that have an avian influenza and pandemic preparedness plan. Number (percent) of markets with active surveillance at least twice per month. In case of outbreak, percent of outbreaks responded by rapid response teams within 24 hours. Percent of provinces with an effective (staff-trained and PPE provided) rapid response team. Percent of provinces with effectiveness of trained rapid response team tested through simulation exercise or outbreak. Percent of AI specimens tested within 24 hours. Percent of targeted influenza specimens tested.

Indicator Reference Sheet The sheet provides detailed documentation for each indicator: § basic information § description § plans for data acquisition § data quality issues § plans for data analysis, reporting, and review

Example of an Indicator Reference Sheet