Frameworks 1 March 2011 Monitoring and Evaluation of

Frameworks 1 March 2011

Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Session Outline § Introduction to frameworks § Conceptual frameworks § Results Frameworks § Logical Framework § Hands on activity: developing frameworks Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

-- Why frameworks Designing M&E Frameworks assists in the development of § Clearly understood program/project goals and measurable, long-term, short-term, and intermediate objectives § Clearly defined relationships between program/project inputs, processes, outputs, and outcomes, § Understanding between program/project activities and the external context (environmental factors) § Sound implementation of programs § Design sound M&E plans Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Characteristics of frameworks § All types of M&E frameworks: § Inter-relate components, levels and directions of action § Enable understanding of how § programs influence health outcomes (conceptual framework) § programs should operate (logical framework) § Programs achieve goals (results) § Facilitate consensus building around a common paradigm Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Types of frameworks § Different origins § Many types: § Conceptual § Results § Logical & Logic model § Specific types § Results Based Management framework Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Learning Objectives: Conceptual Framework § At the end of this session, participants will be able to: § Understand why and how conceptual frameworks are useful for understanding programs and planning M&E § Describe the role of conceptual frameworks in program design, program evaluation, and evaluation research. § Describe the components of conceptual frameworks § Design a conceptual framework to be most useful for M&E planning § Design a conceptual framework for an intervention program Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

§ Conceptual framework § Research framework Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Group work- 1 § § § § § Why did we have political revolt in Egypt Why do we have poverty in province B Why do we have crime in City Z Why do we have famine in Country X How come we have street abortion in South Africa Why do we have teenage pregnancy in Community B How come Spain won the FIFA soccer world cup How come China has a booming economy How come the people of Thailand are so hospitable How did Uganda manage to control an epidemic Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

§ What are concepts Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

What is happening? Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

What is happening § World § Society/community § Workplace § hospital Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

When we conceptualise § Definitions § What is the meaning of the word or term or idea § What is the actual use of the word § What are the boundaries within which a term operates § What are the contrary examples of the word Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Concepts § Enable interpretation of a subject § Analyse complex subject matter § Synthesize separate pieces of a subject matter into a unit § Perceive similarities and differences § Enable us to make value judgments § Extend our knowledge Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

In program design/programming § What do you need to know § Why do have to know Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

M&E CONCEPTUAL FRAMEWORKS Conceptual, or “research”, frameworks are diagrams that identify and illustrate the relationships among all relevant systemic, organizational, individual, or other salient factors that may influence program/project operation and the successful achievement of program or project goals. Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Purpose of conceptual frameworks § Provides a perspective for understanding program objectives within a complete context of relevant factors in a program’s operating environment § Clarifies analytical assumptions and their implications for program possibilities or limitations on success, as well as measuring and analyzing that degree of success Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Purpose of Conceptual Framework: cont. § Makes explicit connections among relevant contextual (environmental) factors and your program § Helps to clarify the “why” and “how” questions of program operation and design: § Assumptions that underlie the chosen activities § Factors the activities are expected to affect § Causal linkages leading to achievement of program objectives § Guides identification of appropriate indicators § Guides impact analysis Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Conceptual Frameworks Individual characteristics Service utilization Program supply Institutional capacity Technical Monitoring and Evaluation of HIV/AIDS Programs inputs Pretoria, South Africa 2011 Healthy practices Health status Program sustainability

Proximate Determinants Model for HIV/STI (Boerma and Weir) Underlying Context Sociocultural Socioeconomic Programmatic Interventions VCT STD control Condom promote IEC Proximate Biological Partner acquis. Mixing patterns Concurrency Exposure to infected Condom use Concurrent STI Risky sex Treatment Efficacy of Transmission Per contact Treatment Health Outcome HIV incidence STI incidence Duration of activity Socioeconomic Impacts: Socio-Economic Decline (household and national) -Orphans and Vulnerable Children Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011 Demog. Outcome Health Impacts: Morbidity Mortality

Research § Concepts –connotations-defines the problem and constructs for measurement § Operationalise-denotations- variables § Framework allows integration of the program(research) with existing knowledge ( accepted thinking) of the problem. Concepts are identified and linked to one another Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Research cont. . § Research starts from this conceptual or theoretical framework § Basis to Formulate our hypothesis § Expresses Assumptions § Testing/Falsification § This is the basis of deductive approach in research Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Trying it out!! § Identify the problem ( concepts) § Define drivers of the problem § Define elements of your program( concepts) § Identify factors(concepts) in your environment § identify individual factors(concepts) § Establish relationship between the factors § Simplify and refine diagram § Create and present your framework Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

summary § To show where program fits into wider context § To clarify assumptions about causal relationships § To show program components will operate to influence outcomes § To guide identification of indicators § To guide impact analysis (causal pathways) Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Activity § Get into your project/program groups § Develop a conceptual framework for your intervention § After 45 minutes, a member of each group will share the framework with all participants Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Strategic Framework

§ What are strategies § What are results Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Revisit your conceptual framework § What is your vision? § What is your mission? § Where are your strengths? Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

M&E STRATEGIC (RESULTS) FRAMEWORKS Strategic/Results frameworks are diagrams that identify steps, or levels, of results, and illustrate the causal relationships linking all levels of a program’s objectives. Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Purposes: § Provides a clarified focus on the causal relationships that connect incremental achievement of results to the comprehensive program impact § Clarifies project/program mechanics and factors’ relationships that suggest ways and means of objectively measuring the achievement of desired ends Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Strategic framework § Results framework § Diagrams § Summary of results § They identify and illustrate the causal relationships linking all levels of program strategy. § Objectives to impacts Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Results Frameworks § Presents program strategy for achieving specific objective § Usually presented as diagram, § Includes objective and intermediate results § Includes assumptions behind hypothesis (why the program/strategy is expected to work) § Both a planning and management tool Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Goals and Objectives § Goal: § A broad statement of a desired, long-term outcome of the program § Objectives: § statements of desired, specific, realistic and measurable program results Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011 Source: GAP 2003

Terminology ! Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Results Framework Example – PEPFAR funded ART Program SO: Utilization of ART services IR-1: Availability of quality services IR-1. 1: Increase ART sites IR-2: Demand for services IR-2. 1: Increase knowledge of ART IR-1. 2: Supply sites with ARVs IR-2. 2: Increase referral from VCT IR-1. 3: Training for providers Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

EXAMPLE: STRATEGIC OR RESULTS FRAMEWORK Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Hierarchy of Program Dimensions Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

USAID labelling Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Results Frameworks: Key Elements § Goal § Strategic Objective § Should be appropriate & realistic § Balance ambition and accountability § Intermediate Results § Smaller goals § Integral to achieving SO § Hypothesized cause-effect (based on conceptual theory) linkages § Flow from one intermediate result or to many § Move “up” in the results framework § Critical assumptions § Relative to the achievement of the SO § General condition which holds true for SO to be achieved § Be realistic! Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Results Frameworks: National TB Program SO 1: Increase tuberculosis case detection rate to 70% IR 1: Increased availability of quality services IR 1. 1: Services increased IR 1. 2: Practitioners’ skills and knowledge increased IR 1. 3: Improved Monitoring and Evaluation of HIV/AIDS Programs program management Pretoria, South Africa 2011 IR 2: Increased demand for quality services IR 2. 1: Customer knowledge of TB improved IR 2. 2: Social support for TB practices increased Source: MEASURE Evaluation, 2004 (draft)

Results Frameworks FIVE-YEAR GOAL: Foundation Established for Reducing Famine Vulnerability, Hunger and Poverty SO 14: Human capacity and social resiliency increased IR 14. 1: Use of high impact health, family planning, and nutrition services, products, and practices increased IR 14. 1. 1: Community support for high impact health interventions increased IR 14. 1. 2: Availability of key health services and products improved IR 14. 1. 3: Quality of key health services improved IR 14. 1. 4: Health sector resources and systems improved IR 14. 2: HIV/AIDS prevalence reduced and mitigation of the impact of HIV/AIDS increased IR 14. 2. 1: Reduced risk behavior IR 14. 2. 2: Mother-to-child transmission of HIV reduced IR 14. 2. 3: Access to care and treatment for people living with HIV and AIDS increased IR 14. 2. 4: Care and support for orphans and vulnerable children expanded IR 14. 2. 5: A more supportive environment for responding to HIV/AIDS Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011 IR 14. 3: Use of quality primary education services enhanced IR 14. 3. 1: Community participation in the management and delivery of primary education services strengthened IR 14. 3. 2: Planning, mgmt and monitoring and evaluation for delivery of primary education services strengthened IR 14. 3. 3: Quality of primary education improved IR 14. 3. 4: Equitable primary education services strengthened Source: USAID/Ethiopia Integrated Strategic Plan, February, 2004 (revised draft)

Results Frameworks: basis for programmatic process § Agreement within operating unit and donor expectations § Selecting appropriate indicators for the M&E system § Use performance information for management decisions § Analyzing and reporting on program results Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Building a Results Framework § Group work § Using your conceptual framework as a basis § Discuss and draw a strategic framework for your program § After 45 minutes, a member of each group will share the framework with all participants Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Logical frameworks M&E for HIV/AIDS Programs

LFA § Stakeholder analysis § Problem analysis § Objective analysis § selection of preferred implementation strategy Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Logical Frameworks-logic matrix Means of Verification Project Description Performance Indicators Assumptions Goal: The broader development impact to which the project contributes - at a national and sectoral level. Measures of the extent to which a sustainable contribution to the goal has been made. Used during evaluation. Sources of information and methods used to collect and report it. Purpose: The development outcome expected at the end of the project. All components will contribute to this Conditions at the end of the project indicating that the Purpose has been achieved and that benefits are sustainable. Used for project completion and evaluation. Sources of information and methods used to collect and report it. Assumptions concerning the purpose/goal linkage. Component Objectives: The expected outcome of producing each component's outputs. Measures of the extent to which component objectives have been achieved and lead to sustainable benefits. Used during review and evaluation. Sources of information and methods used to collect and report it. Assumptions concerning the component objective/purpose linkage. Outputs: The direct measurable results (goods and services) of the project which are largely under project management's control Measures of the quantity and quality of outputs and the timing of their delivery. Used during monitoring and review. Sources of information and methods used to collect and report it. Assumptions concerning the output/component objective linkage. Activities: The tasks carried Implementation/work program out to implement the project targets. Used during and deliverand the identified monitoring. Programs Monitoring Evaluation of HIV/AIDS outputs. Pretoria, South Africa 2011 Sources of information and methods used to collect and report it. Assumptions concerning the activity/output linkage.

Logical Frameworks-Log frame matrix Present a standardized summary of the project and its logic. Purposes: § Summarizes what the project intends to do and how § Summarizes key assumptions § Summarizes outputs and outcomes that will be monitored and evaluated Other terms used: § Logframe matrix Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

M&E LOGIC FRAMEWORKS or Logic model Logical frameworks are diagrams that identify and illustrate the linear relationships flowing from program inputs, processes, outputs, and outcomes. Inputs or resources affect Processes or activities which produce immediate results or Outputs, ultimately leading to longer term or broader results, or Outcomes. Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Purposes: § Provides a streamlined interpretation of planned use of resources and desired ends § Clarifies project/program assumptions about linear relationships between key factors relevant to desired ends Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Logic Frameworks - Components § Inputs § Program resources like staff, curricula, money § Activities § What the program does like outreach, training, testing § Outputs § Deliverables of program like people trained or tested § Outcomes § Program results like change in knowledge, service use § Impact § Long term change in health status like decreased HIV incidence Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011 Source: CDC Evaluation Guidance Handbook at: http: //www. cdc. gov/hiv/aboutdhap/perb/guidance/chapter 4. htm

Log frame logic model Inputs Activities Outputs Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011 Outcome I M P A C T

Logic Models: Training INPUT PROCESS OUTPUT Develop clinical training curriculum Conduct training events Practitioners trained in new clinical techniques Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011 OUTCOME IMPACT Increase in Declining clients served morbidity by (newly) levels in target trained population providers

; Logic Model: Portion of model for tuberculosis control relating to increasing demand for quality services INPUT PROCESS OUTPUT OUTCOME IMPACT • Human and financial resources to develop and print educational brochure • Distribute brochure to health facilities • Brochure distributed to clients of facilities • Increased customer knowledge of TB transmission and treatment • Decreased TB infection, morbidity and mortality • Meet with physicians to promote distribution of brochure Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011 • Increased demand for quality TB services

Logic Models Voluntary Counseling and Testing (VCT) Problem Statement: HIV infection rates continue to rise, underscoring the importance for people to know their serostatus, develop personalized risk-reduction strategies, and access care and treatment services. INPUTS Financial Resources Demand for Services PROCESSE S Supervision and training for VCT personnel Provide pre-test counseling OUTPUTS People know their HIV status Infrastructure VCT MIS** VCT protocols, guidelines, and training documents** Counseling and Testing Personnel* HIV test kits Provide HIV testing Provide post-test counseling Refer HIV+ clients to appropriate services (PMTCT, care and support, treatment) Maintain patient records HIV+ people are referred to appropriate services Client records are available Service Referral reports are Monitoring of HIV/AIDS Programs system forand Evaluation. Complete Pretoria, South Africa 2011 produced prevention & reporting Tx services** requirements OUTCOMES Clients (HIV+ and -) develop & adhere to personalized HIV riskreduction HIV+ Clients develop & adhere to personalized HIV care, support, and treatment plans Continuity of care is available Program and services are improved Risk behaviors decreased Increase in care, prevention, and treatment services for HIV+, HIV-, and discordant couples Health outcomes of HIV + improve IMPACTS HIV transmissi on rates decreased HIV incidence decreased HIV morbidity & mortality decreased

Frameworks for M&E Planning § Purposes: § clarifying assumptions, goals, and interrelationships between factors relevant to the project or program § defining objectives § selecting activities § defining levels of performance and desired results in terms of planned activities and realistic, objective impacts § Monitoring and evaluation plans incorporate: § program managers’ assumptions and objectives, in a given context § a schematic design displaying the directional linkages between key program elements and/or planned results, and other relevant factors Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

M&E FRAMEWORKS Designing an M&E framework assists in determining: Appropriate program elements to measure Appropriate indicators and data Appropriate methodology Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011

Summary of Frameworks Type of Framework Brief Description Program Management Basis for Monitoring and Evaluation Conceptual Interaction of various factors Determine which factors the program will influence No. Can help to explain results Results Logically linked program objectives Shows the causal relationship between program objectives Yes – at the objective level Logic model Logically links inputs, Shows the causal processes, outputs, and relationship between inputs Monitoring and Evaluation of HIV/AIDS Programs outcomes, and the objectives Pretoria, South Africa 2011 Yes – at all stages of the program from inputs to process to outputs to outcomes/ objectives

References § § § Measure : A trainers guide to the fundamentals of Monitoring and Evaluation for population, health, and Nutrition Programs. 2002. Carolina Population Centre. Chapel Hill. NC. Aus. Guide. The Logical Framework Approach. http: //www. ausaid. gov. au/ausguidelines/1 -1 -1. cfm Bertrand, Jane T. , Magnani, Robert J, and Rutenberg, Naomi, 1996. Evaluating Family Planning Programs, with Adaptations for Reproductive Health, Chapel Hill, N. C. : The EVALUATION Project. Global AIDS Program. (2003) Monitoring and Evaluation Capacity Building for Program Improvement Field Guide, Version 1. U. S. Centers for Disease Control and Prevention, Atlanta, GA. Marsh, David. 1999. Results Frameworks & Performance Monitoring. A Refresher by David Marsh (ppt) http: //www. childsurvival. com/tools/Marsh/sld 001. htm § Tsui Amy. 2004. Frameworks (ppt). Presented at the Bill & Melinda Gates Summer Institute. John Hopkins, Townson. Maryland. § Tsui, Amy. 1999. Frameworks (ppt). Presented at the Summer Institute, University of North Carolina, Chapel Hill. § Bloom S. Pretoria, 2007 South Lecture Africanotes 2011. Asia HIV M&E Slides 2007 Monitoring and Evaluation of HIV/AIDS Programs

Activity § Return to small groups from previous activities § Develop Logical model for one of the activities of your program. § After 30 minutes, a member of each group will share one of the logical frameworks Monitoring and Evaluation of HIV/AIDS Programs Pretoria, South Africa 2011