Evaluation A Systematic Approach Joan Mangan Ph D

Evaluation: A Systematic Approach Joan Mangan, Ph. D, MST Gorgas Tuberculosis Initiative University of Alabama at Birmingham 1

Why Evaluate? § Essential to the design, implementation, and appraisal of any program § Enable program administrators and staff to refine and increase effectiveness of a program § Established § New J. Mangan Ph. D, MST 2

Why Evaluate? § Assist administrators to satisfy the accountability requirements of program sponsors § Internal Sponsors § Ministry of Health § Ministry of Security § External Sponsors § USAID § Global Fund § Sponsors require evidence that was paid for and deemed desirable was actually accomplished J. Mangan Ph. D, MST 3

Outcomes Situational Analysis • Services • Case detection • Resources • Incidence • Obstacles • Baseline data • Incidence • Prevalence • Morbidity • Treatment Default • Drug Resistance • Prevalence ? • Treatment completion • Treatment failure • Drug Resistance • Morbidity • Mortality • Target/objectives created J. Mangan Ph. D, MST 4

The Keys to Success Outcomes And Program Objectives Resources and Program Activities and Intended Outcomes § In order for a TB program to achieve objectives, 3 criteria must be met…… § Measurable objectives have been specified § Demonstrate links in the operation of the TB Program § The individuals in charge have the motivation, ability, and authority to manage the program J. Mangan Ph. D, MST 5

Different Purposes…. Different Strategies § Three major classes of evaluation § Situational Analysis § a. k. a. - Planning Evaluation § Process Evaluation § Impact / Outcome Evaluations J. Mangan Ph. D, MST 6

Situational Analysis a. k. a Planning Evaluation § Assess § The extent and severity of the TB problem § Current resources § Begin to identify needs § Conceptualization and design § New programs § Expansions to existing programs J. Mangan Ph. D, MST 7

Process Evaluation § Determine a program’s success in delivering resources, services, and benefits § Monitoring program implementation § What is being provided § Alternative approaches to the diagnosis of TB? § Alternative approaches to treatment and follow-up of patients § How well services are provided § All levels § staff , patients , patient families , surrounding communities J. Mangan Ph. D, MST 8

Process Evaluation § Monitoring program implementation § When & Where § Day to day operations are efficient and meet the needs of the patient population § Staff are equipped with needed resources to accomplish objectives § Equipment, documentation forms, etc. J. Mangan Ph. D, MST 9

Impact / Outcome Evaluations § Examine the extent to which a program is the cause of a positive change § Traditional indicators § Numbers of new TB cases § Numbers of patients who complete treatment § etc J. Mangan Ph. D, MST 10

Impact / Outcome Evaluations § Aid in decision-making to continue, expand, or cut back a program § Consider cost in relation to the benefits of a program § Compare program costs with alternative strategies when working with few $$ resources J. Mangan Ph. D, MST 11

Situational Analysis Process Evaluation • Existing services • Commitment / Support • Existing resources & needed resources • New / expanded services • Obstacles to effective TB control • Baseline data • Incidence • Prevalence • Case Notification • Morbidity • Treatment Default • Drug Resistance • Mortality • Target/objectives created • Coverage/availability • Delivery of Services • Additional resources Impact / Outcome Evaluation • Case detection • Incidence • Prevalence • Treatment completion obtained • Treatment failure • Policies / Procedures • Drug Resistance • Staff training • Reporting / Documentation • Morbidity • Mortality • Diagnosis / Treatment • Information, education and communication campaigns • Knowledge • Attitudes • Beliefs J. Mangan Ph. D, MST 12

The Process Evaluation § Some may look at the process evaluation as an audit – there are similarities …but they are not identical activities § Audits – compare “what is happening” with “what should be happening” § Process Evaluation – identify and measure what has happened and why it has happened § Example – monitoring may reveal a delay in physicians obtaining the results of sputum smears to confirm a diagnosis of tuberculosis due to (a) not enough laboratory staff to read smears and (b) collection of sputum from numerous un-symptomatic individuals J. Mangan Ph. D, MST 13

The Process Evaluation: Monitoring for Success § Coverage - Is the program providing care to those at greatest need / highest risk for developing TB? § Measuring Coverage § Review program records § Surveys of TB patients § Community surveys § Examine difference between persons who take advantage of the TB program’s services versus those who are eligible but don’t use program services or default on treatment § Question: can the population access TB program services? J. Mangan Ph. D, MST 14

Monitoring for Success § Delivery of TB Program Services Systems § Three main questions to ask: § Is diagnosis /treatment being provided? …. or if the services for diagnosis/ treatment are available in an area are there sufficient services to cover the population? § Is the correct diagnosis / treatment being provided? § Is the diagnosis / treatment standardized? § Also Important: Presentation - Is the program presented in such a way that patients accept the diagnosis procedures and treatment that is being offered J. Mangan Ph. D, MST 15

Monitoring for Success § Measuring Delivery of TB Program Services Systems § Observation of activities § Checklist § Medical records § Accuracy in diagnosis? § Standardized treatment? § Healthcare providers and staff § Individual or group interviews with patients J. Mangan Ph. D, MST 16

Analysis of Monitoring Data § Address 3 Issues § Describe the TB program § Compare TB program implementation between sites § Assess conformity to the TB program’s policies and procedures J. Mangan Ph. D, MST 17

Analysis: Description of the Program § Annual Program Reports § Estimates coverage of population § Describes the services delivered § Reactions of patients to the program § Examples: § Patient’s and community knowledge of TB, program services, access to care § Patients seeking services – why? / why not? § Defaulting – why? / why not? J. Mangan Ph. D, MST 18

Analysis: Comparison between Sites § Helps identify reasons for the differences that may exist in outcomes (case detection, treatment completion, etc. ) § These differences may be good or bad § An assessment between sites can identify “best practices” …. Or why some sites are more effective than others § Sharing this information can enhance the overall program § Increase standardization J. Mangan Ph. D, MST 19

Analysis: Comparison between Sites § Bi-annual evaluation meetings in Honduras § An opportunity for staff on the “front lines” to learn and share ideas – creative problem solving § Provide teaching moments / clarify a policy or procedure § Increased “owner-ship” in the program § Motivation to improve … a “nicer” form of accountability § Allow program administrators to identify and prioritize those sites that need more attention/assistance J. Mangan Ph. D, MST 20

NTP Evaluation Meeting Form Efficiency and Effectiveness J. Mangan Ph. D, MST 21

Analysis: Conformity to the TB Program Policies and Procedures § Discrepancies between how a program is operating and the policies and procedures may lead to: § Efforts to “move” closer to the originally planned policies and procedures § Redesign policies and procedures to reflect the best practices that arise out of necessity or creativity of staff on the “front lines” J. Mangan Ph. D, MST 22

Using Evaluation Information § Decision making § A rationale for action / change § “Go / No-Go” Decisions § § § Accountability Advocacy Policy J. Mangan Ph. D, MST 23

Take Home Message § An evaluation of the “process” will provide you with specific targets for action … with specific targets there is a decreased risk of wasting time and money J. Mangan Ph. D, MST 24

Take Home Message § To obtain good information – there is a need to get out in the field and talk to those people responsible for the day to day operation of a program as well as patients and their family members § Data obtained from traditional TB Program indicators are not always enough § Those on the front lines are in the best position to tell you: § § What works, What doesn’t work, What is needed, ……. . and are a good source of the best ideas J. Mangan Ph. D, MST 25

The Pragmatic Side of Any Evaluation § Interests of sponsoring organizations may change § Are these interests incorporated / reflected in evaluation activities? § Unanticipated problems/solutions may arise and impact the way a program works § Early identification and action can § Avoid further damage § Reap the benefits (example: IEC programs in Honduras) J. Mangan Ph. D, MST 26

The Pragmatic Side of Any Evaluation § Partial evaluations may provide knowledge that a program is not producing the desired outcomes … the need for program refinement may be identified sooner rather than later J. Mangan Ph. D, MST 27