Comprehensive Evaluation Outcome Assessment Process Evaluation Concept Design

Comprehensive Evaluation Outcome Assessment Process Evaluation Concept & Design Analysis

Part 3: Objectives Participants will be able to ¡ Distinguish outcome evaluation from other levels: 1) concept and design evaluation 2) process evaluation 3) outcome evaluation 4) economic evaluation ¡ Match evaluation questions and standards of comparison

Sources of Failure … Program Theory Good Idea Population Outcome Population Impact! Implementation Set in motion Measurement/ Power Sufficient power, credible outcome measures Study Outcome Sufficient effect Application Generalizability Applied widely in similar populations & settings Likely to succeed in similar populations & settings

Potential Sources of Failure in Program Evaluations … Progra m Theory Good idea Implemen Measurement/Powe -tation r Evaluat Set in ed with Motion poor with measure good s, low reach, power, dose, very and weak fidelity design Study Outcom e Inaccurat e results: False negative or false positive

Sources of Failure … Progra m Theory Good idea Implemen Measure. Study -tation ment/Powe Outcom r e Sensitive Set in Sufficien measure motion t s, effect sufficient Population Generalizability Outcome Application power No populatio n impact Applied to populations in need But unlikely to succeed in similar populations &

Outcome Evaluation ¡ ¡ Outcome Process Concept & Design Is the program in achieving the intended outcomes? Is the program the real cause of the observed outcomes? Does the program cause harm? Does it have positive “side effects”? To which populations and settings can the outcomes of this evaluation be generalized?

Identifying Program Outcomes ¡ ¡ ¡ Use the logic model to consider long-term, intermediate, or short-term outcomes You may not need to measure long-term outcomes l If you are in a formative phase and do not yet have evidence that you can achieve short-term outcomes l If the epidemiologic evidence provides a strong link between intermediate and long-term outcomes and the intermediate outcome has good validity Rule of thumb: Whatever outcome you decide to measure, also measure the more short-term outcomes to capture the causal chain

Falls and the Elderly Outcomes Activities Shorter-term Shortterm Intermed Long Fewer meds w/ side effects Less Meds Coordination of or dizziness, session w/ meds & other interactions hypoten. Pharm. D drugs with other sion meds Volunteers to modify home Fix fall hazards, install stabilization devices More knowl & stronger beliefs about benefits Strength & of exercise; balance More knowl training about which More Intervention exercises; Hypothesis social support Fewer hazards in the home, more stabilizing devices More walking, balance & strength exercises Falls Disability, loss of mobilit y Better balance, strength Causal Hypotheses

Don’t forget to look for plausible negative outcomes ¡ ¡ Condom protection programs for female commercial sex workers at risk for HIV/AIDS typically presume condom use will serve as an effective contraceptive But, although condom use increases with many behavioral and social programs, condoms tend not to be used with primary partners, leaving women at risk for unplanned pregnancy

Basic Study Designs ¡ ¡ Descriptive l Case Series l PMR l Ecological Observational l Cohort l Case Comparison l Cross Sectional ¡ Experimental Clinical Trials ¡ “Controlled, Experiment, Randomized trials l Quasi Experimental ¡ Community trials l

Study Designs - Descriptive In depth case studies with a lot of data collected that can help rule out rival explanations and suggest causal relations l Best condition is when the outcomes of interest are unique, knowledge or behavior that have few other plausible causes l Useful for hypotheses development l

Study Designs - Correlational Studies ¡ ¡ ¡ Provides a crude way of exploring associations, but the causal direction is unclear Hypothesis generating Does not relate exposure and outcome to an individual No control or very little control over distorting factors Sample selection is usually problematic

Study Designs - Cohort Studies ¡ ¡ ¡ Provides stronger evidence of exposure (program) - disease (outcome) associations Evidence of a temporal relationship (when prospective) Minimizes bias in ascertainment of exposure (when prospective)

Study Designs – Experimental ¡ Randomized trials l Individual RCT l Group or cluster RCT (need to analyze correctly) l What constitutes appropriate comparison or control group

Sample Nonparticipants Randomization to groups Intervention group(s) Control group Lost to follow-up Measure outcome Schematic Diagram of a Clinical Trial

Study Designs – Experimental ¡ Non-randomized (quasi-experimental) trials l Can also be individual or group unit of assignment l Concurrent vs non-concurrent (historical) comparison group l Initial comparability of study groups l Statistical adjustment to improve comparability

Confounding Program Independent Variable HIV Counseling & testing for IDUs Outcome Dependent Variable HIV+ report using condoms with sex partners Awareness that they should protect partners Confounder