INSTITUTE OF MEDICINE Obesity Evaluation Toolkit Resources for

INSTITUTE OF MEDICINE Obesity Evaluation Toolkit: Resources for Evaluating Community-Level Obesity Prevention Efforts Webinar INSTITUTE OF MEDICINE August 25, 2015

1. Introduction Leslie Sim, M. P. H. Institute of Medicine of The National Academies of Science, Engineering, and Medicine 2. Overview of report, Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress Lawrence W. Green, Dr. P. H. , M. P. H. University of California, San Francisco INSTITUTE OF MEDICINE 2

3. Steps in evaluating communitylevel obesity prevention efforts Stephen Fawcett, Ph. D. Work Group for Community Health and Development, University of Kansas 4. Web-based resources to support your efforts Christina Holt, M. A. Community Tool Box, Work Group for Community Health and Development, University of Kansas 5. Questions INSTITUTE OF MEDICINE 3

Submit questions - toolbox@ku. edu Issues connecting? - 1 -866 -770 -8162 INSTITUTE OF MEDICINE 4

Evaluating Obesity Prevention Efforts: A Plan for Measuring Progress Lawrence W. Green, Chair; and 13 members of The Committee on Evaluating Progress of Obesity Prevention Efforts INSTITUTE OF MEDICINE 5

Committee on Evaluating Progress of Obesity Prevention Efforts Lawrence W. Green, Dr. P. H. (Chair) University of California, San Francisco Christina Bethell, Ph. D. , M. P. H. , M. B. A. Johns Hopkins University Ronette R. Briefel, Dr. P. H. , R. D. Mathematica Policy Research Ross C. Brownson, Ph. D. Washington University in St. Louis Jamie F. Chriqui, Ph. D. , M. H. S. University of Illinois at Chicago Stephen Fawcett, Ph. D. University of Kansas Brian R. Flay, D. Phil. Oregon State University Deanna M. Hoelscher, Ph. D. , R. D. , L. D. , C. N. S. The University of Texas School of Public Health, Austin James W. Krieger, M. D. , M. P. H. Action for Healthy Food Laura C. Leviton, Ph. D. The Robert Wood Johnson Foundation K. M. Venkat Narayan, M. D. , M. Sc. , M. B. A. Emory University Nico P. Pronk, Ph. D. Health. Partners, Inc. Lorrene Ritchie, Ph. D. , R. D. University of California, Nutrition Policy Institute Elsie Taveras, M. D. , M. P. H. Massachusetts General Hospital Harvard Medical School INSTITUTE OF MEDICINE 6

AN URGENT NEED FOR EVALUATION INSTITUTE OF MEDICINE 7

What we need to know from evaluation… • Where are we in making progress and with whom? (current status) • How are we doing in making progress? (trends over time in assessing needs and implementation of policies and strategies) • What works in which populations? • What are the unintended consequences? INSTITUTE OF MEDICINE 8

Committee on Evaluating Progress of Obesity Prevention Efforts Study Charge: “to develop a concise and actionable plan for measuring progress in obesity prevention efforts for the nation and adaptable guidelines for community assessments and evaluation. ” INSTITUTE OF MEDICINE 9

INSTITUTE OF MEDICINE 10

The Concise and Actionable Plan • Enhanced Leadership • Enhanced Roadmap • Enhanced Capacity and Infrastructure Included resources: • Indicators for the evaluation plans • Tools and methods for assessing progress in populations at greater risk for obesity • Community health assessment, surveillance, and monitoring of interventions INSTITUTE OF MEDICINE 11

Interdependence of National and Community Obesity Evaluation Plans Core indicators, Data sources, Resources, Methodologies Community Obesity Evaluation Plan National Obesity Evaluation Plan Contextual data, Feasibility, Local innovation INSTITUTE OF MEDICINE 12

Recommendations 1. Improve Leadership and Coordination 2. Improve Data Collection 3. Provide Common Guidance 4. Improve Access to and Dissemination of Information 5. Improve Workforce Capacity 6. Address Disparities and Health Equity 7. Support a Systems Approach INSTITUTE OF MEDICINE 13

To access the full report and related dissemination materials (url: http: //iom. nationalacademies. org/Reports/2013/Evaluating -Obesity-Prevention-Efforts-A-Plan-for-Measuring. Progress. aspx) 4 -page report brief Interactive indicator widget Pull-out summary of indicators INSTITUTE OF MEDICINE 14

Indicators of Progress (Excerpt from Table 4 -1) Indicator Topic Objective OVERARCHING/SYSTEM-LEVEL INDICATORS Obesity-adult Reduce the proportion of adults who are obese (body mass index (BMI) ≥ 30) APOP GOAL AREA 1: PHYSICAL ACTIVITY ENVIRONMENT Adult physical activity Increase the proportion of adults who meet current federal physical activity guidelines for aerobic physical activity and for muscle-strengthening activity … INSTITUTE OF MEDICINE 15

National and State Level Indicators (excerpt, Table 6 -4) Indicator Topica Data Source or Documentationb National Planc State Plansc WHO Proposed Indicatorsd Overarching/System Level 1. Obesity-adult BRFSS; NHANES Core 2. Obesity-adolescent NHANES; YRBSS Core 3. Obesity-child NHANES Core Goal Area 2: Food and Beverage Environment 28. Adult energy intake NHANES Expanded 29. Child and adolescent energy NHANES Expanded BTG; CLASS; SHPPS; SNDA Expanded intake 30. Sugar-sweetened beverage policies in schools INSTITUTE OF MEDICINE 16

Community Level Indicators (excerpt, Table 7 -2) Indicator Topica Data Source Current Availability by Community Sizeb Larger Smaller Overarching/System-Level Obesity-adult BRFSS Overweight-adult BRFSS Obesity-adolescent YRBSS, School reports Goal Area 2: Food and Beverage Environment Sugar-sweetened beverage policies in schools (school district) SHPPS Sugar-sweetened beverage consumption YRBSS (adolescent) School policies to facilitate access to clean drinking water SHPPS Note: Larger – population >50, 000; Smaller – population < 50, 000 INSTITUTE OF MEDICINE 17

Some Steps in Evaluating Community. Level Obesity Prevention Efforts Stephen Fawcett, Work Group for Community Health and Development, University of Kansas INSTITUTE OF MEDICINE 18

Vision for IOM Report Assure collection and analysis of timely and meaningful data to inform and improve obesity prevention efforts at national, state, and community levels INSTITUTE OF MEDICINE 19

CONTEXT for Evaluation • • • Multi-component, multi-sector, and multi-level interventions Indicators/measures of varying quality & utility Varying capacity, capabilities, leadership, and resources INSTITUTE OF MEDICINE 20

ACTIVITIES: Develop Resources for Training, Technical Assistance, and Dissemination • • • Products/protocols/templates Web-based support & training Curricula Clearinghouses for measures Communications INSTITUTE OF MEDICINE 21

Steps/Components of a Community Evaluation Plan (Box 8 -1) 1. Design stakeholder involvement. 2. Identify resources for the monitoring and summative evaluation. 3. Describe the intervention’s framework, logic model, or theory of change. 4. Focus the monitoring and summative evaluation plan. 5. Plan for credible methods. 6. Synthesize and generalize. INSTITUTE OF MEDICINE 22

1. Design stakeholder involvement • Identify stakeholders • Consider the extent of stakeholder involvement • Assess desired outcomes of monitoring and summative evaluation • Define stakeholder roles in the monitoring and summative evaluation INSTITUTE OF MEDICINE 23

Participatory Evaluation (CBPR) INSTITUTE OF MEDICINE 24

2. Identify resources for monitoring and summative evaluation • Person-power resources • Data collection resources INSTITUTE OF MEDICINE 25

Some Web-Based Data Sources (Table 7 -5) • Community Commons & CHNA. org • County Health Rankings • USDA Food Atlas • CDC Diabetes Interactive Atlas • Census ACS and County/Zip Business Patterns • HHS Community Health Status Indicators INSTITUTE OF MEDICINE 26

What indicators available at local level? Available for all/ many communities: • Adult obesity/overweight • Activity: Active transport by walking, bicycling, density of recreational facilities, leisure-time PA • Nutrition: Adult F/V, food outlet density, farmers’ market density, food deserts, SNAP/WIC store density Some larger communities also have: • Youth obesity/overweight • Activity: Youth PA, screen time, school PA participation • Nutrition: SSB consumption, youth F/V INSTITUTE OF MEDICINE 27

Data Availability (Table 7 -2 – Example) Large INSTITUTE OF MEDICINE Small

3. Describe the intervention’s framework or logic model, or theory of change. • • • Purpose or mission Context or conditions Inputs: resources and barriers Activities or interventions Outputs of activities Intended effects or outcomes INSTITUTE OF MEDICINE 29

Generic logic model for community obesity prevention (Figure 8 -1, adapted). INSTITUTE OF MEDICINE 30

4. Focus the monitoring and summative evaluation plan. • Purpose or uses: What does the monitoring and summative evaluation aim to accomplish? • Set priorities by end-user questions, resources, context • What questions will the monitoring and summative evaluation answer? • Ethical implications (benefit outweighs risk) INSTITUTE OF MEDICINE 31

End-User Focused Evaluation Questions—Some examples: • How fully was intervention implemented? • Did the intervention have desired effects? • What was the impact on participants/ population? • With whom? • Under what conditions? INSTITUTE OF MEDICINE 32

5. Plan for credible methods. • Stakeholder agreement on methods • Indicators of success • Credibility of evidence INSTITUTE OF MEDICINE 33

Some Emerging Methods for Data Collection Environmental change data • Documentation of initiatives • Unobtrusive observations • Secondary data (e. g. , GIS) Policy change data • Documentation of initiatives • Surveillance Systems change data • Mapping changing relationships INSTITUTE OF MEDICINE 34

Indicators of Success • Translate expected effects (logic model) into specific measurable units • Examples include: – Program Outputs—units of activities delivered – Intermediate Outcomes—changes in communities and systems (program, policy, environment) – Behavioral Outcomes—changes in diet and physical activity – Population-level Outcomes—reduced prevalence of obesity INSTITUTE OF MEDICINE 35

Evaluation Designs – Match to Goal/Context Qualitative methods: interviews, focus groups, photo-voice, etc. INSTITUTE OF MEDICINE 36

6. Synthesize and generalize. • • • Disseminating and compiling studies Learning more from implementation Ways to assist generalization Shared sense-making and cultural competence Disentangling effects of interventions INSTITUTE OF MEDICINE 37

Logic Model Design and Shared Sensemaking Obesity INSTITUTE OF MEDICINE 38

Obesity Evaluation Toolkit: Web-based Resources for Community Evaluation CONTEXT: Distributed evaluation workforce • • People we will never see In places we will never be TOOLKIT: Just-in-time resources for: • • Training Technical Assistance INSTITUTE OF MEDICINE 39

Web-based resources to support your efforts Christina Holt, M. A. Community Tool Box, Work Group for Community Health and Development, University of Kansas INSTITUTE OF MEDICINE 40

Navigating to the Toolkit The Obesity Evaluation Toolkit is available online: http: //iom. nationalacademies. org/activities/nu trition/obesityprevprogress/resourcesevaluating-community-level-obesityprevention-efforts INSTITUTE OF MEDICINE 41