Readiness for Program Evaluation Assessing and supporting readiness

Readiness for Program Evaluation Assessing and supporting readiness for evaluation among community partners delivering Healthy Weight Programs SESSION 2: COLLABORATIVE LEARNING PROJECT

Session 1 We Considered… • Purpose of evaluation • Types of program evaluations • Logic models and framework for evaluation • Standards for and step-by-step process for evaluation SESSION 2: COLLABORATIVE LEARNING PROJECT

Today We Will… • Conceptualize a framework for thinking about a healthy weight program’s (HWP) readiness for evaluation • Learn how to assess an HWP’s readiness for evaluation using existing tools • Anticipate and plan for challenges that might limit readiness when evaluating in community settings • Review case studies to see “getting ready to evaluate” in action! SESSION 2: COLLABORATIVE LEARNING PROJECT

Conceptual Frameworks What does it mean to be “ready” to conduct an evaluation? SESSION 2: 2: SESSION COLLABORATIVE LEARNING PROJECT

Framework 1: Ready, Set, Change! Holt et al, 2010. Individual Psychological Individual Structural Organizational Psychological Organizational Structural SESSION 2: COLLABORATIVE LEARNING PROJECT

Framework 2: The NIRN Hexagon Metz, A. & Louison, L. (2018). The Hexagon Tool: Exploring Context. Chapel Hill, NC: National Implementation Research Network, Frank Porter Graham Child Development Institute, University of North Carolina at Chapel Hill. SESSION 2: COLLABORATIVE LEARNING PROJECT

Assessing Readiness to Conduct an Evaluation Finding strengths and areas for improvement to prepare for a robust evaluation SESSION 2: 2: SESSION COLLABORATIVE LEARNING PROJECT

Wilder Collaboration Inventory Mattessich and Johnson, 2018 SESSION 2: COLLABORATIVE LEARNING PROJECT

Wilder Scoring SESSION 2: COLLABORATIVE LEARNING PROJECT

Organizational Readiness for Implementing Change (ORIC) Shea CM, Jacobs SR, Esserman DA, Bruce K, Weiner BJ. (2014). Organizational readiness for implementing change: a psychometric assessment of a new measure. Implementation Science.

Measures: Other Individual Readiness for Organizational Change • (Holt et al. , 2007) Perceived Organizational Readiness for Change • (Cinite et al. , 2009) Organizational Change Recipients’ Beliefs Scale • (Armenakis et al. , 2007) Organizational Change Questionnaire – Process, Context, and Readiness • (Bouckenooghe et al. , 2009) TCU Organizational Readiness for Change • (Lehman et al. , 2002) SESSION 2: COLLABORATIVE LEARNING PROJECT

The Right Assessment for Your Program • http: //readiness. knowledgetranslation. ca/ SESSION 2: COLLABORATIVE LEARNING PROJECT

Preparing to Evaluate in a Community-Based Setting Differences from traditional research and strategies to address common challenges SESSION 2: COLLABORATIVE LEARNING PROJECT

Common Challenges Faced When Evaluating in Community-Based Settings Regulations Resources Relationships SESSION 2: COLLABORATIVE LEARNING PROJECT

Active Recreation through Community. Healthcare Engagement Study (ARCHES) Current Cumberland 1 2 Durham 3 Montgomery 4 Rowan 5 Wake 6 Wayne Planned 7 Forsyth 8 Mecklenburg 9 Moore 10 Richmond Ongoing, funded Ongoing, unfunded Accepting referrals, funded Proposed in CDC Funded by NC Office of Rural Health

Mapping Challenges to Solutions Regulations • IRB: Define question to minimize burden, provide context for training • Consent: e-consent or phone consent can be managed by study team Resources • Online data entry • Minimize unnecessary variables • Objective and remote data collection Relationships • Collaborative engagement • Ensure key stakeholders have input • Iterative communication and feedback

Case Study 1 Evaluation Readiness: Start with the Big Picture Ihuoma Eneli, MD, MS Center for Healthy Weight and Nutrition Nationwide Children’s Hospital Columbus, Ohio SESSION 2: 2: SESSION COLLABORATIVE LEARNING PROJECT

Evaluation Readiness: Establishing the Need and Urgency • Need for evaluation has to be driven by a clarity of the purpose, goals, and survivability of the program • All team members have to appreciate their role in attaining the purpose, goals, and survivability of the program • Starts with defining mission and vision (OR goals) for the program • Link to a larger organization SESSION 2: COLLABORATIVE LEARNING PROJECT

Vision § A vision statement describes the desired end-state of the Center for Healthy Weight and Nutrition (CHWN). § It is meant to be a clear, aspirational, and inspirational longterm desire achieved from the work the CHWN performs. § CHWN Vision: An obesity-free society for all children Mission § A mission statement defines the reason why an entity exists. § A mission statement typically does not change over time. § Mission statements are used to help guide decisions about priorities, actions, and responsibilities. § CHWN Mission: Empowering children and families to establish a foundation for a healthy lifestyle through clinical care, education, research, community engagement, and advocacy SESSION 2: COLLABORATIVE LEARNING PROJECT

Phases and Types of Evaluation Program Stage Phase Before Program Begins New Program FORMATIVE Evaluation Type Needs Assessment (Some) Questions Asked To what extent is the need being met? What can be done to address this need? Based on slides from Jennifer Nichols, Porter Novelli **Credit to NCCOR Collaborative Learning Session 1 (More) Established Program Mature Program SUMMATIVE Process Evaluation Outcome Evaluation Impact Evaluation Is the program operating as planned? Is the program achieving its short -term outcomes/ objectives? Is the program achieving it’s long-term outcomes and impacts? ? SESSION 2: COLLABORATIVE LEARNING PROJECT

Evaluability Assessment (EA) • Developed by Joseph Wholey in 1979 • Systematic process that helps identify whether program evaluation is justified, feasible, and likely to provide useful information • Cheaper than an evaluation plan (10% rule of thumb) • Can provide useful information regardless if a program has an evaluation plan or not • Provides stakeholders with valuable information on adapting the program structure to support future evaluations • Ideally conducted by an external evaluator BUT depending on resources, can be conducted by program staff SESSION 2: COLLABORATIVE LEARNING PROJECT

Examples of Evaluability Questions • Does the program serve the population for whom it was designed? • Does the program have the resources discussed in the program design? • Are the program activities being implemented as designed? § Does the logical connection between activities and goals and objectives exist? • Does the program have the capacity to provide data for an evaluation • How well are data collected? Are they reliable? Is the program capable of collecting and managing the data needed for an evaluation? § Who enters the data? What are their qualifications? SESSION 2: COLLABORATIVE LEARNING PROJECT

Evaluability Case Study: Program Model Not Sound • Community-based afterschool program with programming targeted at healthy eating and physical activity • Outcome measures: fitness level and change in weight • Evaluability assessment § No reliable documentation from programs to assess delivery of the lessons § Evaluation periods not standardized § Attendance was routinely tracked § Only 15% of the same children have pre- and post-testing • Full evaluation plan suspended SESSION 2: COLLABORATIVE LEARNING PROJECT

The Primary Care Obesity Network (PCON) • Network of primary care practices linked to a tertiary care obesity center at a children’s hospital to prevent and treat childhood obesity • Driven by policy initiative in the State of Ohio • Embedded into a tertiary care obesity center • Goals in both prevention and treatment • Functions as an outreach program for the hospital and center Eneli, IU, et al. (2017). The primary care obesity network: Translating expert committee guidelines on childhood obesity into practice. Clinical Pediatrics. SESSION 2: COLLABORATIVE LEARNING PROJECT

PCON Model of Care The Hub: • Provides training and support to PC PC • • Tertiary Care Hub PC • PC PC PC practices Serves as integrator between the community and practices Provides access for nutrition services Stage 3 & 4 referral center The Practices (PC): • Participate in prevention activities geared • • • to all patients Stage 1 & 2 support for patients Catch patients earlier and refer as needed to tertiary care (Hub) Link patients to community resources The Community: • Provides community-based resources to Community • practices Participates in educational activities for the network, patients, and families SESSION 2: COLLABORATIVE LEARNING PROJECT

Readiness for Evaluation: Examples of Questions Metz, A. & Louison, L. (2018) The Hexagon Tool: Exploring Context. Chapel Hill, NC: National Implementation Research Network, Frank Porter Graham Child Development Institute, University of North Carolina at Chapel Hill. SESSION 2: COLLABORATIVE LEARNING PROJECT

Evaluation Plan: Outcomes • Patient/client outcome: Will the proposed program or change affect the patients knowledge, health, or well being? • Process outcome: Will implementing the program or change go as planned? • Balancing outcome: Will the proposed change lead to a negative outcome in another part of the process? Eneli IU, Cannon-Smith G. Systems of Care and Quality Improvement: Guiding the Care of the Child with Obesity. Clinical Care of the Child with Obesity: A Learner’s and Teacher’s Guide. SESSION 2: COLLABORATIVE LEARNING PROJECT

Program Outcomes Eneli, IU, et al. (2017). The primary care obesity network: Translating expert committee guidelines on childhood obesity into practice. Clinical Pediatrics. SESSION 2: COLLABORATIVE LEARNING PROJECT

Case Study 2 Evaluation Readiness: Drilling Down to the Nitty Gritty Alexis Tindall MHA, RD, LD Center for Healthy Weight and Nutrition Nationwide Children’s Hospital Columbus, Ohio SESSION 2: 2: SESSION COLLABORATIVE LEARNING PROJECT

Organizational Readiness The extent to which individuals or groups are psychologically and behaviorally prepared to implement a specific program or practice A key determinant of implementation success and a mediator of the effectiveness of implementation interventions Implementation Evaluation SESSION 2: COLLABORATIVE LEARNING PROJECT

Ready, Set, Change! Holt et al, 2010. Individual Psychological Individual Structural Organizational Psychological Organizational Structural SESSION 2: COLLABORATIVE LEARNING PROJECT

Understanding Where You Are At The State of Childhood Obesity in Franklin County *Includes youth ages 10 -17. (Remaining body mass index percentages includes youth ages 6 -18. ) http: //www. centralohiohospitals. org/documents/Health. Map_2016. pdf SESSION 2: COLLABORATIVE LEARNING PROJECT

Translating to Columbus Healthy Weight Quick Facts § Children 2 and older § BMI at or above 95 th percentile § Monthly visits § Individualized treatment plan § Physician or NP § Dietitian § Physical therapy § Social work/psychology § New U evening group programs—child and parent § ATC § Dietitian § +/- Psych § Bariatric program SESSION 2: COLLABORATIVE LEARNING PROJECT

Goals Strategic Roadmap Improve Patient Experience and Operational Excellence • Increase patient engagement Strategies • Improve Center efficiencies • Increase availability and leverage technology • Establish treatment pathways and explore different models of care Improve Community and Internal Relationships Increase Communication and Teamwork Enhance Research and Center Visibility • Expand community reach • Increase accountability, communication, and teamwork within the department • Establish and sustain research infrastructure • Increase awareness of programs in the community • Increase interactions with Nationwide Children’s Hospital providers • Enhance Center visibility • Establish consistent communication channels within NCH, subspecialty clinics, and the community • Enhance staff development opportunities SESSION 2: COLLABORATIVE LEARNING PROJECT

The “Keys” for Assessing Program Evaluation Readiness • Consider “The Who” § Community Awareness • Physician practices, community organizations (e. g. , YMCA), after school programs, stakeholders § Organizational Awareness • Leadership support, relationship building with other sub-specialists SESSION 2: COLLABORATIVE LEARNING PROJECT

The “Keys” for Assessing Program Evaluation Readiness • Consider “The How” § Staffing needs (MD, RD, Psych, Health Coach, etc. ) and the requested amount of FTEs per discipline § Training and knowledge expectations § Resources needed (outside of staffing) • Space, equipment, materials, sites § Infrastructure • Electronic health record, Excel, Access SESSION 2: COLLABORATIVE LEARNING PROJECT

The “Keys” for Assessing Program Readiness for Managing Data • Has the organization/program identified and prioritized its desired results? • Is there an established means to measure progress toward those results or can it be created? • Is there a process for tracking and measuring progress toward its desired results, which includes effective means to display data? • Is there a communication plan where individuals working to achieve the desired results can exchange and provide ongoing feedback? • Is there an established plan to periodically review progress? Monthly? Quarterly? • Is there a process to intervene when needed as a means to improve progress? (HRSA, nd) SESSION 2: COLLABORATIVE LEARNING PROJECT

Evaluation is a Continuous Cycle • Continuous Quality Improvement (CQI) cycle § Planning – What actions will best reach our goals and objectives? § Performance measurement – How are we doing? § Evaluation – Why are we doing well or poorly? ** Credit to NCCOR Session 1 What do we do? Why are we doing well or poorly? How do we do it? How are we doing? SESSION 2: COLLABORATIVE LEARNING PROJECT

Staying Accountable IHI Model for Improvement • Three fundamental questions 1. What are we trying to accomplish? 2. How will we know that a change is an improvement? 3. What changes can we make that will result in improvement? • • The Plan-Do-Study-Act (PDSA) cycle. Key Driver Diagram (KDD) SESSION 2: COLLABORATIVE LEARNING PROJECT

Weiner’s Theory of Organizational Readiness to Change commitment: Intent to change is shared Change efficacy: Shared belief that they can jointly make the change Change valence: How much the change is valued Informational assessment: Perception that resources are available to implement the change Context: Organizational culture, climate, resources, structure, and past experiences with implementing change SESSION 2: COLLABORATIVE LEARNING PROJECT

Bonus Case Study: ARCHES SESSION 2: 2: SESSION COLLABORATIVE LEARNING PROJECT

Active Recreation through Community-Healthcare Engagement Study (ARCHES) • Model to Evaluate: Bull City Fit, a clinic—parks and recreation partnership to deliver child obesity treatment • Goal: Implement and evaluate the model in 6 new community setting • Challenges: § No existing relationships in new counties § Unclear need/priority of child obesity in counties § No known relationships between clinics and parks/recreation centers in communities § Likely very low capacity for new data collection § Unknown electronic medical record capacity • Resources: § Funding for a local “connector” and equipment grant § The “Trailguide” (curriculum) and our team for technical support SESSION 2: COLLABORATIVE LEARNING PROJECT

ARCHES Project Aims Children and Teens with Obesity • Aim 1: Evaluate the implementation feasibility of an integrated clinic-community child obesity treatment program Community Recreation Center • Clinical care model, including ability to meet the existing child obesity treatment recommendations. • Improved Health Aim 2: Evaluate the clinic-community Aim 3: Report patient outcomes associated with participation in the integrated model SESSION 2: COLLABORATIVE LEARNING PROJECT

Current and Planned Sites Current 1 Cumberland 2 Durham 3 Montgomery 4 Rowan 5 Wake 6 Wayne Planned 7 Forsyth 8 Mecklenburg 9 Moore 10 Richmond Ongoing, funded Ongoing, unfunded Accepting referrals, funded Proposed in CDC Funded by NC Office of Rural Health SESSION 2: COLLABORATIVE LEARNING PROJECT

Readiness for Evaluation • Chose sites intentionally • 1: 1 conversations with key clinic and community stakeholders • Large group “kickoff” meeting with both clinic and community partners • Site visits • Administered Wilder Collaboration Inventory (pre/post) SESSION 2: COLLABORATIVE LEARNING PROJECT

Assessment • Site: § Providers say they need something for kids with obesity; parks and rec express desire to meet “wellness” domain § Lots of discussion about best “fit” recreation center. § Discussion about past success, what they need from us, to have capacity SESSION 2: COLLABORATIVE LEARNING PROJECT

Assessment • Intervention: § Discussion of readiness to implement – staff to refer from clinic, to deliver program § Opportunity to discuss evidence base for program § Each site has different resource needs, identified that the key resource is personnel – referral coordinator on the clinic side, program staff on community side SESSION 2: COLLABORATIVE LEARNING PROJECT

Identified Gaps and Solutions • Referrals cannot be EMR-embedded, must have HIPAA, paper will slow down clinicians § Created standardized referral form that can be a letter, line for parent signature • Program staff have limited data skills for entering data § REDcap data entry page simplified and streamlined, only requires attendance/program description • IRB processes is very intimidating § Research assistant helps staff through CITI modules, minimize number needed • Informed consent requirement leading to low accrual § E-consent, video delivered SESSION 2: COLLABORATIVE LEARNING PROJECT