Logic models and qualitative synthesis Ideas and issues

Logic models and qualitative synthesis Ideas and issues

2 Outline • Why logic models? • What are logic models? • How do I construct a logic model? • Qualitative synthesis • Where does and does it not get you? 12/20/2021 © The University of Sheffield

3 Examine processes and systems Link practice to outcomes Healthcare research Include multiple sources of evidence 12/20/2021 © The University of Sheffield Explore complex relationships Identify further research needs

4 Logic models/conceptual models/impact models A systematic and visual way to present relationships and outcomes Read from left to right following a chain of reasoning 12/20/2021 © The University of Sheffield

5 Health promotion literature Conveying connections between a programme activities and outcomes (the logic of the programme) Lando et al. , 2006 12/20/2021 © The University of Sheffield

6 Inputs Activities Shortterm outcomes Mid-term outcomes Long-term outcomes Lando et al. , 2006 12/20/2021 © The University of Sheffield

7 Programme evaluation Relationships among programme resources planned activities, and anticipated changes or results The Kellog Foundation, 2004 12/20/2021 © The University of Sheffield

8 Resources /inputs Activities Planned work Outputs Outcomes Impact Intended results W. K. Kellog Foundation, 2004 12/20/2021 © The University of Sheffield

9 Public health Explore ways in which intervention impacts on different levels of outcomes Joly et al. , 2007 12/20/2021 © The University of Sheffield

10 Inputs Strategies Outputs Shortterm outcomes Intermediate outcomes Long-term outcomes Joly et al. , 2007 12/20/2021 © The University of Sheffield

11 Worked example • We are examining interventions to improve the happiness of systematic reviewers • We can find little published evidence reporting happiness outcomes • We are wanting to model how an intervention might help reviewers 12/20/2021 © The University of Sheffield

12 Intervention Programme of lunch time group walks Short term outcomes Medium term Personal contact Social support Heart rate Reduce adverse stress Biscuit intake Headaches/ back/neck Caffeine Improved fitness Reduction of pain Improved diet Cortisol 12/20/2021 © The University of Sheffield Longer term impacts Improved mental health Increased happiness Improved physical health

13 Building on the method 12/20/2021 © The University of Sheffield

14 NICE Centre for Public Health Excellence What interventions are effective in enhancing mental wellbeing in the workplace? 12/20/2021 © The University of Sheffield

15 • Previous systematic review (Graveling et al. 2008) lack of evidence for effectiveness of interventions • BUT was this lack of evidence or lack of study designs meeting inclusion criteria? 12/20/2021 © The University of Sheffield

16 2883 de-duplicated retrieved citations Are we missing something? Reject title/abstract 2584 Full paper reviewed 299 Reject 297 Language 2, population 20, not relevant 43, Not peer reviewed 4, discussion/review 102, study design 101 Included papers 2 12/20/2021 © The University of Sheffield

17 Expert reference group identified literature in addition to papers from systematic review Proposal to use logic modelling to explore relationships 12/20/2021 © The University of Sheffield

18 Review paper 45 Discussion paper 35 Survey reporting associations 31 Survey reporting prevalence 30 Policy document/report 29 Cohort study 19 Book 10 Meta analysis 8 Cluster randomised controlled trial 5 Case study 4 Book chapter 3 Qualitative study 2 Randomised controlled trial 1 Controlled before and after study 1 Case control study 1 12/20/2021 © The University of Sheffield

19 12/20/2021 © The University of Sheffield

Logic model creation Little detail regarding the method • “was drafted by the authors” • “was developed in conjunction with an expert panel” 12/20/2021 © The University of Sheffield • “A group with a facilitator identified activities, outcomes and resources needed, initial model circulated, then refined and clarified” • A group reviewed existing models, identified principles of new model, the model was refined following feedback 20

21 A systematic way to build a model? 12/20/2021 © The University of Sheffield

22 Systematic review methods Primary qual data analysis Logic model 12/20/2021 © The University of Sheffield Qualitative synthesis

23 Method Analysis Data extraction Synthesis 12/20/2021 © The University of Sheffield

Method Analysis Extraction Description of elements impacting on wellbeing and associations between elements extracted on developed form Synthesis 12/20/2021 © The University of Sheffield 24

25 WARNING Reviewers of a nervous disposition look away now Treating all data equally Treating all data as qualitative 12/20/2021 © The University of Sheffield

Method 26 Analysis Extraction Line by line reading of extractions, coding of data, use of CAQDA software (Nvivo) for storage and retrieval Synthesis 12/20/2021 © The University of Sheffield

27 Qualitative synthesis • Combining of primary qualitative data • Emerging approaches (Meta-synthesis, thematic synthesis, meta-ethnography etc) • Seeking and developing recurring patterns or themes across the data 12/20/2021 © The University of Sheffield

28 Thematic synthesis (Thomas & Harden 2008), examine themes across set, and by process of comparison and contrast establish meta-themes (integrate and/or create) 12/20/2021 © The University of Sheffield

For example 29 Primary analysis – a paper describes outcomes following a buddy system intervention on employee wellbeing, elements of data coded as peer interaction. Another paper describes the link between working at home and reduced wellbeing, elements of data coded as social interaction. Secondary analysis – peer interaction and social interaction data combined into social support theme, later when other data added a category of social resources is created. 12/20/2021 © The University of Sheffield

Wider influences Intervention points Work context National polices and practices Health and safety Management priorities/values Supervisor behaviour Feedback & appraisal Organisational climate Organisational justice Work-place support Employee participation Communication systems Industry type Equal opportunities Anti-discrimination policies Family-friendly policies Maximum working hours legislation Economic And social trends Fixed term contracts Flexible employment Health inequalities Cultural diversity Demographic changes New technology ICT Globalisation Change from production to service economy 30 Outcomes Behavioural outcomes Health-related behaviour Sickness absence Staff turnover Presenteeism Burnout Citizenship Work content Business outcomes Work demands Job control/decision latitude Effort required Rewards Role Working schedules Opportunity for learning/development Monotony Skills utilisation Worthwhile/fulfilling Stability/job future Occupational group Absence/turnover costs Performance Productivity Customer satisfaction Profitability Optimal staffing Satisfy statutory regulations Safety/avoidance of litigation Corporate social responsibility Corporate image Recruitment cost savings Attracting and retaining skilled workforce Competitiveness Insurance premiums/healthcare costs Health Well being Job satisfaction Individual factors Individual characteristics Gender Age Social circumstances Education Ethnicity Marital status Predispositions Individual response to work Personality traits Self esteem Self efficacy Psychological flexibility Person-environment fit Values Social resources Individual responses to management style or working practices Optimal workload threshold Coping response 12/20/2021 © The University of Sheffield Attitudinal outcomes Commitment Motivation Engagement Employee expectations Perceptions of fair treatment

Where could logic modelling get you? • May act as a common frame of reference, defining poorly developed concepts • Used as a participatory and communication tool, user involvement? 12/20/2021 © The University of Sheffield 31

32 Provides a systems perspective • Increasing recognition of systems perspective in healthcare practice eg pt safety • Recognition of complexity eg MRC Developing and evaluating complex interventions 12/20/2021 © The University of Sheffield

33 Implementation Research eg Eccles et al. 2009, Cooksey Report Transfer of research findings into practice Complexity of context Fidelity of interventions 12/20/2021 © The University of Sheffield

34 • Used as a research development tool, consider issues that a research agenda needs to address • Takes an inclusive approach to evidence, areas where little trial work • Pilot work, Mapping review? 12/20/2021 © The University of Sheffield

35 Links between advice services for patients, and health outcomes Financial outcomes Advice Health outcomes 12/20/2021 © The University of Sheffield

36 Where does it not get you? • Effectiveness • Issues with identification of material – selection bias • Issues with hierarchy of evidence – representation of strength? • Hierarchy of qual designs? Frequency? Shows where research has been done rather than evidence? 12/20/2021 © The University of Sheffield

37 More issues • Critical appraisal especially qualitative sources • Issues with integration of diff types of data • Removal of context • Use of primary material in secondary synthesis • Integration versus interpretation 12/20/2021 © The University of Sheffield

38 Logic is the beginning of wisdom, not the end. Leonard Nimoy Logic will get you from A to B. Imagination will take you everywhere. Albert Einstein 12/20/2021 © The University of Sheffield

Unknown author 39 Logical Model for Creating Achievable and Sustainable Change Modalities over Time Realize/Engage Target audience Actualize/Energize Create message Promote/Enhance Engage audience Realize that you look important if it’s really confusing. Put paper in printer Expose/Repose Check with Clients Analyze everything you can think of in depth Report Outcomes Engage 1 st gear Realize that you look important if you’re the only one who understands it. Build conceptual engine Hire chimp Engage graphics generator Capture Vision Let Vision Go Re-capture Vision Engage bullshit generator Put orange box here. 12/20/2021 © The University of Sheffield Keep close watch on vision. Detailed cause and effect follows Start thinking about creating a really comfusing chart. Internalize Conceptualize Roll down hill Jump start engine Achieve Goals Finalize Results Realize that even you don’t understand any of this. Wax Eloquent Promise improvement, Exact retribution Wax Car The world is a better place because they actually paid you to do this. Fire chimp

References 40 Baxter R, Killoran, A, Kelly MP, Goyder E. Synthesizing diverse evidence: the use of primary qualitative data analysis methods and logic models in public health reviews. Public Health, 2010; 124: 99 -106. Barbour R, Barbour M. Evaluating and synthesising qualitative research: the need to develop a distinctive approach. J Eval Clin Prac 2003; 9: 179– 86. Bunton R, Macdonald G, editors. Health promotion: disciplines, diversity and developments. 2 nd edn. London: Routledge; 2002 Cooksey D, A Review of UK health research funding. Norwich: HMSO; 2006. Eccles M, Armstrong D, Baker R et al. An Implementation Research Agenda. Implementation Science, 2009: 4: 18 doi: 10. 1186/1748 -590804018. Graveling R, Crawford J, Cowie H, Amati C, Vohra S. A review of workplace interventions that promote mental wellbeing in the workplace. Edinburgh: Institute of Occupational Medicine; 2008. Joly B, Polyak G, Davis M, Brewster J, Tremain B, Raevsky C, et al. Linking accreditation and public health outcomes: a logic model approach. J Public Health Manag Prac 2007; 13: 349– 56. Kelly M, Stewart E, Morgan A, Killoran A, Fischer A, Threlfall A, Bonnefoy J. A conceptual framework for public health: NICE’s emerging approach. Public Health 2009; 123: 14– 20. Kellog. Foundation, The logic model development guide. Michigan: Kellog Foundation; 2004. Lando J, Williams S, Williams B, Sturgis S. A logic model for the integration of mental health into chronic disease prevention and health promotion. Prev Chron Disease Pub Health Res Prac t Policy 2006: 3; 1 -4. Schmitz C. Everything you wanted to know about logic models but were afraid to ask. Michigan: Kellog Foundation; 1999. 12/20/2021 © The University of Sheffield

41 Tannihill A. Beyond evidence to ethics: a decision-making framework for health promotion, public health and health improvement. Health Promot Int 2008; 23: 380– 90. Thomas J, Harden A. Methods for thematic synthesis of qualitative research in systematic reviews. BMC Med Res Meth 2008; 8: 45. Tones K, Green J. Health promotion: planning and strategies. London: Sage; 2004. Further reference lists/resources: http: //www. cdc. gov/eval/resources. htm#logic model 12/20/2021 © The University of Sheffield