Introducing Systematic reviews Series of doctoral lectures on

Introducing Systematic reviews Series of doctoral lectures on behalf of the Cochrane Qualitative and Implementation Methods Group http: //methods. cochrane. org/qi/ Systematic Reviews in social sciences. Mixed methods and qualitative research reviewed. Doctoral course, Ughent, March 21 -22, 2016 Prof. Dr. Karin Hannes Centre for Sociological Research Social Research Methodology Group, KU Leuven Karin. hannes@kuleuven. be

Outline • Intro - daily practice: evidence-based? • Systematic reviews – What is it and why do we need them? – How do they look like? – Who is producing them? – Where can we locate them? – What is there (potential) impact? – How can we contribute to them?

Daily practice: Evidence-Based? • 1861: “I firmy believe that if the whole material medica as used now, could be sunk to the bottom of the sea, it would be all the better for mankind and the worse for the fishes. ” (Holmes) • 1979: “Only 10 to 20 percent of all procedures currently used in medical practice have been shown to be efficacious by controlled trial. . . ” (The US congressional office of technology assessment) • How many of our daily health care, educational, criminological, social welfare interventions actually ARE evidence-based? • Let’s Raise our Hand for…

Daily practice: Evidence-Based? 80 to 90 % based on -Magic? -Intuïtion? -Expert opinion -? ? ?

Daily practice: Evidence-Based? • “EBP is the integration of the best available research with (clinical) expertise in the context of patient characteristics, culture and preferences. ” American Psychological Association. (2006). APA presidential task force on evidence based practice. Washington, DC: Author. • We all want the best for our target group • We have always been evidence-based… • Our academic education guarantees a sufficient level of scientific knowledge, skills and attitudes Half of what you learn in school will be shown to be either dead wrong or out-of-date within 5 years of your graduation; the trouble is that nobody can tell you which half! (Dr. Sydney Burnwell)

Daily practice: What is evidence? • Evidence of ‘effectiveness’: the extent to which an intervention, when used appropriately, achieves the intended effect. QUANTITATIVE REVIEWS MIXED METHOD REVIEWS • Evidence of ‘feasibility’: the extent to which an intervention is practical and practicable, whether or not an intervention is physically, culturally or financially practical or possible within a given context. • Evidence of ‘appropriateness’ the extent to which an intervention fits with a situation, how an intervention relates to the context in which it is given. • Evidence of ‘meaningfulness’: the extent to which an intervention is positively experienced by the population and relates to the personal experience, opinions, values, beliefs and interpretations of the population. QUALITATIVE REVIEWS

PART 1: SYSTEMATIC REVIEW WHAT IS IT AND WHY DO WE NEED THEM?

What is it? The simplistic version A systematic review is a ‘systematic identification, evaluation and synthesis of all relevant studies regarding a specific topic, based on an explicit and pre-defined methodology’ If high quality research studies exist. . . Summarize them!

What is it? Predefined format: To help review authors be systematic To help people reading the reviews to find information quickly Transparancy Reliability

Explosive growth. MEDLINE: Meta-analysis (Publication Type) OR “systematic review” Systematic review on published qualitative evidence syntheses (Hannes & Macaitis, 2012)

Why do we need them? “Over two million articles are published annually in the biomedical literature in over 20, 000 journals – literally a small mountain of information … a stack … would rise 500 metres” Cynthia Mulrow, in Systematic Reviews (BMJ Publishing Group, 1995) And within that stack, we would be looking for a needle… Control for bias! Garbage in – garbage out l l Provide reliable basis for making decisions Inform and influence future research

Traditional versus systematic review • Lack rigor • Methodology not transparent • Different reviewers reach different conclusions • Become out of date • Scientific rigor to minimise bias • Explicit and reproducible methodology • Regularly updated (Cochrane/Campbell)

Key characteristics of a SR 1. A clearly stated set of objectives with pre-defined eligibility criteria for studies 2. An explicit, reproducible methodology (or at least a good audit trail)

Key characteristics of a SR 1. A clearly stated set of objectives with pre-defined eligibility criteria for studies 2. An explicit, reproducible methodology 3. A well defined, systematic search that attempts to identify all studies that would meet the eligibility criteria (or at least a rational statement of the choice for selecting particular studies)

Key characteristics of a SR 1. A clearly stated set of objectives with pre-defined eligibility criteria for studies 2. An explicit, reproducible methodology 3. A systematic search that attempts to identify all studies that would meet the eligibility criteria 4. An assessment of the methodological quality of the findings of the included studies (or at least a statement on why such a quality assessment has not been conducted)

Key characteristics of a SR 1. A clearly stated set of objectives with pre-defined eligibility criteria for studies 2. An explicit, reproducible methodology 3. A well defined, systematic search that attempts to identify all studies that would meet the eligibility criteria 4. An assessment of the methodological quality of the findings of the included studies (assessment of risk of bias ) 5. A systematic extraction, synthesis, and presentation of the characteristics and findings of the included studies

Key characteristics of a SR • Answers a specific question • Aims to identify and include all relevant studies • Uses pre-planned methods and strategies (to guarantee the quality of the process) • Quantitative reviews MAY include META-ANALYSIS: statistical synthesis of the results of included studies, or a narrative form of analysis • Helps practitioners and their target groups to make decisions about appropriate care, programs, therapies or how to improve them. • Provide a scientific rather than subjective summarisation of literature • Can reveal “new” evidence • Help decision makers to cope with the sheer volume of literature by summarising it • More reliable evidence base with which to aid decision making

PART 2: SYSTEMATIC REVIEWS HOW DO THEY LOOK LIKE?

What is a meta-analysis? Optional part of a systematic review Systematic reviews Meta-analyses • To identify a common effect among a set of studies • To improve precision of an estimate • To investigate whether the effect is constant • To answer controversies arising from conflicting studies or to generate new hypotheses

Meta-analysis Skills training versus usual curricula in the prevention of drug use in school kids. Gives a more precise estimate of effect than when derived from the individual studies included within a review 1. Summary statistic + CI for each study individually (RR, OR, RD…) 2. Pooled intervention effect + CI is calculated as a “weighted average” Intervention no effect Comparison

Meta-analysis Skills training versus usual curricula in the prevention of drug use in school kids. Facilitates investigations of the consistency of evidence across studies, and the exploration of differences across studies. Intervention no effect Standard

CC & C 2: meta-analyse ‘Skill-based interventions’ prove to be an effective strategy to prevent from marihuana use in schoolkids.

Example: Narrative report of quantitative studies e n u s a p e t s v i e t ’ a n t i i l c a u Ce q e s è h t n sy Narrative synthesis because pooling is not possible? Rather not labelled as QES…

What is a Qualitative Evidence Synthesis ‘A process of summarizing qualitative research findings, either aggregative or interpretive, by comparing and analysing texts derived from multiple accounts of an event or situation as reported in basic qualitative research studies. ’ Explore questions such as • how do people experience a phenomenon or what does it mean to them? • why does an intervention or program work (or not), for whom and in what circumstances…? • what are the barriers and facilitators to accessing care, school, society, …? • what impact do specific barriers and facilitators have on people, their experiences and behavior?

What is a Qualitative Evidence Synthesis Qualitative Inquiry Qualitative Science Framework Thematic synthesis Meta- aggregation ethnography Grounded Theory Meta. Critical Ecological narrative Interpretive triangusynthesis lation

Qualitative Evidence Synthesis Example Barroso J, Powell-Cope GM. Metasynthesis of Qualitative Research on Living with HIV Infection. Qual Health Res vol 10, nr 3, 2000. – – Understand experience of adults living with HIV infection 21 articles Method: Formal grounded theory: constant comparative analysis Themes: • • • Finding meaning in HIV: death, loss, surviving Shattered meaning: fear, irreparable Human connectedness: meaningfull relationships alienation, isolation Focusing on the self: actions to enhance fysical and emotional health Negociating care: active role Dealing with stigma: personal, family, society

Qualitative Evidence Synthesis Example Adolescents’ experiences and perceived effectiveness of self-help groups for addiction: A systematic review of qualitative research evidence

Qualitative Evidence Synthesis Example A Meta-Ethnography of Paid Dementia Care Workers’ Perspectives on Their Jobs Cheryl Cook , B. A. , Sherri Fay , M. A. , Kenneth Rockwood , MD, FRCPC, FRCP DOI: http: //dx. doi. org/10. 577 0/cgj. 15. 37

PART 3: SYSTEMATIC REVIEWS WHO IS PRODUCING THEM?

Archie Cochrane’s challenge “It is surely a great criticism of our profession that we have not organised a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomised controlled trials. ” (1979)

Cochrane Collaboration: 1993 an international not-for-profit organisation which aims to help people make well-informed decisions about healthcare by preparing, maintaining and promoting the accessibility of systematic reviews of the effects of health care interventions

Centres: Help organise and register review groups Facilitate collaboration among reviewers Provide training and consultation Establish liaisons Promote the Cochrane Collaboration Provide unique contribution Cochrane Collaboration Steering Group: Policy making body of the Cochrane Collaboration. Elected representatives of each of the Collaboration entities Review Groups: Focus on Treatment of disease or health problems To prepare and maintain systematic reviews To develop a specialised trials register Collaborative Review Groups Steering Group Fields: Represent a population, group, or type of care that overlaps multiple Review Group area. Examples: Primary Health Care, Health Care of Older Adults, Complementary Medicine Centres Methods Groups: Methods Groups. Develop methods and Fields products integral to internal functioning of the Collaboration The Develop state of the art Consumer methods for systematic Network review Examples: Statistical, Consumer Network: Economics, Placebo Provides consumer input Effects, Informatics, Helps set priorities Qualitative Helps with dissemination

Our challenge: 2006! It is increasingly being Recognized that evidence from qualitative studies that explore the experience of those involved in providing and receiving interventions, and studies evaluating factors that shape the implementation of interventions, have an important role in ensuring that systematic reviews are of maximum value to policy, practice and consumer decision-making.

Campbell Collaboration (C 2): 2000 an international organisation which aims to help people make wellinformed decisions about public policy (crime & justice, education, social welfare) by preparing, maintaining and promoting the accessibility of systematic reviews of the effects of social and behavioural interventions

Steering group: strategic and policy making body Campbell Collaboration Coordinating Groups: Provide editorial services and support to authors of Campbell reviews and build links with users of systematic reviews: • • Crime and Justice Social Welfare Education International development Partners: Institutes who have a formal agreement with Campbell and support the same goals. Coordinating Groups Partners Steering Group Users Group Methods Groups: Improving the methodology of research synthesis, and disseminating guidelines for state-of -the-art review methods Users group: Knowledge translation to increase the impact of Campbell reviews in policy and practice arenas, and to make the information more accessible

CC & C 2 Collaboration goals • To ensure high quality, up-to-date systematic reviews are available across a broad range of topics • To promote access to systematic reviews • To develop an efficient, transparent organisational structure and management system for the Collaboration • To achieve sustainability of the Collaboration

Principles • • • Collaboration Building on the enthusiasm of individuals Avoiding duplication Minimising bias Keeping up to date Striving for relevance Promoting access Ensuring quality Continuity

PART 4: SYSTEMATIC REVIEWS: WHERE CAN WE LOCATE THEM?

The Cochrane Library • Database of Systematic Review • Database of Abstracts of Reviews of Effectiveness • Controlled Trials Register • Database of Methodology Reviews • Methodology Register • Health Technology Assessment Database • NHS Economic Evaluation Database http: //www. thecochranelibrary. com /view/0/index. html

http: //www. ca mpbellcollabor ation. org/lib/

PART 5: SYSTEMATIC REVIEWS WHAT IS THEIR (POTENTIAL) IMPACT?

Evaluate a solution • Situation: 1 mj. people die each year due to traffic accidents • Solution: Educate the drivers • Systematic Review: 24 studies on education, no evidence that it reduces traffic accidents • Recommendations: Stimulate teleconferences and the use of safe, public transport “Driver education is big business – our results show that it is also a big con” (Prof. Roberts)

Identify gaps in ‘evidence’ • Situation: anti-psychotics are subscribed for pregnant women with a psychotic disease • Systematic Review: No literature on adverse effects of anti-psychotics on the mother or the development of the (unborn) baby – – • RCT’s with pregnant women are considered unethical Medicals’ referals are based on habits and opinions Recommendation: Research on the effects of anti-psychotics in pregnant women is necessary “The continued use of antipsychotic drugs in women during pregnancy and lactation without sound evidence raises serious clinical and ethical concerns. ” (Webb)

Identify solutions that cause harm • Situation: Illinois law: Chicago Public Schools mandated to identify children at-risk for future criminal behaviour. • Solution: Scaired straigth programs: take them on tours of adult prison facilities • Systematic Review: Not only does a scared straight program fail to deter crime, it actually leads to more offending behaviour. “Governments need to adopt rigorous evaluation to ensure that they are not causing more harm to the citizens they intend to protect” (Dr. Petrosino).

Detect under which circumstances programs may work • Situation: A Cochrane review found that school feeding programmes significantly improve the growth and cognitive performance of disadvantaged children, however the results from the included trials were heterogenic. • Systematic Review: The trials had many different designs and were implemented in varying social contexts and educational systems; by staff with different backgrounds, skills, and cultural beliefs; and with huge variation in the prevailing social, economic, and political context. • Recommendation: Programmes are more likely to be effective when designed in partnership with the local community and interventions are piloted. In situations of absolute poverty even severely malnourished children may not benefit from school feeding programmes because they may receive less food at home. “Simply knowing that feeding programmes work is not enough for policymakers to decide on the type of intervention that should be implemented. “( Dr. Greenhalgh)

Unravel the complexity of a situation • Situation: Governments highly invest in smoking cessation programs for employees via their employers, to reach a large amount of people • Systematic Review: A consideration of the qualitative evidence of employees’ views regarding workplace interventions or programmes might help to explain outcomes, that is, why some interventions might work and others not. • Recommendation: Workplace interventions should employ a range of different elements if they are to prove effective in reducing smoking among employees, because an employee who feels ready and able to change behaviour has different needs and preferences from an employee who is not at that stage. “There is nothing to suggest that the workplace itself offers a particularly special setting for smoking cessation or reduction interventions, as the intervention preferences described and the key concepts of priority and perceived ability to quit can all exist outside the workplace. “( Dr. Caroll)

Authors of SR can make a difference! Anyone who wants to make a contribution. . . • www. cochranecollaboration. org • www. campbellcollaboration. org

1. Select a topic 2. Make a review team 3. Register your title PART 6: SYSTEMATIC REVIEWS HOW CAN WE CONTRIBUTE TO THEM?

1. Select a topic • Learn to ask questions (Treatment or Intervention): S setting • P population • I intervention I (topic of) Interest • C comparison (c) comparison • O outcome E evaluation PICO example Does providing information (I) at school have a preventive effect on the use of drugs (O) in high school students (P) compared to sanctions when someone is caught using drugs (C) ? SPICE example How do Western (S) high school students with a drug and/or alcohol problem (P) perceive or experience (E) self help groups (I) (compared to residential treatments (C))?

1. Select a topic “Information campagnes in the prevention of drug use” Scope of the question? “Does information provision at school decreases the risk of drug use in high school students (compared to the standard approach)? BROAD (LUMP) Advantages NARROW (SPLIT) l Comprehensive l Manageable for review team l Generalizibility l Easy to read l Effect modifiers (why different effect between apples and oranges? ) Disadvantages l More resources l Evidence sparse? l “mixing apples and oranges” l Lack of generalizibility l Choice of scope “biased” by authors

2. The review team Why more than 1 person? • Detection of errors (selection of eligible studies, data extraction) • Reduces risk of bias • When more than one person / team is interested in topic Who should be in the team? • Expertise in topic area • Expertise in systematic review methodology • Incorporate view of “users” (clinicians, consumers) • Address questions that are important to people • Take account of outcomes that are important to those affected • Make it accessible to people making decisions • Reflects variability in populations, settings…

3. Registering a title 2. Contact a Cochrane Review Group (CRG) or Campbell Coordinating Group (CCG) Cochrane Review Group: • > 52 • Specific condition (eg incontinence, learning & developmental disorder group) • Group of diseases (eg ear nose and throat disorders group) • Organ (eg renal group, heart group) • Patient (eg neonatal group, pregnancy and Campbell Coordinating Group: • 4 • Crime & Justice CG • Education CG • Social welfare CG • International Development childbirth group, pain and palliative care) There is a potential to co-register reviews!

3. Registering a title 3. Fill in “Title Registration Form” 4. Review by CRG/CCG editors • Topic fits in with CRG/CCG? YES • Questions specific and answerable? YES • Overlap with ongoing review? NO • Review team has necessary expertise? YES Good luck with making the protocol!

Register title with CRG Title registered No Contact other CRG Contact other review team Write protocol Peer review Protocol published Write review Peer review Review published Update review Peer review Update of review published

Systematic Review Summary Most commonly used procedure • • • Establish objectives, selection criteria Search for eligible studies Apply selection criteria Assess study quality, extract data Analyse results using meta-analysis, if appropriate and possible (quant part) and/or a particular form of meta-synthesis (qual part) • Perform sensitivity analyses • Prepare report (keep it up-to-date!)

Want more? Cochrane handbook Qualitative evidence synthesis