Using Study Design to Determine the Level of

Using Study Design to Determine the Level of Evidence Critical Appraisal Skills depend on understanding the relationship between study design and levels of evidence Al Best, Ph. D Perkinson 3100 B ALBest@VCU. edu V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Evidence-Based Dentistry n Derek Richards "Is it worth reading this paper? " Evidence-based Dentistry (2000) 2, 50 -52. Integration of: Best Evidence Available Clinical Judgment Patient Values/Circumstances V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Critical Appraisal Skills Are the results of the study valid? n What are the results? n Will the results help locally? n Best Evidence Available V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Review Yeudall and Gordon – EBD Methodology / Practice – PICO PECO questions n Types of Research and Study Design – Levels of Evidence n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

EBD Methodology / Practice 1. ASK an answerable question ASK 2. SEARCH for the evidence ACQUIRE 3. EVALUATE the evidence APPRAISE 4. APPLY the evidence APPLY 5. EVALUATE the outcome ASSESS V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Niederman, R. et al. JADA 2011; 142: 4: 364 -367 Figure The evidence pyramid displays the quality of evidence according to type of study. “Quality” refers to the likelihood of predicting what would occur in one’s own practice (and the least probability of bias). Guidelines explicitly based on this evidence pyramid offer clinicians a simplified mechanism for obtaining and potentially using the knowledge identified in this pyramid. Conversely, the lowest level of evidence, with the least likelihood of predicting what would occur in one’s practice (and the highest probability of bias), can provide useful background information (such as laboratory and animal studies, cross -sectional epidemiologic studies, and expert opinion or narrative reviews). “Filtered information” is so-called secondary research. These reports systematically search for, critically appraise, distill and present the results of primary research, called Vhere “unfiltered information. ” Adapted from Harvey Cushing/John Hay Whitney Medical Library, Yale University I R G I N I A C O M M O N W E A L T H U N I V E R. S I T Y

Niederman, JADA 2011 “a significant evolution from – intuition-based care, – to experience-based care, – to evidence-based care. n The intuition, experience and now evidence on which we base our care simply steps on the evidence pyramid…” n pubmed/21454838 or JADA V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Niederman, JADA 2011 n “Those who are proponents of experience-based care … place themselves and their patients at risk, clinically and legally. n Clinically, it is unlikely that the majority of clinicians are implementing the current best evidence. ” V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

"The Toothpuller", ascribed to Caravaggio (1571 -1610). V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

My goals for you Be able to answer four questions: n Based on the study design, what is the level of evidence? n How were threats to validity addressed? n Based on the goals of the study, How do you describe the results? n To justify the conclusions, were comparisons done appropriately? V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Example A 63 year old male patient has a heart murmur caused by an aortic valve prolapse and is scheduled for tooth cleaning. n Do you prescribe an antibiotic to be taken before dental treatment? n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Google: does tooth cleaning cause bacteremia? Bacteremia can have several consequences. The immune response to the bacteria can cause sepsis and septic shock, which has a relatively high mortality rate. V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Meta Cognition “a significant evolution from – intuition-based care, – to experience-based care, – to evidence-based care. n The intuition, experience and now evidence on which we base our care simply steps on the evidence pyramid…” n Niederman, R. et al. JADA 2011; 142: 4: 364 -367 V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Example A 64 year old male patient has a heart murmur caused by an aortic valve prolapse and is scheduled for tooth cleaning. n Do you prescribe an antibiotic to be taken before dental treatment? n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

A 63 year old male patient has a heart murmur caused by an aortic valve prolapse and is scheduled for tooth cleaning. Do you prescribe an antibiotic to be taken before dental treatment? n PICO / PECO –Population / Participants / Problem –Intervention / Exposure –Comparator / Control –Outcome V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Appraise: Forner, JClin. Perio 2006 pubmed/16677328 V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Appraise: van der Meer, Arch Inter Med. 1992 pubmed/1520053 V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Appraise: van der Meer, Lancet 1992 pubmed/1346008 V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Appraise: Lockhart, Circulation 2008 pubmed/18541739 V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Appraise: Friend, USA Today The most common strain of bacteria in dental plaque can cause blood clots that induce heart attacks when they escape into the bloodstream, researchers said Monday. V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

V I R G I N I A C O M M O N W E A L T One glossary: http: //ktclearinghouse. ca/cebm/glossary/ H U N I V E R S I T Y

Which Study Design? Focus on – Selection of participants – Interventions compared – Flow of time – Outcomes measured n Also – Purpose – Advantages & Disadvantages n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Case Report, Case Series n n n Characteristics: The description of interesting observations. No controls. Purpose: to describe. Generate hypotheses. Advantages: Simple; easy. Disadvantages: Not conclusive. Bias: Findings due to chance? V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Cross-sectional Study n n n AKA: a survey, a poll Characteristics: data collected on subjects at one point in time. Often looks at prevalence. Purpose: What is happening right now? Advantages: Short duration. Disadvantages: Short duration. Only looks at “here and now, ” not across time. Bias: The inclusion/exclusion of subjects is critical. Representativeness: The sample is representative of what? Source: Figure 2 -2, Dawson & Trapp. Basic and Clinical Biostatistics V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Longitudinal Study Types (usually) Retrospective Case-Control n (usually) Prospective Cohort Study n (occasionally) Retrospective Cohort Study n Source: Figure 2 -5, Dawson-Trapp V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Case-Control Study Characteristics: Begins with an outcome of interest, then look back to detect risk factors. n Purpose: to explain outcomes by evaluating previous events. What happened? n Advantage: Good for rare diseases or for those that develop over a long time period. – Quick and easy. n Disadvantage: The largest number of possible biases and errors. Strongly depends on highquality historical records. n Bias: The major problem is the selection of the control group. Source: Figure 2 -1, Dawson Trapp n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Prospective Cohort Study Characteristics: Follow a cohort—a group of individuals with something in common—across time. Note that some are exposed to the risk factor. n Purpose: What will happen? n Advantage: Studies the normal course of events. n Disadvantage: Takes time n Bias: Generally less subject to bias than a case-control study. Source: Figure 2 -3, Dawson Trapp n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Retrospective Cohort Study AKA: historical cohort study n Characteristics: Attempt to follow a cohort back in time. Try to determine the risk factor. n Purpose: How did disease happen? n Advantage: Studies the normal course of events. n Disadvantage (major): depends on near perfect records. n Bias: Perhaps as biased as a case-control study. Source: Figure 2 -4, Dawson Trapp n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Experimental Studies In vitro (eg, cellular assays) n Animal models n Humans (AKA Clinical Trials) n Note: Studies with concurrent controls are highest quality. Historic controls are problematic. n We will talk about random allocation and blinding/masking next time. n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Randomized Control Trial (RCT) Characteristics: Specific inclusion of subjects, controlled exposure to risk factor or intervention, defined outcome measure(s) n Purpose: Does this cause that? n Advantages: Masked interventions, Blind assessment, Gold standard n Disadvantages: Difficult, expensive, time consuming n Bias: Are subjects who consent typical? Source: Figure 2 -6, Dawson-Trapp n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Cross-over Trial n Characteristic: All subjects get both treatments Source: Figure 2 -7 Dawson Trapp V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Systematic Reviews n Narrative review n Meta-Analysis n Practice Guidelines V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Deciding Evidence-Based Guideline Systematic Review Yes Guideline or Systematic Review Yes Experiment, maybe a RCT Yes Cohort Study Yes Case-Control Study or Case Series No Exposure or intervention controlled by the investigator No Subjects followed over time No Subjects selected according to outcome No Descriptive or Analytic PICO PECO Opinion, other PO Cross sectional / Survey V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Study Design ≈ Level of Evidence E-B Guideline or Systematic Review RCT Experiment, maybe a RCT Experimental trial Prospective Cohort Study Retrospective Cohort Article Synopses Case-Control Study or Case Series Narrative, Opinion, other Animal, In Vitro Cross sectional / Survey V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

My goals for you Be able to answer four questions: n Based on the study design, what is the level of evidence? n How were threats to validity addressed? n Based on the goals of the study, How do you describe the results? n To justify the conclusions, were comparisons done appropriately? V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

A mother asks how often her family should come in for a dental checkup for the prevention of disease. How Often? n PICO / PECO –P – I E –C –O V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Mettes D: Insufficient evidence to support or refute the need for 6 -monthly dental check-ups EBD 2005(6): 62 -63 Results: “Only one study (with 188 participants) was Results: “Only included in this review and was assessed as having a high risk of bias. This study provided limited data for dental caries outcomes (dmfs/DMFS increment) and economic cost outcomes … n Conclusions: “There is insufficient evidence from randomised controlled trials (RCT) to draw any conclusions regarding the potential beneficial and harmful effects of altering the recall interval between dental check-ups. … pubmed/16184154 2013 Cochrane review: same conclusions n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

In-class Exercise Cross-sectional Survey Case-Control Retrospective Cohort TODAY Prospective Cohort RCT V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

A mother asks how often her family should come in for a dental checkup for the prevention of disease. How Often? n PICO –P children –I 6 -month recall –C 12 -month recall –O Mean dmft/DMFT V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Groups n 4 -5 people n One reporter n Take 5 minutes V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Case-Control Study n Selection of participants n Intervention n Measurement variables – What is measured/recorded? – When? V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Cross-sectional Survey n Selection of participants n Intervention n Measurement variables V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Prospective Cohort n Selection of participants n Intervention n Measurement variables V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Randomized Control Trial n Selection of participants n Intervention n Measurement variables V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

My goals for you Be able to answer four questions: n Based on the study design, what is the level of evidence? n How were threats to validity addressed? n Based on the goals of the study, How do you describe the results? n To justify the conclusions, were comparisons done appropriately? V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

“I just know” replaces systematic reviews at top of evidence pyramid “After much research and deliberation, we feel we cannot ignore what a parent or conspiracy theorist feels “in their gut”. There are just too many anecdotes and too many people buying untested alternative health products to ignore this any longer, ” explained spokesman Dr. Harold Rami. Homeopaths, Naturopaths, Chiropractors and anti-vaxxers the world over are celebrating this as a huge victory. “Even though my son was showing signs of autism before he got vaccinated, I know in my gut and in my heart that it was still the vaccines that caused it, ” said mother and anti-vaccine advocate Cheryl Jones. “This is a big win for us, ” said Naturopath and homeopathy dispenser, Paul Theroult. “I have seen it many times. I sell my patients a homeopathic remedy – for say the common cold- and then bam, they are cured within one to two weeks. There is no science backing up my claim that the homepathic pill cured their cold, but in my V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y gut I just know it did. ” thespudd. com