Pharmacy Critical Appraisal Level 2 Evaluating studies using

Pharmacy Critical Appraisal Level 2 Evaluating studies using the Critical Appraisal Skills Programme (CASP) Tools Quality Education for a Healthier Scotland

Pharmacy What is Critical Appraisal? Critical appraisal is the process of systematically examining research papers before using the evidence to form a decision Critical appraisal allows us to make sense of research evidence and close the gap between research and practice Quality Education for a Healthier Scotland

Pharmacy Why Critically Appraise? • • In evidence based medicine the clinician should use the best available evidence to decide which treatment option is best for their patient To determine the best treatment option we must have critical appraisal skills to assess the quality of research Quality Education for a Healthier Scotland

Pharmacy Advantages of Critical Appraisal • • Provides a systematic way of assessing the validity, results and usefulness of published research Allows us to improve healthcare quality Encourages objective assessment of the usefulness of all information/evidence Critical appraisal skills are not difficult to develop – mostly common sense! Quality Education for a Healthier Scotland

Pharmacy Disadvantages of Critical Appraisal • • • Can be time consuming It may highlight that current practice is in fact ineffective May highlight a lack of good evidence in an area of interest Quality Education for a Healthier Scotland

Pharmacy Help with Critical Appraisal • • • It is difficult to remember all the issues which must be taken into account when reading research papers To help with this the Public Health Resource Unit in Oxford have produced tools to evaluate most types of study as part of the Critical Appraisal Study Programme (CASP) These are called CASP tools Quality Education for a Healthier Scotland

Pharmacy CASP Tools • • • CASP Tools have been developed to help us appraise: Systematic Reviews RCTs Case-control studies Cohort Studies Tools for other types of study (e. g. qualitative research, economic evaluations and diagnostic studies) are also available Quality Education for a Healthier Scotland

Pharmacy CASP Tools (2) The tools break critical appraisal of each study type into 10 -12 manageable questions The questions in each tool will vary slighty as study designs are different They are available at: http: //www. sph. nhs. uk/what-we-do/public-healthworkforce/resources/critical-appraisals-skillsprogramme Quality Education for a Healthier Scotland

Randomised Controlled Trials (RCTSs) Pharmacy RCTs are the best type of research study to determine if a specific intervention produced the desired outcome in a specific population e. g. The HOPE study: Does ramipril (the intervention) prevent myocardial infarction, stroke or death from cardiovascular causes (the outcomes) in patients who were at high risk for cardiovascular events but who did not have left ventricular dysfunction or heart disease (the population) Quality Education for a Healthier Scotland

RCTs (continued) RCTs should have a clearly defined research question in relation to the population, outcome and intervention RCTs can minimise bias and use the most appropriate study design for investigating the effectiveness of a specific intervention or treatment RCTs are, however, not automatically of good quality and should be appraised critically Quality Education for a Healthier Scotland Pharmacy

Appraising RCTs Three broad issues should be considered when examining a report of an RCT: • Validity • Results • Relevance • The CASP tool prompts us to ask questions which will assess each of these issues in more detail Quality Education for a Healthier Scotland Pharmacy

Pharmacy Screening Questions in CASP tool Questions 1 and 2 in the RCT CASP tool are screening questions to assess if: The RCT asks a clearly focused study question (i. e. is population, intervention and outcome being investigated clearly stated? ) Whether an RCT is the most appropriate study design to ask this question If the answer to either of these is “no” then it is probably not worth continuing with the rest of the questions Quality Education for a Healthier Scotland

Pharmacy Validity When assessing validity the methods used in the study are appraised If the research methods are flawed then this may invalidate the results of the trial To critically appraise the methodology of an RCT you will look at sample size, randomisation and baseline characteristics, blinding, follow-up, data collection and interventions. These elements are dealt with by questions 3 -7 in the CASP tool for RCTs Quality Education for a Healthier Scotland

Randomisation and Baseline Characteristics (Q 3 in CASP Tool) Pharmacy Randomisation reduces the possibility of bias Method of randomisation should be robust and neither the participants or clinicians should be aware which group patients will be in before randomisation Most robust methods are computer- generated numbers or tables of random numbers (gives 50: 50 chance of being in either group) Stratification can be used to ensure similar baseline characteristics in both groups Quality Education for a Healthier Scotland

Randomisation and Baseline Characteristics (Q 3 in CASP Tool) The make-up of treatment and control groups should be very similar and the only difference should be the intervention under investigation in the study This is so that we can be confident that the outcome is due to the intervention and not to any other confounding factors Consider: Is the sample large enough? (see also Q 7) Was the randomisation process robust? Was there stratification? Quality Education for a Healthier Scotland Pharmacy

Pharmacy Blinding (Q 4 in CASP Tool) If the people involved in the study are not aware who is in the treatment or control group this reduces the possibility of bias i. e. ideally a study should be double-blind Single-blind and open-label studies would be expected to have higher chance of producing biased results To blind patients and clinicians the intervention and placebo must appear to be identical – this is not always possible Consider – if every effort was made to achieve blinding -if it matters in the study being reviewed (i. e. could there be observer bias in the results that are used? ) Quality Education for a Healthier Scotland

Pharmacy Follow-up (Q 5 in CASP tool) If a large number of participants withdraw from a trial before its conclusion their loss may distort the results Crossover between groups can also distort the results as the effects of randomisation can be lost In most cases intention-to-treat analysis should be used – i. e. the final results should be analysed according to the original randomisation Quality Education for a Healthier Scotland

Pharmacy Interventions (Q 6 in CASP tool) It is important that, aside from the intervention under investigation, the groups are treated equally as this means the outcome of the study can be attributed to the intervention Consider: • Were groups reviewed at the same time intervals? • Were any other treatments allowed in either group? Quality Education for a Healthier Scotland

Pharmacy Interventions (Q 6 in CASP tool) Also consider: • If tests or measurement s were used were they conducted by appropriate personnel and established tests were used? • Were assessments frequent enough to show a pattern of response? • Is the duration of the study sufficient? Quality Education for a Healthier Scotland

Pharmacy Sample Size (Q 7 in CASP tool) • • A study should include enough participants so that the researchers can be reasonably sure that there is a high chance of detecting a beneficial effect A power calculation should be carried out before the participants are enrolled to estimate how many people need to be recruited to achieve a certain level of certainty (usually aim for 80 -90%) Quality Education for a Healthier Scotland

Pharmacy Results The results of a RCT should be scrutinised in a similar way to the method Broad considerations are: • What are the results? • How are the results presented? • How precise are the results? This is dealt with by questions 8 and 9 in the CASP RCT tool Quality Education for a Healthier Scotland

Presentation of Results (Q 8 in CASP tool) Results can be expressed as: • Relative Risk – if we hope that the intervention will lead to LESS outcomes (e. g. MI, stroke, death) we want a hazard ratio of less than 1 • Absolute Risk – proportion of people experiencing an event in each group Results may be reported as either of the above or as both Quality Education for a Healthier Scotland Pharmacy

Pharmacy Presentation of Results (2) The results presented should relate to the objectives set out in the original description of the study method Relative or absolute measures may have little meaning in relation to clinical practice Numbers needed to treat (NNT) may make measures more understandable NNT is the number of people who must be treated to produce one additional successful outcome Quality Education for a Healthier Scotland

Pharmacy Precision of Results (Q 9 in CASP tool) Statistical tests are used to establish whether the results of a trial are real or whether they occurred purely by chance There will always be some doubt as the trial only looks at a sample of the population Confidence intervals and p-values indicate the level of certainty around the results Quality Education for a Healthier Scotland

Pharmacy Confidence Intervals (CI) Inferences based on random samples are uncertain because different results would be obtained each time a study was repeated CIs indicate the range of doubt around the results and represent the range of values within which the true value lies Therefore, the narrower the range, the more convincing the results If CIs overlap this indicates that the study has failed to demonstrate a difference Quality Education for a Healthier Scotland

Pharmacy P-values describe the probability that a result has happened by chance P<0. 05 is usually described as statistically significant and means that the results are unlikely to be due to chance Quality Education for a Healthier Scotland

Pharmacy Statistics Used There should be adequate statistical analysis of the relevant results and the statistical analysis used should be appropriate to trial size and design Statistical tests used should be adequately described and referenced Consider whether a statistician is listed as one of the authors Quality Education for a Healthier Scotland

Clinical vs. Statistical Significance Pharmacy A statistically significant difference in favour of a drug does not always indicate a clinically significant difference e. g. in a study of two antihypertensives a difference in BP of 12 mm. Hg may be statistically significant but is this likely to confer significant clinical benefit? Also consider surrogate markers versus clinical endpoints (e. g. an increase in bone mineral density versus a decrease in fracture rate) Quality Education for a Healthier Scotland

Pharmacy Relevance If the methodology and results are acceptable then the applicability of the results to the local population should be considered Some broad issues include: • Relevance to the local population • Were the outcomes considered clinically important? • Risk versus benefit of treatment Q 10 in the CASP tool prompts us to consider the issues around the relevance of our study to our local population Quality Education for a Healthier Scotland

Pharmacy Relevance to Local Population It is important to consider if there any differences between the participants in the trial and the local population that would make it impossible to apply the results locally. Think about: Inclusion/exclusion criteria – e. g. age, ethnicity, co-morbidities, concomitant medication Local healthcare provision – the setting may have been in a different healthcare system and it may not be possible to provide similar care locally Control group – is the standard treatment used in the study better or worse than local standard? Are realistic comparative doses used? Quality Education for a Healthier Scotland

Pharmacy Importance of outcomes Were all clinically important outcomes considered? A single RCT is unlikely to address all the clinically important outcomes but consider if the original question has been answered and if any other important outcomes have been missed out. Think about outcomes form the point of view of: • The patient and their family/carers • The clinician using the treatment • Policymakers • The wider community Quality Education for a Healthier Scotland

Pharmacy Risks versus Benefits Risks: Safety – the risk of serious side effects may outweigh the benefits of treatment (can calculate NNT vs. NNH) Tolerability – what was the drop-out rate in the study compared to withdrawal rates on the current standard treatment? Is the benefit of the treatment large enough to outweigh these risks? Quality Education for a Healthier Scotland

Pharmacy Other Considerations Cost implications – financial information is not normally included in a trial. Economic evaluations may be available Simplicity of use – patient factors such as compliance, method of administration or complicated devices may limit the usefulness of a treatment in clinical practice Quality of life data – gives more information from the patient’s perspective Sponsorship – sponsorship of the research or author affiliations to drug companies may affect how the results are presented Quality Education for a Healthier Scotland

Pharmacy Look out for……. . A lot of work and funding goes into drug development and clinical trials and drug companies and researchers will want a positive result from their hard work When apprasing RCTs beware of the negative aspects of the study being glossed over to make it appear positive. This can include: 1. Use of sub-group analysis 2. Use of composite end points 3. Analysis of secondary outcomes Quality Education for a Healthier Scotland

Pharmacy Sub-group analysis is when results are broken down into patient sub-groups e. g. elderly, patients with a history of stroke/diabetes If the result of the overall trial is negative it may be positive in a specific sub-group Trials may not be powered to detect differences in sub-groups containing small numbers of patients Quality Education for a Healthier Scotland

Pharmacy Use of Composite Endpoints Sometimes the desired outcome of a clinical trial is relatively rare e. g. fatal MI To show a difference between intervention and control huge numbers of participants in very long term trials would be required Composite end-points may be used instead e. g. risk of MI, stroke, death or hospital admission due to cardiac causes Not all the individual components are always of equal importance to all patients Quality Education for a Healthier Scotland

Pharmacy Analysis of Secondary Outcomes The primary outcome is the most important outcome of an RCT If no statistically significant increase in efficacy is observed with the new treatment then secondary outcomes which are significant may be quoted in the results The number of patient enrolled in a trial is based on the primary outcome It is poor practice to disregard the primary outcome and quote a secondary outcome as the main result Quality Education for a Healthier Scotland

Pharmacy Remember! Healthcare decisions are not usually made on the basis of one trial. Other factors and other evidence may also have to be considered when making a decision Quality Education for a Healthier Scotland

Pharmacy Appraising other publications In addition to RCTs other publications may be used to contribute to evidence-based decision making In order of importance the usual hierarchy is: l Systematic reviews l RCTs l Observational studies e. g. cohort, case-control or cross-sectional studies l Case reports, case studies l Expert consensus Quality Education for a Healthier Scotland

Pharmacy Systematic Reviews Systematic reviews seek to bring the same level of rigour to reviewing research evidence as should be used in producing research evidence. They: • Identify relevant published and non-published evidence • Select studies for inclusion and assess the quality of each • Present a summary of the findings with due consideration Quality Education for a Healthier Scotland

Pharmacy Systematic Reviews (2) Meta-analysis is a statistical technique for combining the results of independent studies and is the technique normally used to combine the results of selected studies for systematic review. Validity of meta-analysis depends on the quality of the systematic review on which it is based. Like RCTs systematic reviews should not be considered to automatically be of good quality and should be critically appraised. Quality Education for a Healthier Scotland

Pharmacy Observational Studies Includes cohort studies, case-control studies and crosssectional studies These studies lack the controlled design of RCTs and the evidence which comes from this type of study is therefore considered less robust In some cases observational studies provide the only evidence available (e. g. emerging safety issues) and therefore it is also important that we are able to critically appraise these Quality Education for a Healthier Scotland

Pharmacy Over to you…… Now use what you have learned from this presentation and the tools available online to either critically appraise a randomised controlled trial and a cohort study of your choice or use the example on the NES website. Quality Education for a Healthier Scotland