Developing Practice Guidelines for the U S Preventive

Developing Practice Guidelines for the U. S. Preventive Services Task Force: A Hands-On Approach Kenneth Lin, MD Medical Officer, AHRQ STFM Annual Spring Conference May 2, 2009

Advancing Excellence in Health Care Learning Objectives n Define important attributes of good practice guidelines. n Review the guideline development process of the U. S. Preventive Services Task Force. n Using USPSTF methods and evidence from a systematic review, update a recommendation on screening for oral cancer.

When do you consult practice guidelines? How do you use practice guidelines? What are some attributes of good practice guidelines?

Advancing Excellence in Health Care Attributes of Good Practice Guidelines n Comprehensive, systematic evidence search n Evidence linked directly to recommendations via strength of recommendation grading system n Recommendations based on patient-oriented rather than disease-oriented outcomes n Development process is transparent n Potential conflicts of interest identified and addressed n Prospective validation n Clinical flexibility

Advancing Excellence in Health Care Attributes of Good Practice Guidelines n Comprehensive, systematic evidence search – Evidence-based vs. opinion-based – How to distinguish a systematic from a narrative review – How quickly reviews become out of date

Advancing Excellence in Health Care Attributes of Good Practice Guidelines n Evidence linked directly to recommendations via strength of recommendation grading system – Evidence-based is not always evidence-linked – SORT / USPSTF / GRADE n Recommendations based on patient-oriented rather than disease-oriented outcomes – DOEs versus POEMs

Advancing Excellence in Health Care n n n History of the USPSTF 1976 - Canadian Task Force on PHE 1984 - USPSTF established by USPHS 1996 - Community Task Force (CDC) 1998 - 3 rd USPSTF reconvened by AHRQ 2001 - Standing USPSTF Task Force

Advancing Excellence in Health Care n n n Composition of the USPSTF Government support but independent Experts in primary care, prevention, methods Family medicine, internal medicine, pediatrics, obstetrics/gynecology, nursing, behavioral medicine, preventive medicine Scientific support from a university Evidence. Based Practice center (currently OHSU) Liaisons from primary care subspecialty societies, Federal agencies

Advancing Excellence in Health Care Steps in the Recommendation Development Process Define questions and outcomes of interest Define and retrieve relevant evidence Evaluate QUALITY of individual studies Synthesize and judge STRENGTH of overall evidence 5. Determine balance of benefits and harms 6. Link recommendation to judgment about net benefits 1. 2. 3. 4.

Advancing Excellence in Health Care Step 1: Analytic Framework on Screening for a Condition

Advancing Excellence in Health Care Step 1: Analytic Framework for Prostate Cancer Screening 1 Treat Screen: Asymptomatic Men PSA, DRE 2 radiation, prostatectomy Early Prostate Cancer 3 Reduced prostate cancer morbidity, mortality 5 4 Adverse effects of screening: Adverse effects of Rx: false positive, false negative, inconvenience, Labeling, worry Impotence, incontinence, death, overtreatment

Advancing Excellence in Health Care Steps in the Recommendation Development Process Define questions and outcomes of interest Define and retrieve relevant evidence Evaluate QUALITY of individual studies Synthesize and judge STRENGTH of overall evidence 5. Determine balance of benefits and harms 6. Link recommendation to judgment about net benefits 1. 2. 3. 4.

Advancing Excellence in Health Care Steps in the Recommendation Development Process n Create inclusion/exclusion criteria based on the key questions from the analytic framework – Types of studies – Outcomes – Population – Setting – Time period n Sources of Evidence – Medline, Cochrane literature searches – Reference mining- key articles, editorials, review articles – Expert recommendations

Advancing Excellence in Health Care Steps in the Recommendation Development Process Define questions and outcomes of interest Define and retrieve relevant evidence Evaluate quality of individual studies Synthesize and judge STRENGTH of overall evidence 5. Determine balance of benefits and harms 6. Link recommendation to judgment about net benefits 1. 2. 3. 4.

Step 3: Evaluate Quality of Individual Studies Advancing Excellence in Health Care n Good: – – Evaluates relevant available screening tests Uses a credible reference standard Interprets reference standard independently of screening test Large sample size, ~ 100 broad spectrum patients n Fair: – – Evaluates relevant available screening tests Uses reasonable although not best standard; Interprets reference standard independent of screening test; Moderate sample size, ~ 50 -100 “medium” spectrum patients n Poor: Has fatal flaw such as: – – Uses inappropriate reference standard Screening test improperly administered Biased ascertainment of reference standard Very small sample size or very narrow selected spectrum of patients.

Advancing Excellence in Health Care Steps in the Recommendation Development Process Define questions and outcomes of interest Define and retrieve relevant evidence Evaluate quality of individual studies Synthesize and judge STRENGTH of overall evidence 5. Determine balance of benefits and harms 6. Link recommendation to judgment about net benefits 1. 2. 3. 4.

Advancing Excellence in Health Care Step 4: Synthesize and Judge Strength of Overall Evidence n Evidence reports – Evidence tables summarizing characteristics and quality of studies – Narrative discussing overall strength of evidence n Systematic reviews from others – Other AHRQ reports, Cochrane, etc. n Meta-analysis n Modeling – Projected outcomes tables, decision analysis

Advancing Excellence in Health Care n n n Critical Appraisal Questions Do the studies have the appropriate research design to answer the key question? To what extent are the existing studies high quality? To what extent are the results of the studies generalizable (or “applicable”) to the general US primary care population and situation? How many studies have been conducted that address the key question? How large are the studies? How consistent/coherent are the results of the studies? Are there additional factors that assist us in drawing conclusions about the certainty of the evidence? (e. g. , presence or absence of dose-response effects; fit within a biologic model)

Advancing Excellence in Health Care Step 4: Synthesize & Judge Strength of Evidence for Each Key Question Convincing: Well-designed, well-conducted studies in representative populations that directly assess effects on health outcomes Adequate: Evidence sufficient to determine effects on health outcomes, but limited by number, quality, or consistency of studies, generalizability to routine practice, or indirect nature of the evidence. Inadequate: Insufficient evidence to determine effect on health outcomes due to limited number or power of studies, important flaws in their design or conduct, gaps in the chain of evidence, or lack of information on important health outcomes

Advancing Excellence in Health Care Step 4: Synthesize and Judge Strength of Overall Evidence: Certainty n Definition: The U. S. Preventive Services Task Force defines certainty as “likelihood that the USPSTF assessment of the net benefit of a preventive service is correct”. n The net benefit is defined as benefit minus harm of the preventive service as implemented in a general, primary care population. The USPSTF assigns a certainty level based on the nature of the overall evidence available to assess the net benefit of a preventive service.

Advancing Excellence in Health Care Levels of Certainty: High, Moderate, or Low n High: The available evidence usually includes consistent results from well-designed, wellconducted studies in representative primary care populations. n Moderate: The available evidence is sufficient to determine the effects of the preventive service on health outcomes. As more information becomes available, the magnitude or direction of the observed effect could change, and this change may be large enough to alter the conclusion. n Low: The available evidence is insufficient to assess effects on health outcomes.

Advancing Excellence in Health Care n n n Critical Appraisal Questions Do the studies have the appropriate research design to answer the key questions? To what extent are the existing studies high quality? To what extent are the results of the studies generalizable (or “applicable”) to the general US primary care population and situation? How many studies have been conducted that address the key questions? How large are the studies? How consistent/coherent are the results of the studies? Are there additional factors that assist us in drawing conclusions about the certainty of the evidence? (e. g. , presence or absence of dose-response effects; fit within a biologic model)

Advancing Excellence in Health Care 1. 2. 3. Steps in the Recommendation Development Process Define questions and outcomes of interest Define and retrieve relevant evidence Evaluate quality of individual studies 4. Synthesize and judge STRENGTH of overall evidence 5. Determine balance of benefits and harms 6. Link recommendation to net benefits

Advancing Excellence in Health Care Step 5: Determine Balance of Benefits and Harms Estimate Magnitude of Net Benefits – Harms = Net Benefit

Advancing Excellence in Health Care Step 5: Determine Balance of Benefits and Harms: Assessing Harms n Potential harms real but hard to quantify n Include psychological and physical consequences of false-positives, “labeling, ” overtreatment n Opportunity costs – Time and effort required by patients and the health care system to provide the preventive service – Not the same as financial costs n Magnitude of harm subjective, hard to compare to benefits – NNH (eg, GI bleeds from aspirin)

Advancing Excellence in Health Care Step 5: Determine Balance of Benefits & Harms: Assessing Magnitude of Net Benefit n No explicit criteria for magnitude categories – Zero/negative, small, moderate, substantial n Substantial benefit : impact on condition with high population burden or major effect on uncommon outcome n Problems: requires evidence on harms and common metric for benefit and harms n Always requires judgment

Advancing Excellence in Health Care 1. 2. 3. Steps in the Recommendation Development Process Define questions and outcomes of interest Define and retrieve relevant evidence Evaluate quality of individual studies 4. Synthesize and judge STRENGTH of overall evidence 5. Determine balance of benefits and harms 6. Link recommendation to net benefits

Advancing Excellence in Health Care Step 6: Link recommendation to net benefits: USPSTF Grades of Recommendations Certainty of Net Benefit Magnitude of Net Benefit Substantial Moderate Small Zero/negative High A B C D Moderate B B C D Low Insufficient

Advancing Excellence in Health Care n n n Step 6: Link recommendation to net benefits: Insufficient Evidence Lack of studies on harms or benefits Poor quality of existing studies Good quality studies with conflicting results

Advancing Excellence in Health Care Grade Step 6: Link recommendation to net benefits: USPSTF Wording of Recommendations Grade Definition Suggestion for Practice A The USPSTF recommends the service. There is high certainty that the net benefit is substantial. Offer or provide this service. B The USPSTF recommends the service. There is high certainty that the net benefit is moderate or there is moderate certainty that the net benefit is moderate to substantial. Offer or provide this service. C The USPSTF recommends against routinely providing the service. There may be considerations that support providing the service in an individual patient. There is moderate or high certainty that the net benefit is small. Offer or provide this service only if there are other considerations that support offering or providing the service in an individual patient. D The USPSTF recommends against the service. There is moderate or high certainty that the service has no net benefit or that the harms outweigh the benefits. Discourage the use of this service. I The USPSTF concludes that the current evidence is insufficient to assess the balance of benefits and harms of the service. Evidence is lacking, of poor quality, or conflicting, and the balance of benefits and harms cannot be determined. Read “Clinical Considerations” section of USPSTF Recommendation Statement. If offered the service, patients should understand the uncertainty about the balance of benefits and harms.

Recommendation Exercise: Screening for Oral Cancer

Advancing Excellence in Health Care USPSTF Recommendation, 2004 n The USPSTF concludes that the evidence is insufficient to recommend for or against routinely screening adults for oral cancer. I recommendation

Advancing Excellence in Health Care USPSTF Rationale and Clinical Considerations, 2004 n The USPSTF found no new good quality evidence that screening for oral cancer leads to improved health outcomes for either highrisk adults (ie, those over the age of 50 who use tobacco) or for average-risk adults in the general population. n Clinicians should be alert to the possibility of oral cancer when treating patients who use tobacco or alcohol.

Advancing Excellence in Health Care Epidemiology n 2 percent of all cancer cases in U. S. n More than 7, 500 deaths attributed to oral cancer in U. S. in 2008 n Overall 5 -year survival rate of about 60% n 50 -60% have regional or distant metastases at time of diagnosis n 4 in 5 patients with oral cancer have used tobacco products

Potentially premalignant lesions Advancing Excellence in Health Care n Defined based on visual appearance n Leukoplakia – – – White plaque-like lesion Precursor for up to 85% of oral cancers Up to half regress spontaneously n Erythroplakia – Red patch of oral tissue not associated with inflammation – Rarer than leukoplakia, but much more likely to transform into cancer

Advancing Excellence in Health Care n Leukoplakia, erythroplakia, and oral cancer Sources: www. aafp. org, www. merck. com

Advancing Excellence in Health Care The Oral Cancer Examination n Includes inspection and palpation of all the extraoral and intraoral soft tissues n Extraoral: lips, salivary glands, submandibular and cervical lymph nodes n Intraoral: interior lips, buccal mucosa, tongue, floor of mouth, gingivae, tonsils, soft palate, and retromolar trigone (area behind the wisdom teeth) n In 1998, 13. 8% of U. S. adults age 40 and older reported having had an oral cancer examination within the past year (HP 2010 goal is 20%)

Advancing Excellence in Health Care n The Oral Cancer Examination Source: American Family Physician 2002; 65: 1379 -84

Advancing Excellence in Health Care Current clinical context for oral cancer screening n Since 2004, multiple “adjunctive diagnostic aids” have been developed and aggressively promoted for oral cancer screening by commercial interests n While organizations such as the American Dental Association have generally concluded that evidence for the usefulness of these aids is lacking, they have also reaffirmed the necessity of routine oral cancer screening with visual examination

Advancing Excellence in Health Care Critical Evidence Gaps identified from previous USPSTF reviews n No direct evidence linking screening for oral cancer to improved health outcomes n Limited information on the accuracy of the oral examination for detecting cancers or potentially pre-malignant lesions

Advancing Excellence in Health Care Analytic Framework KQ 1 - Does screening for oral cancer reduce morbidity or mortality? (KQ 1) Adults without oral cancer symptoms KQ 2 – What is the yield and accuracy of screening for oral cancer? Screening for oral cancer Identification of oral cancer or pre-malignant lesions Harms Improved morbidity or mortality Harms

Advancing Excellence in Health Care 2009 Update Critical Key Questions n KQ 1: Does screening for oral cancer reduce morbidity or mortality? n KQ 2: What is the yield and accuracy of the screening oral examination in identifying oral cancer or lesions that have a high likelihood of progression to oral cancer?

Advancing Excellence in Health Care Contextual Questions n What are the incidence and mortality rates from oral cancer in U. S. adults who are not at increased risk? n How much time does it take for a primary care clinician to perform a comprehensive screening oral examination? n What are the observed rates of progression from potentially premalignant oral lesions to oral cancers? Does HPV infection affect these rates? n Does treatment of potentially premalignant lesions reduce the risk of developing oral cancer?

Contextual Questions

Advancing Excellence in Health Care Oral cancer incidence and mortality rates in “average risk” U. S. adults n No evidence found

Advancing Excellence in Health Care How much time does an oral cancer examination take? n Expert opinion ranged from 1 -5 minutes – “Perform a death-defying act: the 90 -second oral cancer examination” – “Oral cancer screening: 5 minutes to save a life” n No objective studies have measured actual oral examination times for various health professionals

Advancing Excellence in Health Care Natural history of premalignant lesions. Does HPV matter? n Greater than half of all leukoplakias spontaneously regress without treatment n Spontaneous regression is more likely to occur in tobacco users who quit n Rate of progression from leukoplakia to oral cancer has been estimated at less than 1% per year in Western European populations n HPV types 16 and 18 are predominantly associated with cancers of the pharynx that are not detectable on visual examination

Advancing Excellence in Health Care How effective is treatment of premalignant lesions? n Treatments for leukoplakia include surgery, systemic and topical medications, and counseling for tobacco and alcohol cessation n 2006 Cochrane review identified no RCTs of surgery that included a placebo arm; no benefit of treatment seen with non-surgical interventions n Incidence of oral cancer after surgical excision of leukoplakia in retrospective studies is similar to that observed in some populations without treatment

Critical Key Questions

Advancing Excellence in Health Care Methods n Searched Pub. Med and the Cochrane Library from January 2001 – October 2008 (KQ 1) and 1966 – October 2008 (KQ 2) – KQ 1: RCTs, systematic reviews, and metaanalyses that reported morbidity or mortality outcomes in a screened group vs. usual care – KQ 2: Diagnostic accuracy studies comparing the result of a uniformly applied screening test for oral cancer (which could include diagnostic aids) to a reference standard

Advancing Excellence in Health Care Literature Search Results n Potentially relevant articles retrieved – KQ 1: 252 – KQ 2: 474 n 42 articles pulled for full-text review n 8 articles met inclusion criteria for KQ 1 or KQ 2

Advancing Excellence in Health Care Key Question 2 n What is the yield and accuracy of the screening oral examination in identifying oral cancer or lesions that have a high likelihood of progression to oral cancer?

Advancing Excellence in Health Care KQ 2: Results n One systematic review and 4 primary studies met all inclusion criteria n All of the primary studies were published in 1997 or earlier n Reported the accuracy of the oral examination for detecting potentially premalignant lesions, since detection of true oral cancers was uncommon (5 out of 310 positive findings from all 4 studies)

Advancing Excellence in Health Care KQ 2: Limitations n Not best reference standard n Possibility that reference standard may have been affected by patient’s knowledge of screening exam result n Community or dental clinic samples may not be representative of primary care population n Some detected lesions (e. g. lichen planus, submucous fibrosis) now believed to have no malignant potential

Advancing Excellence in Health Care KQ 2: Summary n There is reasonable agreement between screening oral examination findings of generalists and oral medicine specialists n Accuracy of visual examination for detecting asymptomatic oral cancer is not known n Similarly, meaningful false positive rates cannot be calculated from existing data

Advancing Excellence in Health Care Convincing, Adequate, or Inadequate Evidence? n What is the yield and accuracy of the screening oral examination in identifying oral cancer or lesions that have a high likelihood of progression to oral cancer?

Advancing Excellence in Health Care Key Question 1 n Does screening for oral cancer reduce morbidity or mortality?

Advancing Excellence in Health Care KQ 1: Results n 3 articles met inclusion criteria n 1 systematic review and 2 separate reports (interim and final) of a single study of screening for oral cancer in Kerala, India

Advancing Excellence in Health Care KQ 1: the Kerala trial - methods n Cluster RCT involving more than 190, 000 adults age 35 years and older n Screening group invited to 3 rounds of screening between 1996 and 2004 (30% of subjects participated in all 3 rounds) n Screeners were health workers with non-medical university degrees who received 3 months of training in interviewing, tobacco and alcohol use counseling, and oral cancer examination n Persons with positive screens were referred to a specialized clinic and re-examined by a dentist or oncologist; one pathologist evaluated all biopsy samples

Advancing Excellence in Health Care KQ 1: the Kerala trial - results n Primary outcome: death from oral cancer n Overall, no effect of screening on outcome – 77 OC deaths in screening group, 87 OC deaths in control group – RR 0. 79 – NS – (95% CI, 0. 51 -1. 22) n Benefit of screening seen in “users of tobacco, alcohol, or both” (mostly men) – 70 OC deaths in screening group, 85 OC deaths in control group – RR 0. 66 (95% CI, 0. 45 – 0. 95)

Advancing Excellence in Health Care Limitations of the Kerala trial n Small number of clusters relative to study population size and geographical proximity n Low participation (30%) in all 3 rounds of screening n Question about generalizability of “high risk” results to U. S. patients given very high rates of oral cancer seen in Kerala population

Advancing Excellence in Health Care KQ 1: Summary n One fair quality randomized controlled trial showed no difference in oral cancer mortality from screening a population with a very high incidence of oral cancer

Advancing Excellence in Health Care Convincing, Adequate, or Inadequate Evidence? n Does screening for oral cancer reduce morbidity or mortality?

Advancing Excellence in Health Care What Is the Net Benefit of Screening for Oral Cancer? Estimate Magnitude of Net Benefits of Screening – Harms of Screening = Net Benefit may be Zero/Negative, Small, Moderate, or Substantial.

Advancing Excellence in Health Care n n n High, Moderate or Low Certainty? Do the studies have the appropriate research design to answer the key questions? To what extent are the existing studies high quality? To what extent are the results of the studies generalizable (or “applicable”) to the general US primary care population and situation? How many studies have been conducted that address the key questions? How large are the studies? How consistent/coherent are the results of the studies? Are there additional factors that assist us in drawing conclusions about the certainty of the evidence? (e. g. , presence or absence of dose-response effects; fit within a biologic model)

Advancing Excellence in Health Care Step 6: Link recommendation to net benefits: USPSTF Grades of Recommendations Certainty of Net Benefit Magnitude of Net Benefit Substantial Moderate Small Zero/negative High A B C D Moderate B B C D Low Insufficient

Advancing Excellence in Health Care Vote on one of 3 letter grades n C – “The USPSTF recommends against routinely screening for oral cancer. There may be considerations that support screening in an individual patient. ” n D – “The USPSTF recommends against screening for oral cancer. ” n I – “The USPSTF concludes that the current evidence is insufficient to assess the balance of benefits and harms of screening for oral cancer. ”

Advancing Excellence in Health Care What did the USPSTF decide? n Stay tuned… n Updated recommendation to be released in late 2009 or early 2010 n Sign up for Task Force updates at www. preventiveservices. ahrq. gov

Advancing Excellence in Health Care Acknowledgements n Evidence review co-authors – Melissa Camp, MD, MPH n Johns Hopkins Bloomberg School of Public Health – Val W. Finnell, MD, MPH n Uniformed Services University of the Health Sciences – Lolita Ramsey, MSN, RN n George Mason University School of Nursing

Advancing Excellence in Health Care Acknowledgements n Expert Reviewers – Celeste Abraham n Baylor College of Dentistry, Texas A&M Health Science Center – Daniel Deschler n Department of Otology and Laryngology, Harvard Medical School – William Kohn n Division of Oral Health, NCCDPHP, CDC