The Scientific Method and Statistics Critical Appraisal Skills

The Scientific Method and Statistics Critical Appraisal Skills depend upon an understanding of the scientific method and the role statistics plays Al Best, Ph. D Perkinson 3100 B ALBest@VCU. edu V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Critical Appraisal Skills n n n V I Are the results of the study valid? What are the results? Will the results help locally? R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

My goals for you Be able to answer four questions: n Based on the study design, what is the level of evidence? n How were threats to validity addressed? n Based on the goals of the study, How do you describe the results? n To justify the conclusions, were comparisons done appropriately? V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Stats 2: Outline Science: Using data to answer questions n Case-control study (example) – Estimate prevalence in a population § Sampling, measurement, randomness § Estimates and Confidence Intervals – Compare cases and controls § Hypothesis testing § P-value, confidence interval – Interpret the results § Prevalence difference n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

A Scientist’s Quandary Are the results of the study valid? – Most experiments are highly local but have general aspirations. – How can findings generalize to other people, in other settings, with comparable interventions, and other outcomes. n How will you assess whether the paper’s findings will generalize to your situation? – This is the question of external validity. n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Solution? Use a process where sample data does generalize to the units, treatments, variables and settings not directly observed. n Follow the process called Statistical Inference using the Scientific Method. n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Statistical Inference V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Example: Barasch’s “Risk Factors for Osteonecrosis of the Jaws” “We conducted a case-control study in dental practices to determine the risk associated with bisphonates and to identify other risk factors for ONJ, …” n From the introduction of Barasch, et al. (2011) J Dent Res 90(4), 439444. pubmed/21317246 V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Classic Steps: Inference using Statistics What’s the question? (Introduction) – Conceptualize the population – State the question 3. Answer the question 2. How will you answer the question? (Methods) (Results) – The sample – The measurements – Analysis technique 4. What does it mean? (Discussion) 1. V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

“a case… study … of the risk associated with BP and ONJ” Conceptualize the population – BP or ONJ? n State the question: Which? – In ONJ cases, estimate the prevalence of BP use. Or – In BP cases, estimate the prevalence of ONJ. n Abbreviations: BP=bisphophonates, ONJ=osteonecrosis of the jaw V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

“a case… study … of the risk associated with BP and ONJ” Main question: Compare – the prevalence of BP use higher in ONJ cases to – the prevalence of BP use higher in controls. n Design a measurement system – ONJ = case or control – BP = use or non-use n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

“a case… study … of the risk associated with BP and ONJ” Design a measurement system – ONJ case n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Segue: Data is information in context. n That is, data is the set of information—usually numbers—arising out of measurements of individuals. n Part of the information in data is also its context— how did this information come about? n V I R G I N I A C O M M O N W E Tonight we’re going to let the statistics speak for themselves Ed Koren, © The New Yorker, 9 December 1974. A L T H U N I V E R S I T Y

Segue: Data classification Distinguishing Type of data Characteristics Categorical Observations or grouped into qualitative distinct classes Nominal Classes without a natural order or rank Ordinal Classes with a predetermined or natural order V I R G I N I A C O M M O N W E Examples Sex, treatment group, presence or absence Disease severity, bone density, plaque accumulation, bleeding A L T H U N I V E R S I T Y

Segue: Data classification Distinguishing Type of data Characteristics Examples Continuous or Observations may quantitative assume any value on a (numeric) continuous scale Interval Numeric value with equal unit differences Temperature, GPA, age, duration of disease, income amount, serum cholesterol, number of decayed teeth Synonyms: continuous, interval V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

“a case… study … of the risk associated with BP and ONJ” Design a measurement system – ONJ case n What type of data is this? – Nominal or Ordinal or Interval (continuous) ? n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

“a case… study … of the risk associated with BP and ONJ” Measure the subjects – BP use n Section H. Medications Now I would like to ask you about some of the medications that you have taken during your lifetime. This is the last section of the interview. It would be helpful to use the sheets that we sent you in our last letter. 1. Have you EVER taken any of the following drugs orally or BY MOUTH? V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Looking Ahead V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

“a case… study … of the risk associated with BP and ONJ” Measure the subjects – BP use n What type of data is this? – Nominal or Ordinal or Interval? n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Backing up: State the question In ONJ cases, estimate the prevalence of BP use. n The population has a parameter—call it π —we are trying to estimate this using data. n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

The population has a parameter— call it π Conceptualization: The population True: n. ONJ = true count of everyone who has ONJ True: n. BP = true count of everyone who has ONJ and also used BP π = true prevalence proportion of BP in ONJ patients. π = True: n. BP / True: n. ONJ n n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Estimation of population parameter using sample statistic Definition: A statistic is a single descriptive number computed from the data. Conceptualization: The sample n n. ONJ = count in sample who have ONJ n n. BP = count in sample who have ONJ and also used BP n p = estimated prevalence proportion of BP in ONJ patients. p = n. BP / n. ONJ n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Backing up: State the question In ONJ cases, estimate the prevalence of BP use. n π = True: n. BP / True: n. ONJ V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Classic Steps What’s the question? (Introduction) – Conceptualize the population – State the question n Answer the question n How will you answer the question? (Methods) (Results) – The sample – The measurements – Analysis technique n What does it mean? (Discussion) n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Actual n. ONJ? Actual n. BP? Analyze the data – BP use in ONJ patients – n. ONJ = count in sample who have ONJ – n. BP = count in sample who have ONJ and also used BP n p = estimated prevalence proportion of BP in ONJ patients. p = n. BP / n. ONJ n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Estimate of p? Analyze the data – BP use n – From sample n. ONJ=191, in these n. BP=113 p = 113/191 ? = 83% = proportion 0. 83 – Point estimate of p = 0. 83 V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Segue: estimation error In ONJ cases, estimate the prevalence of BP use. Variability due to sampling: 308 cases down to 191 n Variability due to measurement: n. BP = 113 n. No. BP = 24 Unknown = 24 . 825=113/137 V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Answer the question In ONJ cases, estimate the prevalence of BP use. n The population has a parameter—call it π—we are trying to estimate—using data. n In ONJ patients, 83% reported BP use V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Classic Steps What’s the question? (Introduction) – Conceptualize the population – State the question n Answer the question n How will you answer (Results) the question? (Methods) – The sample – The measurements – Analysis technique n What does it mean? (Discussion) n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Defining the research question: Testable consequence? Conceptual progression from general to specific n General question – Is Bisphonate use a risk factor for ONJ? n Specific hypothesis – Is the prevalence of bisphonate use higher in ONJ cases than in controls? n Testable consequence – Prediction of a relationship – Potentially refutable by data n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Defining the research question: Prediction: a statistical relationship between exposure and outcome n – BP prevalence will be higher in ONJ cases than in controls How do we arrive at this? Using a refutable hypothesis n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Defining the research question: Formalization n. A refutable hypothesis Statistical formalization: n Ho: proportion BP (ONJ) = proportion BP (controls) – Which may be disproved beyond a reasonable doubt through falsification by data via statistical hypothesis testing, in favor of: n Ha: proportion BP (ONJ) > proportion BP (controls) n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Critical Appraisal Is it a testable, research question? How did they try to rule out bias, confounding, chance? How did they consider multiple outcome measures and multiple predictors? Did they disclose what was done with enough detail so others may replicate? n n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

How Science Advances Clinical Knowledge Science forms a question and brings data to bear to answer the question. n Informally: 1. Frame a clinical research question. 2. State its testable consequences as either “just random variability” or “unusual outcomes”. 3. Compare the actual data with these two choices and decide which to believe. 4. Discuss our present understanding. n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Testing Hypotheses Or, linking the four steps to the standard IMRD organization of a paper: 1. What’s the question? (Introduction) 2. How do you answer the question? (Methods) 3. Answer the question. (Results) 4. What does it mean? (Discussion) n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

What’s the Question? “We conducted a case-control study in dental practices to determine the risk associated with bisphonates and to identify other risk factors for ONJ, …” n From the introduction of Barasch, et al. (2011) J Dent Res 90(4), 439 -444. V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

How do you answer the question? Outline – Propose two states of nature – Use the rule of simplicity – Take into account that “noise happens” – Use a test statistic to decide: Signal or Noise? n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Two states; Two hypotheses We begin by conceiving the true state of nature as being either: – no difference or – a difference. n We always start by assuming that nothing is going on—that any apparent differences are purely because of chance. Our preference, as scientists, is to believe the simplest explanation for a phenomenon. – Assume: no difference (AKA null hypothesis). n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Rule of Simplicity When you have two competing theories which make exactly the same predictions, the one that is simpler is the better. n The simplest explanation for some phenomenon is more likely to be accurate than more complicated explanations. n The explanation requiring the fewest assumptions is most likely to be correct. n AKA “Occam’s Razor” V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Statistical World View Thou shalt not interpret randomness. Chance happens. Noise exists. Making an interpretation that goes beyond this requires justification. If random noise, measurement error, or chance occurrence can account for variations (differences) in the observations, then there is no need to formulate a more complicated explanation. – We embody this preference in the statement of the null hypothesis. n n n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Null Hypothesis We evaluate this proposition using statistical techniques. The null hypothesis is the statement that is tested. It’s abbreviated H 0: A null-hypothesis is the simplest explanation of events: There is no difference. There is no change. There is no improvement. Nothing unusual is occurring. A null-hypothesis is the statement we hope to contradict with data. That is, we usually hope to reject the null hypothesis. n n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Assume: Nothing is going on V n Prevalence of bisphonate use within those who do have ONJ is equal to the prevalence of bisphonate use within those who do not have ONJ. n HO: πcases = πcontrols I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Two Hypotheses Prevalence of bisphonate use within those who have ONJ is equal to the prevalence of bisphonate use within those who do not have ONJ. n HO: Pcases = Pcontrols n Can we reject the above, in favor of: n Prevalence of bisphonate use within those who have ONJ is different than the prevalence of bisphonate use within those who do not have ONJ. n HA: Pcases ≠ Pcontrols ü So: Done with step 1: We’ve stated the question. Next: How will you answer the question? V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Proof By Contradiction Nature is either in one state or the other. – We prefer to believe the simplest explanation. n Collect data from the real world. n Assess the likelihood of observing this data under the null hypothesis. n Choose to believe: – If the data is within what we would expect then we retain our preference for the null-hypothesis as the best explanation. – If the data is very unlikely, then we reject the null hypothesis in favor of its alternative. n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

What if: HO is true? In ONJ cases, estimate the prevalence of BP use. n In controls, estimate the prevalence of BP use. n π = πONJ= πcontrol V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

What if? Assess the likelihood of observing various data possibilities under the null hypothesis. n Assume this is true: – HO: Pcases = Pcontrols n Then the sample estimate of Pcases will be “close to” the sample estimate of Pcontrols. – By “close to” we mean that, because of sampling variability and measurement error we expect them to be somewhat different. n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Contingency Table “Results: … Therefore, 137 191 cases together with 473 573 controls were included in the analyses…. Bisphonate use was reported by 113 cases (83%) and 71 controls (15%), …” Note: 113+71=184. V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Contingency Table Assume this is true: – HO: Pcases = Pcontrols = 0. 302 n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Contingency Table Assume this is true: – HO: Pcases = Pcontrols = 0. 302 – 30. 2% of 137= 41 n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Contingency Table Assume this is true: – HO: Pcases = Pcontrols = 0. 302 – 30. 2% of 473 = 143 n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Contingency Table Assume this is true: – HO: Pcases = Pcontrols = 0. 302 – 137 – 41 = 96 – 473 – 143 = 330 n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Test Statistic Test stat: Difference in prevalence – HO: Pcases – Pcontrols = 0. 0 – Expected difference = 0% n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Testing Hypotheses Recall: – Frame a clinical research question. – State its testable consequences as either “just random variability” or “unusual outcomes”. – Compare the actual data with these two choices and decide which to believe. – Discuss our present understanding. n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

I presume the null-hypothesis is true, do the data support this? Observed difference = 0% n P-value = 1. 0 n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

I presume the null-hypothesis is true, do the data support this? Observed difference = 0. 6% n P-value = 0. 885 n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

I presume the null-hypothesis is true, do the data support this? Observed difference = 1. 6% n P-value = 0. 724 n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

I presume the null-hypothesis is true, do the data support this? Observed difference = 90% n P-value < 0. 001 n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Trade offs Conclusion Do not reject null. Reject nullhypothesis (p-value >. 05) (p-value <. 05) Truth Null-hypothesis (no difference) correct Type I error Alternative hypothesis (difference) Type II error correct Alpha = Type I error = prob. of rejecting a true null hypothesis Beta = Type II error = prob. of not finding a true difference V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Significance Level The significance level is represented by the Greek symbol “alpha”, α. n It is the probability of rejecting a true null hypothesis. n The researcher chooses the risk of making this error: concluding that the null hypothesis is false when it really is true. – The most typical values are α =. 05, . 01, or. 10. n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Universal Decision Rule Choose to believe: n HO: null-hypothesis (For non-extreme values of the test statistic) – Choose this if p-value ≥ α (usually 0. 05) n HA: alternative-hypothesis (For extreme values of the test statistic) – Choose this if p-value < α (usually 0. 05) n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

! Are we done yet? ! What’s the question? (Introduction) – Conceptualize the population – State the question n Answer the question n How will you answer (Results) the question? (Methods) – The sample – The measurements – Analysis technique n What does it mean? (Discussion) n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

I presume the null-hypothesis is true, do the data support this? Observed difference = 67. 5% n P-value <. 0001 n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Chi-square test V n Expected data n Observed data (chi-square, P-value <. 0001) I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

State a Conclusion Prevalence of bisphonate use within those who have ONJ is equal to the prevalence of bisphonate use within those who do not have ONJ. – HO: Pcases = Pcontrols n Prevalence of bisphonate use within those who have ONJ is different than the prevalence of bisphonate use within those who do not have ONJ. – HA: Pcases ≠ Pcontrols n Evidence: 82. 5% prevalence vs 15%, p-value <. 0001 n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

P-value The p-value is the probability that the data occurred by chance, assuming the null hypothesis is true. n The p-value is NOT the probability that the nullhypothesis is true. n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

The p-value is NOT the probability that the null -hypothesis is true. and: 1−pvalue is NOT the probability that the alternative hypothesis is true V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Trade offs Conclusion Do not reject null. Reject nullhypothesis (p-value >. 05) (p-value <. 05) Truth Null-hypothesis (no difference) correct Type I error Alternative hypothesis (difference) Type II error correct Alpha = Type I error = prob. of rejecting a true null hypothesis Beta = Type II error = prob. of not finding a true difference V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Actuality Results Do not reject null. Reject nullhypothesis (p-value >. 05) (p-value <. 05) Truth Null-hypothesis (no difference) Blind alley ? Alternative hypothesis (difference) ? Discovery! Alpha = Type I error = prob. of rejecting a true null hypothesis Beta = Type II error = prob. of not finding a true difference V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

P-value A modest reality: The p-value is simply the probability that the data occurred by chance. n n Big leap: A significant p-value is a license to make up a story. V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Results The Prevalence of bisphonate use within those who have ONJ is different than the prevalence of bisphonate use within those who do not have ONJ (p-value <. 0001). n Discussion “In conclusion, this case-control study supports a causal link between bisphonates and ONJ”* n *See page 443, V I R G I N I A C the last Discussion paragraph. O M M O N W E A L T H U N I V E R S I T Y

Review To assess validity – In the study, What’s the question? – Where did the data come from? Sampling and measurement. – What would you expect if “nothing is going on”? – Is the observed data different than that? n But other factors could account for the observed difference – Bias, confounding, multiplicity n V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

Next Time Be able to answer four questions: n Based on the study design, what is the level of evidence? n How were threats to validity addressed? n Based on the goals of the study, How do you describe the results? n To justify the conclusions, were comparisons done appropriately? V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y