Epidemiology Module 3 Systematic and Random Error Biases

Epidemiology Module 3: Systematic and Random Error (Biases and Statistical Precision) Tuhina Neogi, MD, Ph. D, FRCPC Steven Vlad, MD Clinical Epidemiology Research and Training Unit Boston University School of Medicine

Goals • Understand the difference between systematic error and random error • Review types of systematic error (bias) – confounding, information bias (measurement error), selection bias • Review random error (statistical precision) – correct interpretations of confidence intervals and p-values – Type 1 and Type 2 errors

Bias 3

How to interpret results of a study • The result of a study (the stated effect measure) can arise because: – It is the truth – It is due to bias (systematic error) – It is due to chance (random error)

Bias/Systematic Error • Once we know the study result, we need to focus on whether the result is the truth, or whether it could have been the result of bias – If a study’s result is RR=1. 2, could the true (unbiased) result be higher or lower than that value? • How could possible biases influence the reported result?

What are biases (systematic errors)? • Biases can either move the observed result (as opposed to the true result) away from the null, or toward the null – Null = 0 for difference measures (e. g. risk difference) – Null = 1 for ratio measures (e. g. risk ratio) – Away = • if true result > null, effect appears larger • if true result < null, effect appears smaller – Toward = • if true result > null, effect appears smaller • if true result < null, effect appears larger – Bias has nothing to do with sample size • effect of study design

Bias Away from Null 0 1 Observed Value A - ∞ 0 True Value A Null Observed Value A 0 + ∞ True Value B 1 0 - ∞ Observed Value B + ∞ Observed Value B Bias Toward Null + ∞

Direction of Bias • Often, the result of bias is unpredictable – Either toward or away from null • In some circumstances we can predict the effect a bias – Usually toward the null 8

Types of Bias • Only 3 types of bias! – all studies – regardless of study design • The different study designs have ways of dealing with these biases – whether it’s done effectively determines how valid the study is

The 3 different types of biases • Confounding • Information Bias (Measurement Error) • Selection Bias

Confounding 11

Confounding • Question: – Are there other factors that may account for the apparent relationship between exposure (intervention) and disease (outcome)? – or – Is there a common cause of both exposure and disease?

Two ways to look at Confounding Confounder Exposure O O Disease Confounder Exposure Disease

Confounding • Note – Confounder not caused by exposure or disease – Confounder not on causal path from exposure to disease • i. e. not an intermediate Confounder Exposure Disease

Examples Smoking Yellow Finger Lung Cancer Diabetes CRP CAD Depression SSRI Suicide 15

Why Does Confounding Occur? • As an imbalance in proportions of the confounder between the two comparison groups Total Population No. of Cases N Risk Ratio Exposed Unexposed 4, 600 140 1, 000, 000 0. 0046 0. 00014 0. 0046 / 0. 00014 = 33 16

Total Population Exposed Unexposed Cases N 4, 600 140 1, 000, 000 Risk 0. 0046 RR Cases N Risk RR 0. 00014 33 Men Women Exposed Unexposed 4, 500 50 900, 000 100, 000 N 0. 005 0. 0005 Rate 10 Cases RR 100 90 100, 000 900, 000 0. 001 0. 0001 10

Total Population Exposed Unexposed Cases N 4, 600 140 1, 000, 000 Risk 0. 0046 RR Cases 33 Men Women Exposed Unexposed 4, 140 14 N 900, 000 100, 000 Risk 0. 0046 0. 00014 RR 0. 00014 33 Cases 460 126 N 100, 000 900, 000 Risk 0. 0046 0. 00014 RR 33

Cases N Risk Men Women Exposed Unexposed 2, 500 250 500, 000 N 0. 005 0. 0005 Risk RR Cases 10 1, 000 100 500, 000 0. 001 0. 0001 RR 10 Total Population Exposed Unexposed Cases N Risk RR 3, 500 350 1, 000, 000 0. 0035 0. 00035 10

Why Does Confounding Occur? Confounder Exposure Cases N Risk RR Disease Men Women Exposed Unexposed 4, 500 50 900, 000 100, 000 N 0. 005 0. 0005 Rate 10 Cases RR 100 90 100, 000 900, 000 0. 001 0. 0001 10

Why Does Confounding Occur? Smoking Yellow Fingers Lung Cancer Smokers Non-smokers Yellow Not Yellow 4, 500 900, 000 100, 000 Risk 0. 005 RR 1 Crude RR = (4, 600/1 M) / (1, 400/1 M) = 3. 3 Cases N Rate Yellow Not Yellow 100 900 100, 000 900, 000 0. 001

Confounding cont’d • Does hyperlipidemia cause MIs? – Do higher lipid levels cause MIs? BP, age, gender, BMI, smoking, DM High Lipids ? MI

– Does lowering lipid levels lower the risk of MI? • Does statin use lower lipids? – If so, does that have an effect on lowering MI events? BP, age, gender, BMI, smoking, DM, other healthy lifestyle factors, adherence statins levels, MI ? lower lipid

– Does a diet high in cholesterol increase the risk of MI? • Does a ‘heart healthy diet’ lower lipids? – If so, does that have an effect on lowering MI events? BP, age, gender, BMI, smoking, DM, other healthy lifestyle factors, diet, exercise, adherence Healthy heart diet ? lower lipid levels, MI

Confounding by Indication • Particularly common and difficult to deal with in (observational) pharmacoepidemiology studies RA disease severity Lymphoma TNF-antagonist use Depression SSRI Suicide

What Confounding is NOT • Confounding IS NOT – A factor on the causal pathway (intermediate) • high fat diet LDL CAD • smoking adenomatous polyp colon CA – A factor that modifies the relationship between an exposure and a disease • Effect of anti-HTN drug is different in Blacks vs Whites • Effect of blood levels of X on risk of Y in men vs women

Effect Modification (aka Interaction) Total Population High BP Normal BP MI 4, 700 140 N 1, 000, 000 0. 0047 0. 00014 Risk RR 33. 6 White Americans High BP Normal BP MI 4, 500 50 N 900, 000 0. 005 Risk RR 10 Black Americans High BP Normal BP MI 200 90 100, 000 N 100, 000 900, 000 0. 0005 Rate 0. 002 0. 0001 RR 20

How does a RCT address Confounding? • Randomization – evenly distribute both known and unknown confounders (by chance) between the two (or more) exposure groups (i. e. active treatment and placebo) – Can use specific inclusion/exclusion criteria to ensure everyone is the same for a particular confounder (e. g. , all males in the study) [also known as restriction]

Control of Confounding by Randomization – Since potential confounders are balanced between exposure groups, they can’t confound the association Depression SSRI placebo 250 N 500, 000 Risk 0. 0005 Death RR 1 1, 000 get SSRI 1, 000 get placebo No Depression SSRI placebo 100 N 500, 000 Risk 0. 0002 Death RR Depression SSRI 1 Depression distributed equally by chance, so: Crude RR = (350/1 M) / (350/1) = 1 Suicide

Control of Confounding in an RCT 1. Check Table 1 for imbalances between treatment arms – – – Are any of the differences clinically meaningful? (not the same as statistically significant!) How could those differences affect the results? What other potential confounders are missing from Table 1? Could they affect the results if they were imbalanced between the treatment arms? How likely is it that unknown confounders are imbalanced? (with large RCTs, unlikely)

Control of Confounding in a RCT 2. Intention-to-treat analysis – Maintains the balance of potential confounders given by randomization, thereby continuing to (theoretically) address confounding – May need to ‘control’ for very unbalanced factors in the analyses

Control of Confounding in Observational Studies • Study design level: Restriction – Inclusion/exclusion criteria to limit study population to a more homogeneous group • E. g. study of alcohol and MI may exclude smokers since smoking is an important confounder • Data analysis level: Stratification – Analyze data stratified by an important confounder • E. g. evaluate effect of alcohol on MI among smokers and among non-smokers separately • Added advantage: identifies effect modification • Data analysis level: Regression – ‘Control’ for potential confounders in regression models • can also identify effect modification if planned for by investigators • Matching – Match on potential confounding variables

Control of Confounding in Observational Studies 1. Have the investigators identified all the important potential confounders in the study? – What factors could be common causes of both exposure and disease? 2. Have the investigators accounted for these potential confounders either in the design of the study or in the analysis? 3. If they haven’t, how might that affect the 33 results they found?

Confounding: Examples 34

• In large RCTs, confounding is not usually an important issue – Randomization distributes confounders equally between trial arms • Table 1 appears to confirm this – Probable confounders seem to be well balanced 35

Is this difference important? Previous fractures are a risk for future fractures. . . 36

• Questions to ask: 1. What are the potential sources of confounding? 2. Have the authors identified these potential confounders 3. How have the authors addressed potential confounders? 4. Was this sufficient? 37

1. What are the potential sources of confounding? • Risk factors for exposure – i. e. for NSAID use • • confounder RA? (chronic NSAIDs) NSAID CV event CAD prevention? (ASA) prior GI bleeding? (avoid NSAIDs, use coxib) • Are any of these also related to age? (avoid NSAIDs) or type of NSAID the likelihood of having an event? 38

1. What are the potential sources of confounding? • Risk factors for outcome? – i. e. having a CV event • • • confounder smoking? prior CV event? general health? ASA use? (preventative) • age? NSAID or type of NSAID CV event Are any of these also related to the likelihood of being exposed to an NSAID or type of NSAID? 39

1. What are the potential sources of confounding? • Confounding by indication – Almost always a concern in observational studies of drugs – Here, could the reason that NSAIDs are prescribed (the ‘indication’) be related to both exposure (NSAID prescription) and disease (CV outcomes)? – What about reasons to avoid NSAIDs? • Chronic kidney disease? • Other health problems? CKD pain NSAID or type of NSAID CV event NSAID or type of NSAID Poor health CV event NSAID or type of NSAID CV event 40

2. Have the authors identified these potential confounders? • In this study, seems like yes – they even have a section talking about ‘covariates’ 41

3. How have the authors addressed potential confounders? • Statistical analysis, • ‘Advanced methods’ 42

4. Was this sufficient? • Often the hardest of these questions to answer, especially if the authors have done a good job addressing the first three questions – Requires experience, judgement, maybe further analysis – My take on this study: probably sufficient • They’ve acknowledged the difficulties and done something about them; • What they’ve done is appropriate; • The methods are reasonable and go beyond what most studies do; • However, I recognize what an insidious problem confounding, especially confounding by indication, is, and therefore I’m ready to change my opinion if further evidence comes to light 43

• Questions to ask: 1. What are the potential sources of confounding? 2. Have the authors identified these potential confounders? 3. How have the authors addressed potential confounders? 4. Was this sufficient? 44

1. What are the potential sources of confounding? • This is another pharmacoepidemiologic study (kind of), so let’s jump right to confounding by indication again – Here, could the reason that DMARDs were prescribed (the ‘indication’) be related to both exposure (DMARD use) and disease (lymphoma)? • -> Disease severity, i. e. chronic inflammation disease severity DMARD or type of DMARD lymphoma 45

2. Have the authors identified these potential confounders? – Clearly - that is what this paper is all about 46

3. How have the authors addressed potential confounders? • Here: – detailed assessment of disease severity for each subject – mutual control of drug use and disease severity etc. 47

4. Was this sufficient? • Again, this is often the hardest of these questions to answer, – My take on this study: probably sufficient • They’ve acknowledged the difficulties and done something about them; • What they’ve done is appropriate; • The methods are reasonable; • Again though, I recognize what an insidious problem confounding, especially confounding by indication, is, and therefore I’m ready to change my opinion if further evidence comes to light 48

Information Bias (Misclassification) 49

Information Bias • Could exposure (intervention) or disease (outcome) be inaccurately reported or measured – i. e. misclassified?

Information Bias • Information about exposure, disease, and/or confounder(s) is erroneous – Misclassified into the wrong category True exposure Measured exposure True disease status Measured disease status

Examples of Information Bias True Et. OH Exposure misclassification Reported Et. OH Fetal outcome True Gastritis status Outcome misclassification Drug True illicit drug use Gastritis by chart review True blackouts BOTH exposure and outcome misclassification Reported illicit drug use If using the same source for exposure and disease information (e. g. , from the participant), can get very biased results Reported blackouts

Non-differential Misclassification • Chance of misclassifying is the same in – both the exposure groups (or more if more than one) • and the chance of misclassifying is the same in – both the outcome groups (ditto) • i. e. misclassification is random in both exposure and disease • General Rule – If misclassification is non-differential, then the resultant bias is usually toward the null • i. e. the measured effect is probably conservative • Warning: this is true in general. It is not always true in every study! (There can be bias away from the null)

Differential Misclassification • Chance of misclassifying is not the same in either – the two exposure groups (or more if more than one) • or – the two outcome groups (ditto) • i. e. misclassification is not random in either exposure or disease • In this case, the resultant bias is unpredictable! – Consider carefully whether you think any misclassification is likely to be differential or nondifferential!

How does an RCT address Information Bias • EXPOSURE: – Exposure is assigned (randomly) in an RCT • the label of the treatment arm is a proxy for the actual exposure • bias is usually non-differential Assigned exposure Actual exposure Outcome

How does an RCT address Information Bias • EXPOSURE: – Problems: non-adherence and contamination • With less than 100% adherence, intention-to-treat analysis is biased due to information bias (usually non-differential) • If using a ‘completers’ analysis (only analyze those completing the study in their assigned group), information bias is addressed, but • introduces confounding and selection bias (when subjects leave the study) – randomization has been broken – reasons for withdrawal could be related to treatment » TRUTH is somewhere between ITT and completers analysis when <100% adherence 56 (almost always)

Treatment A Treatment B • • ITT - thought to be more conservative estimate • (biased toward null) Completers analysis - thought to usually exaggerate estimate 57

Information Bias in an RCT • OUTCOME • allocation concealment (nobody knows what the next treatment assignment will be) • blinding (investigators and subjects) • same study procedures for all arms – these should prevent non-differential misclassification of the outcome • any bias should be toward the null – Problems: • Unblinding (e. g. side effects from treatment) • Accuracy of outcome assessment – reliability, reproducibility

Information Bias in Observational Studies • Obtain information about exposure and outcome in a structured manner (preferably ‘objective’ sources) – Obtain information about the disease in exactly the same fashion regardless of the exposure status – Obtain information about the exposure in exactly the same fashion regardless of the disease status – Use different sources of information for exposure and disease if possible • If you ask someone about their alcohol use and also ask them about how many times they fell, you are likely to get very skewed data – Et. OH users less likely to report both use and falls

Information Bias: Examples 60

61

Exposure misclassification? • An RCT, so exposure should be random • Pill counts and interviews to assess whether subjects took medications as assigned • Not an ITT (but not a completers analysis either) – (the method they used is probably better than an ITT analysis - you’ll just have to trust me) 62

Outcome misclassification? • Allocation concealment – Not commented upon • unfortunately this is common • we’ll presume allocation was concealed • Blinding – Stated to be double-blind • placebos for both interventions (blinds subjects in absence of noticeable side effects) • BMD scan results withheld from local investigators (blinds investigators) • Outcome assessments – BMD - ‘Quality assurance, cross-calibration adjustment, and data processing were done centrally’ • presumably without knowledge of treatment assignment – Fracture - ‘Radiographs were assessed in a blinded fashion by an independent reader’ • presumably using a standard protocol of some kind

Likelihood of Information Bias • Appears to be minimal • Any bias should be toward the null • Estimates therefore likely to be conservative (underestimate true results) 64

65

Exposure misclassification? 1. How was exposure assessed? – prescriptions for NSAIDs (including coxibs) 2. Could exposure be mis-measured? – here, a prescription does not guarantee that the subject actually took the medication, or that s/he took it as prescribed – OTC medication use may not be measured NSAID use NSAID Prescription CV event

Exposure misclassification? 3. Is any exposure mis-measurement likely to be nondifferential or differential? – Three questions here: 1. Coxibs vs traditional NSAIDs 2. NSAIDs vs comparison meds (glaucoma/thryoid meds) 3. Are chronic NSAID users likely to differ in their ‘compliance’ compared to glaucoma/thyroid med users (or coxib users)? 1. Some traditional NSAIDs are available OTC • coxibs are not • therefore, more likely that NSAID use is mis-measured (underestimated) • therefore, differential misclassification is likely 2. Same potential problem with NSAIDS in general (some available OTC) and comparison meds (prescription

Exposure misclassification? 3. Is any exposure mis-measurement likely to be non-differential or differential? 3. Are chronic NSAID users likely to differ in their ‘compliance’ compared to glaucoma/thyroid med users (or coxib users)? • I’m not sure but I suspect there might be a • difference. Any opinions? 68

Outcome misclassification? 1. How was outcome assessed? – CV outcomes based on various codes in admin records 2. Could exposure be mis-measured? – errors made by coders (doctors, administrators, etc. ) – no verification that a coded event is an actual event Codes for CV events NSAID Prescription CV event

Outcome misclassification? 3. Is any outcome mis-measurement likely to be non-differential or differential • Is it more likely for persons not coded with a CV event to actually have a CV event? • or for persons coded with a CV event to not have one? • Or does each seem equally likely? • To me, I think it’s more likely that persons coded for a CV event were mis-coded compared to persons who never received a code in the first place.

Likelihood of Information Bias • Overall, low-moderate • Likelihood of differential misclassification is moderate • I recognize this risk, think the authors are also aware of it and have considered it, and conditionally accept the results of the study pending further studies

72

Exposure misclassification? 1. How was exposure assessed? • • Exposure here is degree of inflammation Apparently, standardized, protocolized assessment of tender/swollen joints 2. Could exposure be mis-measured? – Physician error in counting joints Tender/swollen joint count Inflammation Lymphoma

Exposure misclassification? 3. Is any exposure mis-measurement likely to be non-differential or differential? • Not 100% sure, but seems to me like it’s likely to be non-differential • i. e. all patients have RA • More likely to count more joints if subject seems • more ‘active’? Less likely to count more joints if subject seems to be doing subjectively well?

Outcome misclassification? 1. How was outcome assessed? – Cases - occurrence of lymphoma – All cases validated and confirmed – Controls - random other RA subjects w/o RA 2. Could exposure be mis-measured? – Would have to argue that some of those without lymphoma actually had missed cases - seems unlikely Lymphoma on biopsy Inflammation Lymphoma

Outcome misclassification? 3. Is any outcome mis-measurement likely to be non-differential or differential • As said, risk seems small but any risk would have to be non-differential • Cases are almost certainly lymphoma • Could be some controls who actually should be cases

Likelihood of Information Bias • Overall, low • Likelihood of differential misclassification is also low • Effort taken to verify cases and relatively rarity of lymphoma in those who are controls virtually eliminates the risk of information bias in this study.

Selection Bias 78

Selection Bias • Is study entry or exit related to exposure (intervention) or disease (outcome)?

Selection Bias • Factors that influence study participation – Subjects entering and staying in the study • exposure-disease association is different among those who participate than among those who do not – E. g. healthy worker effect

Example of Selection Bias Exposure New Drug DM Chemical Lipids Participation Disease Lung Pain MI Gall bladder disease Those who Healthy Hospitalized Health conscious or familial completed worker patient hypercholesterolemia or other high study (compared to risk general population)

How does an RCT address Selection Bias? • Study entry: Randomization after study entry prevents any factors related to the exposure from influencing study participation

How does a RCT address Selection Bias? • Study exit: Try to achieve full follow-up – Problem: Loss-to-follow-up (LTFU) • LTFU is almost always related to either the exposure or disease or both (e. g. drug is ineffective (outcome), or drug has side effects (exposure)) • Equal numbers of LTFU in treatment arms does not guarantee that there is no selection bias (different reasons for dropout) • Analytic techniques such as last observation carried forward or multiple imputations do not “take care of” selection bias

Selection Bias in Observational Studies • Difficult to deal with selection bias in observational studies – Can’t always control factors influencing study participation • Healthy worker effect: use a group in the same office/factory not exposed to the particular chemical – Try to minimize loss-to-follow-up • Use appropriate analytic methods to account for differing lengths of follow-up (incidence ratio, hazard ratio)

‘Representativeness’: This is NOT a bias • Trying to make one’s study population ‘representative’ of a more general population is counter to using restriction for control of confounding – Basic science animal studies ‘restrict’ study population to genetically homogeneous lab animals • To elaborate a scientific theory about a biologic process • If a different group is of clinical interest, should conduct well-designed study in that group to determine whether effects differ in that group – This is a question of whether the BIOLOGY is different in that group • This is an issue of “Effect Measure Modification”

Selection Bias: Examples 86

87

A Clinical Trial • Selection bias due to study entry is virtually never an issue – true here – patients randomized after selection for study • Loss to follow-up – 70 (33%!) in alendronate group – 64 (30%!) in teriparatide group – These are fairly high rates of LTFU! 88

Does it matter? • LTFU is roughly equal in each arm • This does NOT guarantee lack of selection bias • Question: – Is LTFU random in each group? – or is LTFU differential? • e. g: • alendronate subjects drop out because of side effects – (maybe the same people where the drug was effective) • teriparatide subjects drop out because of inefficacy – This combination could make teriparatide look better than it actually is, even though LFTU is equal in each group • teriparatide users who had an effect are left in study • alendronate users who had no effect are left in the study

Judgement • Seems to me that drop out is likely to be random • above scenario is unlikely – Therefore I doubt there is much if any selection bias in this study 90

91

Study entry • The authors recognize that healthy people are less likely to have records in their study database – one reason they used other drug-users (glaucoma and hypothyroid) as a control group • Therefore healthy persons may be less likely to be selected as a comparison group subject – disease estimate in control group could be exaggerated 92 – results in underestimate of true risk

Study exit • Subjects could leave study if they got insurance, e. g. – are persons who did this more likely to be NSAID users? or to have an event? or both? – are healthier persons more likely to get insurance and not have an event? • I would think so • leaves less healthy persons who could have an event in the control group – more events in control group – underestimate true risk 93

Judgement • Selection bias is certainly possible – Recognized by authors – Methods to minimize (drug using control group) • Overall, risk is there but is probably not very high(? ) 94

95

Study entry • Entry in this study is based on case-control status • How could cases or controls be unrepresentative of RA patients? • All RA patients should have been captured in this RA registry • All lymphoma patients should also be captured in the lymphoma registry • One reason why ‘population-based’ studies are 96 nice!

Study exit • As with entry, these registers should capture all RA patients who have had lymphoma • If RA patients left the study before they got lymphoma that could be a source of bias • But these persons could not have been cases or controls. • Again, likelihood of selection bias seems low 97

Judgement • Selection bias seems unlikely in this nested case-control study • Note, this can often be the case in c-c studies where the underlying population is well-defined (esp. population-based studies) • When this is NOT the case (e. g. hospitalbased c-c study) selection bias is much 98 more likely

Interpreting Study Results • Any study’s results must be interpreted in the context of whether or not there any biases that could have deviated those results from being the truth – This is the critical step in understanding whether or not these results are clinically meaningful for your patients – i. e. , are the results valid? • If there are biases, you need to try to determine how much they would have affected the results – would the message still be about the same even taking those biases into account?

Summary • You should now be able to: – Understand that all study designs use various strategies to address systematic error (bias) to varying degrees of success – Understand how to identify confounding, information bias, selection bias

Random Error 101

Goals • Review random error (statistical precision) – Review the correct interpretations of confidence intervals and p-values – Review Type 1 and Type 2 errors

Measured value Random Error vs Systematic Error True value In absence of bias, with perfect precision Study size Syst error Random error

Random Error • Results from variability in the data, sampling – E. g. measuring height with measuring tape: 1 measurement may be off, but multiple measurements will give you a better estimate of height • Relates to precision • We use confidence intervals to express the degree of uncertainty/random error associated with a point estimate (e. g. a RR or OR) – Measure of precision

Confidence Interval • Epidemiologist/biostatisticians have arbitrarily chosen 95% as the level of confidence to report • 95% C. I. =if the data collection and analysis were repeated infinitely, the confidence interval should include within it the correct value of the point estimate 95% of the time – Note: this is NOT the same as saying the confidence interval is 95% likely to contain the ‘true’ value. (INCORRECT!!) – ASSUMES NO BIAS

90% confidence intervals from repeated studies True value for RR

Confidence Interval cont’d • If the confidence interval includes the null (‘ 1’ for relative measures of effect (e. g. RR, OR), ‘ 0’ for absolute measures of effect (e. g. risk difference)), the result is considered to be not statistically significant – Since the CI contains the correct value 95% of the time, the correct value could be the null

Confidence Interval cont’d • “Precision” relates to the width of the confidence interval – A very precise estimate of effect will have a narrow CI – Can improve precision by: • Increasing sample size

P-value • The P-value is calculated from the same equations that lead to a confidence interval • Again, 0. 05 (related to the 95% CI) was arbitrarily chosen • P-values are confounded by magnitude of effect and sample size (precision) – Can get a very small p-value if a study is large enough, even if the effect is very small

P-value • P-value=assuming the null hypothesis is true, the p -value is the probability of this result or one more extreme, assuming no bias/confounding – E. g. RR=2, p=0. 03: “Assuming the null hypothesis is true (RR=1), the probability of this result (i. e. RR=2) or one more extreme (i. e. RR>2) is 3%” – E. g. RR=0. 4, p<0. 001: “Assuming that there truly is no difference between the two treatment groups, the probability of obtaining a RR of 0. 4 or one more extreme (i. e. RR<0. 4) is less than 0. 1%”

• Note: this is NOT the same as saying the pvalue is the probability that the null hypothesis is true (INCORRECT!! since it is conditioned on the null being true) – Just because your p-value isn’t “statistically significant” doesn’t mean that you can say the two arms are “equivalent” • The p-value says nothing about the probability of your alternative hypothesis being true • Think of the p-value as a relative marker of how consistent the data are with the null hypothesis

Statistical Significance Testing • Statistical significance testing is really only about statistical precision – How precise are the study results? – P-values on their own tell you NOTHING about the magnitude of effect; 95% CI confidence intervals at least give you an idea of potential magnitude of effect – P-values and 95% CI tell you NOTHING about how valid the results are (i. e. , whether the results are biased) • We should place LESS emphasis on statistical significance and rather think MORE about clinical significance or meaningfulness when considering the results and validity of a study

Type I, Type II Errors • Emanating from α=0. 05, Type I error is made if one incorrectly rejects the null hypothesis when the null hypothesis is actually true – This probability is determined by the significance level alpha (=0. 05) • Type II error is made if one fails to reject the null hypothesis when the null is false – This probability is determined by β, where 1 -β is the power, which is usually set at 80%

Basic Approach to Understanding Power • Power calculations/sample size calculations: – Use estimate of the expected event rate in the unexposed (placebo group) • Based on published literature/experience (e. g. , 10% of men age 60 -70 have MIs) – Based on that ‘background rate’, determine sample size needed to detect a difference of a specified magnitude between treatment and placebo with 80% power and alpha=0. 05

Random Error: Examples 115

116

Statistical Significance: p-value given, but no confidence interval regarding the difference in means (i. e. the 95% CI for the mean BMD difference of 3. 8%) Can theoretically use the standard error (S. E. ) information to calculate the 95% confidence interval Adequate power? Sample size: 214/arm Had sufficient power to detect 2% difference in BMD at L-spine (found 3. 8% difference)

118

Statistical Significance • 95% confidence interval given for each rate ratio • Adequate sample sizes? – N=several thousand persons for each drug – 100 s-1000 s of events – Null results (lack of association) for some drugs is unlikely due to inadequate power (and did find association for Vioxx, as expected) – ? Type 2 error

120

Statistical Significance 95% Confidence Intervals given for each Odds Ratio Adequate sample size? 378 cases, 378 controls They found an association, so no concerns about inadequate power due to insufficient sample size ? Type 1 error ? Bias

Summary • You should now be able to: – Understand that confidence intervals reflect precision, which is related to sample size – Understand the correct interpretation of confidence intervals and p-values – Understand that power relates to sample size