Classification and Bias of Clinical Research Rick Chappell

Classification and Bias of Clinical Research Rick Chappell, Ph. D. Professor, Department of Biostatistics and Medical Informatics University of Wisconsin Medical School

Good Ethics is Good Science: “If a research study is so methodologically flawed that little or no reliable information will result, it is unethical to put subjects at risk or even to inconvenience them through participation in such a study. … Clearly, if it is not good science, it is not ethical. ” - U. S. Dept. of Health and Human Services, Policy for Protection of Human Subjects (45 CFR 46, 1/1/92 ed. )

Types of Studies Classified by Temporal Point of View l I. Instantaneous Studies - Surveys l II. Longitudinal Studies n A. Retrospective Studies u. Historical Observational Cohort u. Case - Control n B. Prospective Studies u. Prospective Observational Cohort u. Clinical Trial n C. Hybrid Designs

A Schematic for Temporal Classification Retrospective Observational Cohort Prospective Observational Cohort Randomization Case - Control Clinical Trial Now Instantaneous: Survey

I. Instantaneous: Population-Based Studies l Synonyms n Survey n Population-Correlation Study n Ecological Study l Two or more populations are instantaneously compared through the prevalences of both exposure and disease. l As summarized units get smaller (country region neighborhood individual), a survey approaches a historical observational cohort study.

Population-Based Studies Advantages Disadvantages l Instantaneous. l Intervention is usually not feasible. l Easy access to a large and varied population. l Very little information on causality: IARC standards require individual-based evidence. l Good for hypothesis generation.

II. Longitudinal: Individual-Based Studies l A longitudinal study observes exposures and events for individuals over a period of time. l There are two types, depending on whether one is looking forwards (prospective) or backwards (retrospective) from the present.

Longitudinal Studies: A. Retrospective l Historical Observational Cohort n Synonyms - survey, retrospective cohort study. n Examines outcomes among patients with past exposures. n E. g. , track down 1950 s asbestos miners & determine current status. l Case - Control (Breslow and Day, 1980) n Synonyms - case referent, retrospective study. n Examines past exposures among a group of patients with current outcomes. n E. g. , interview mesothelioma patients & determine past exposures.

Historical Observational Cohort Studies Advantages Disadvantages l Quick results - no wait. l l Easy to get large samples by ‘mining’ databases. No opportunity to customize data collection. l No possibility for blinding. l Many possible biases: n Confounding n Selection n Information l Yields wide range of sequelae. l Useful for investigating rare treatments or exposures.

Case - Control Studies Disadvantages Advantages l Cheap, quick - record searching can be automated. l Useful for pilot studies. l Useful for investigating rare disorders. l Gives narrow picture of risks due to treatment or exposure. l Biases: n Confounding n Selection n Recall l Yields only estimates of relative, not absolute risk.

Hypothetical Historical Cohort Study l Exposed Group n 100 Patients n 10 Events n Rate =. 1 Odds Ratio 2 l Control Group n 100 Patients n 5 Events n Rate =. 05

Hypothetical Case-Control Study l Event Group n 100 Patients n 10 Exposures Event Rate per Exposure = ? (Not 100/200). l Non-Event (Control) Group n 100 Patients n 5 Exposures Odds Ratio 2

Longitudinal Studies: B. Prospective l General Advantages n Can collect detailed exposure, treatment, disease, and demographic information. n Blinding is possible. n Recall and information bias may be eliminated. n Useful for investigating rare treatments or exposures. l Classification depends on the presence of intervention.

Prospective Studies l Prospective Observational Cohort n Synonyms - prospective trial, ‘clinical trial’. n No intervention. l Randomized Controlled Clinical Trial n Synonyms - prospective interventional cohort study, experiment, prospective trial, clinical trial. n Experimenters directly intervene in patient treatment, usually on a randomized basis with controls.

Prospective Observational Cohort Study Additional Disadvantages Advantage l Passive observation; no need to dictate treatment. l May take a long time to accrue cases and wait for results. l Potential confounding bias due to lack of randomization and suitable controls.

Clinical Trials Additional Advantages Disadvantages “The most definitive tool for evaluation of the applicability of clinical research” - 1979 NIH release. l As above, may take a long time. l Must be ethically and laboriously conducted. l Biases may be eliminated. l l Good design may make analysis simple. Requires treatment on basis (in part) of scientific rather than medical factors. Patients may make some sacrifice (Meier, 1982). l

Phases of a Clinical Trial l Biochemical and pharmacological research. l Animal Studies (Gart, 1986 & Schneiderman, 1967). l Phase I (Storer, 1989) - estimate toxicity rates using few (~ 10 - 40) healthy or sick subjects. l Phase II (Thall & Simon, 1995) - determines whether a therapy has potential using a few very sick patients.

Phases of a Clinical Trial (cont. ) l Phase III - large randomized controlled, possibly blinded, experiments l Phase IV - a controlled trial of an approved treatment with long-term followup of safety and efficacy.

Longitudinal Studies: C. Hybrid Designs l Prospective Treatment, Historical Controls n Currently treated series of patients is compared with a previous series. n See Gehan & Freireich (1974), Gehan (1984). n Advantages u. Doesn’t assign treatments. u. No need to recruit controls.

Longitudinal Studies: C. Hybrid Designs (cont. ) l Prospective Treatment, Historical Controls n Disadvantages u. Same as in Historical Observational Cohort except that characteristics of treated patients (only) can be collected. u. Selection bias likely because of time lag between groups.

Hybrid Designs l Prospective Treatment with Both Prospective and Historical Controls n Uses both types of controls to maximize efficiency and minimize bias n See Pocock (1976 a and 1976 b).

Bias in Clinical Studies l Definition: Bias is a systematic error in estimation which is not reduced by increasing the study sample size (as opposed to random variation). l See Sacket (1979) and other articles in the same issue; Rose (1982); and Lachin (1988). l Classification is based on whether bias occurs at the time of patient Selection; or at the time of Information collection; or at the time of Publication. l They are all variants of Confounding, in which a third variable is related to both treatment and outcome.

I. Selection Bias l Prevalence - Incidence Bias n Prevalence (observed occurrence) of a trait Incidence (rate of onset). n Cause: gap between exposure, selection of subjects. n Not a problem with irreversible events such as mortality, if detectable. n E. g. , hypertension may disappear with onset of CV disease and can be overlooked as a risk factor. n See Neyman, 1955. n (Any retrospective study, especially case-control. )

Selection Bias l Admission Rate Bias n Patients may differ from noninstitutionalized subjects in size or direction of effects. n E. g. , systemic weakness vs. arthritis: u. Negative relation among inpatients; u. Positive relation among outpatients. n See Berkson, 1946. n (Any nonrandomized study with a mix of patient sources, especially case-control. )

Selection Bias l Nonrespondant (Volunteer) Bias n Nonparticipation may be related to the subject of investigation. n E. g. , smokers ignore surveys more often than do nonsmokers (Seltzer, 1974). n For general methods to analyze data with ‘nonignorable nonresponse’ see Little and Rubin (1987). n (Case-control, though drop-outs can effect any study not analyzed ‘intent to treat. )

Example: Where to add armor to fighter planes? In World War II, the U. S. Air Force conducted an investigation into where armor could most effectively be added to fighter planes. Researchers examined returning aircraft, mapped the locations of bullet holes, and recommended that the most commonly pierced areas be reinforced. Their recommendation neglected the most vital part of the aircraft, which was intact in all returning aircraft: the area surrounding the pilot’s head!

II. Information Bias l Detection Signal (Diagnostic Suspicion) Bias n In unblinded studies, an exposure may be considered a risk factor for an endpoint, and such patients preferentially observed. n In blinded studies, an exposure may make an endpoint more detectable. n E. g. , estrogen causes bleeding from uterine cancer to be more easily detectable. n (Any unblinded study except case-control; also clinical trials with sensitive endpoints. )

Reports of Original Studies JAVMA 191, 12/1/87 “High-rise syndrome in cats” Wayne O. Whitney, DVM & Cheryl J. Mehlhaff, DVM Selection and/or detection bias

Information Bias l Exposure Suspicion Bias n An outcome may cause the investigator to look for a particular exposure. n The temporal reverse of detection signal bias. n E. g. , arthritis and knuckle-cracking. n (Case-control studies. )

Information Bias l Recall (family information) Bias n Similar to exposure suspicion bias, but errors originate with the subject or his/her family. n E. g. , in a study of prescription use among women with fetal malformation, 28% reported unverifiable exposure vs. 20% of the controls (Klemetti & Saxen, 1967). n (Case-control studies. )

III. Publication (Reporting) Bias l Even a perfect study leads to bias if dissemination depends on the direction of its result. l Causes: n Commercial reasons; n Researchers’ personal motivations; n Editorial Policy ! l Vickers, et al. (1998) show that the problem is widespread: in some countries, 100% of publications show treatment effects.

Publication (Reporting) Bias l A version of the multiple comparisons problem (Miller, 1985), or ‘testing to a foregone conclusion’. l E. g. , ORG-2766 protected nerves from cytotoxic injury in 55 women with ovarian cancer - NEJM lead article (van der Hoop, et al. , 1990); a subsequent negative study of 133 women - ASCO Proceedings abstract (Neijt, et al. , 1994). l (All Studies. )

A type of reporting bias: Multiple Comparisons (“Data Dredging”) l A “p-value” is interpreted as the probability of attaining a result as extreme that observed given that the result is false (under the null hypothesis); it can be viewed as the false positive rate under the null hypothesis. l This assumes that only a single test is conducted. If many tests are performed, it is possible to “sample to a foregone conclusion” and produce a falsely low p-value. l For example, if twenty-five independent tests are conducted, the probability of at least one p-value being less than. 01 is. 22. l Often only the significant result is reported, and the 24 others ignored.

IV. Confounding (General) l Caused by any situation in which: n A third variable exists which isn’t known or at least isn’t accounted for; n It is associated with the “cause” and n It is also associated with the “effect”. Then: l The supposed cause-effect relation will be confounded by the third variable. l (Any nonrandomized study)

Do Storks Bring Babies?

Population of Oldenburg, Germany, 1930 -1936 (Ornithologische Monatsberichte 44, Jahrgang, 1936, Berlin) Humans (1000 s) Storks (1000 s)

References Berkson, J. Limitations of the application of fourfold table analysis to hospiital data (1946). Biometrics Bulletin 2, 47 -53. Breslow, N. E. and Day, N. E. (1980). Statistical Methods in Cancer Research 1: The Analysis of Case-Control Ctudies. Oxford: Oxford University Press. Dorr, Robert T. (1997). Personal communication. Gart, J. J. et al. (1986). Statistical Methods in Cancer Research 3: The Design and Analysis of Long-Term Animal Experiments. Oxford: Oxford University Press. Gehan, Edmund A. The evaluation of therapies: Historical control studies, with discussion (1984). Statistics in Medicine 3, 315 -324. Gehan, Edmund A. and Freireich, Emil (1974). The New England Journal of Medicine, 198 -203. IARC. Monographs on the Evaluation of Carcinogenic Risk of Chemicals to Humans. Lyon: IARC. Klemetti, A. and Saxen, L. Prospective vs. retrospective approach in the search for environmental causes of malformations. American Journal of Public Health 57, 2071 -2075. Lachin, J. Statistical properties of randomization in clinical trials (1988). Controlled Clinical Trials 9, 289 -311. Little, R. J. A. and Rubin, D. B. (1987). Statistical analysis with Missing Data. New York: Wiley. Neyman, J. Statistics - servant of all sciences (1955). Science 122, 401. Meier, Paul. Current research in statistical methodology for clinical trials (1982). Proceedings of Current Topics in Biostatistics and Epidemiology: A Memorial Symposium in Honor of Jerome Cornfield. Pages 141 - 150. Biometrics.

Miller, R. Publication bias (1985). Entry in The Encyclopedia of Statistical Sciences, Volume 5. S. Kotz and N. L. Johnson, eds. , pp. 679 -689. New York: Wiley. National Institutes of Health, Division of Research Grants, Research Analysis and Evaluation Branch, Bethesda, MD (1979). NIH inventory of clinical trials: fiscal year 1979, Volume I. Neijt, et al. (1994). Proceedings of the American Society for Clinical Oncology. Pocock, S. J. The combination of randomized and historical controls in clinical trials (1976 a). Journal of Chronic Diseases 29, 175188. Pocock, S. J. Randomized versus historical controls: A compromise solution (1976 b). Proceedings of the International Biometric Conference 9/1, 245 -260. Rose, G. Bias (1982). British Journal of Clinical Pharmacology 13, 157 -162. Rubin, D. B. (1987). Multiple Imputation for Nonresponse in Surveys. New York: Wiley. Sacket, D. L. Bias in analytic research (1979). Journal of Chronic Diseases 32, 51 -63. Schneiderman, M. A. Mouse to man: statistical problems in bringing a drug to clinical trial (1967). Proceedings of the 5 th Berkeley Symposium in Mathematical Statistics and Probability, Volume IV. L. M. Le. Cam and J. Neyman, eds. Berkeley. Seltzer, C. C. et al. Mail response by smoking status (1974). American Journal of Epidemiology 100, 453 -477. Storer, B. E. Design and analysis of phase I clinical trials (1989). Biometrics 46, 33 -38. Thall, Peter F. and Simon, Richard M. Recent developments in the design of phase II clinical trials (1995). In Recent Advances in Clinical Trial Design and Analysis. Peter Thall, ed. , pp. 49 -72. New York: Kluwer. Unger, D. L. Does knuckle cracking lead to arthritis of the fingers? [letter]. Arthritis & Rheumatism 41. 949 -50, 1998. van der Hoop, et al. (1990). New England Journal of Medicine 322, 89 -84. Vickers, et al. (1998). Controlled Clinical Trials 19, 159 -166.