Drug Screening and the False Discovery Rate Charles

Drug Screening and the False Discovery Rate Charles W Dunnett Mc. Master University 3 rd International Conference on Multiple Comparisons, Bethesda, Maryland, August 5– 7, 2002 1

Objectives 1. Review some early statistical work on drug screening. 2. Discuss applicability of the FDR to drug screening problems. 2

Background • The FDR was introduced by Benjamini and Hochberg (1995) as a new error rate to replace the FWER in some multiple testing problems. • It provides weak control of the FWER instead of strong control as required by the FWER method (i. e. , FWER ≤ α only for the case where all null hypotheses are true). • Drug screening is one of the applications where they recommended that an FDR controlling procedure be used. 3

Background (continued) • In the mid-1950’s, statisticians at the National Cancer Institute in Bethesda, and also at many pharmaceutical companies, became very active in screening problems. • In analyzing drug screening experiments, they adopted methods which tested each hypothesis separately at a level α, thus providing control of the per comparison error rate (PCER). 4

Typical drug screening experiment: • • Test compounds (20 - 30) Inactive control group (placebo) Known active compound (if available) Compare each test compound with the control and compute a statistic t. Declare a test compound “positive” if t exceeds a critical value c. 5

False negatives and false positives when screening m compounds Compound Active Inactive Totals Error rates: Screening result + T S U V m-R R Totals m – m 0 m FDR = E(V / R) PCER = E(V / m) 6

Special features of drug screening experiments i. Screening is an ongoing program to test large numbers of chemical compounds, most of which do not have the desired activity. ii. Multiple treatments are included in the same experiment for reasons of economy only. iii. The treatments tested in the same experiment are noncompeting. iv. Included in the design of an efficient screening procedure is the need to decide how many compounds to test with the resources available. 7

A Typical Screening Program • 28 separate screening tests in operation. • Screening rates: varied from 500 to 6, 000 compounds per year. • Compound file: 65, 000 compounds. • Rate of increase in file: 6, 000 compounds per year. 8

Designing Drug Screening Tests Based on the OC and ASN Functions • Define θ = activity level of a compound. • OC(θ) = Pr(declare compound positive | θ); this is the Operating Characteristic function. • Choose θ 0 (poor activity) and θ 1 (good activity). • Determine sample sizes and critical value(s) to achieve OC(θ 0) = α and OC(θ 1) = 1 – β, for suitably chosen values of α and β. • ASN(θ) = E(total sample size per compound | θ) using a k-stage screening procedure (e. g. , k = 3); this is the Average Sample Number function. 9

Example of a 3 -stage anticancer screen (Vogel and Haynes, 1963) Let C = mean control tumor weight (n 0 = 6) T = mean tumor weight for a compound (n = 3) At each stage, calculate X = T/C for each compound Assumption: X = log (T/C) distributed as N(θ, σ), σ known. E(T/C) =. 70 for a poor compound, . 20 for a good compound Choose α =. 003, 1 – β =. 976 Stage 1 2 3 T/C X 1 X 2 X 3 Cumulative X 1 X 2 X 3 Accept <. 269 <. 101 <. 052 Reject ≥. 521 ≥. 194 ≥. 052 10

Designing Drug Screening Tests to Maximize Efficiency • Define ‘interesting’ activity: θ ≥ θ 1. • Assume a particular distribution of θ in the population of compounds. • Choose sample sizes and critical constants to maximize a chosen measure of efficiency. • Davies (1958): For a given degree of concentration of positives, maximize number of compounds with θ≥θ 1 found for a given amount of screening effort. • King (1963): Similar to preceding, but emphasis on maximizing the proportion of true positives included. This is equivalent to minimizing the proportion of false positives (i. e. , the FDR). 11

In Drug Screening, should we control the FDR or the PCER? • Since screening experiments are set up for reasons of economy only and the treatments are noncompetitive, is there any need to adjust for multiplicity? • Control of the FDR might work if it were applied over the entire screening program, but this is not feasible as immediate decisions are required as to which compounds should go forward to the next stage of development. • Controlling the PCER has worked well in the past; it appears to be a reasonable approach. … continued on next slide 12

. . . continued • Controlling the FDR at the experiment level leads to the use of stepwise test procedures. Although this increases power, there is a drawback: experiments which contain a higher number of actives make it easier for poor compounds to pass as a result of the decrease in the critical values (analogous to the cheating problem noted by Troendle 2000 and by Finner and Roters 2001). • This seems to be undesirable: whether a compound is declared to be positive should not depend on how well the other compounds in the same experiment perform. • To put it in another way, a screening procedure should provide a level playing field for all the candidates, and stepwise testing does not do this. 13

continued • My conclusion: Use a PCER controlling procedure in drug screening problems. • To include consideration of the FDR, a PCER controlling procedure like that of King (1963), which maximizes the proportion of true positives (or minimizes the FDR), could be designed. 14

Some References on Drug Screening • O. L. Davies (1958). The design of screening tests in the pharmaceutical industry. Bull. Intern. Stat. Inst. , 36: 226 -241. • P. Armitage and M. Schneiderman (1958). Statistical problems in a mass screening program. Ann. N. Y. Acad. Sci. , 76: 896 -908. • A. Vogel and J. Haynes (1962). Experiences with sequential screening for anticancer agents. Cancer Chemotherapy Reports, 22: 23 -30. • E. P. King (1963). A statistical design for drug screening. Biometrics, 19: 429 -440. 15