EXERCISES IN SENSITIVITY SPECIFICITY PPV AND NPV There

EXERCISES IN SENSITIVITY, SPECIFICITY, PPV, AND NPV There are eight questions on Slides 4, 6, 7, 8, 22, 26, 42, and 54. Answer on Word and place in Dropbox by the end of the week. The rest of the material is meant to guide you. 1

Remember If you were asked; how would you estimate: ü Se: get a group of dzed animals and test them ü Sp: get a group of non-dzed animals and test them ü PPV: get a group of test positive animals and determine how many are dzed ü NPV: get a group of test negative animals and determine how many are non-dzed Remember: dzed means the animal has the dz, or infection or condition of interest (which could be anything). 2

Remember ü Se = # Test + / # dzed ü Sp = # Test - / # non-dzed ü PPV = #dzed / # Test + ü NPV = # healthy / # Test – Info about dzed Info about non-dzed Info about Test+ Info about Test- Remember: dzed means the animal has the dz, or infection or condition of interest (which could be anything). 3

Which type of test produces a lot of false positives? – Q 1 1. 2. 3. 4. High sensitivity High specificity Low sensitivity Low specificity 4

Relationship Between Se and Sp ü Sensitivity ONLY deals with diseased animals. ü Se tells you nothing about Sp, because Se and Sp are independent. ü Also, Sp tells you nothing about Se and only deals with non-diseased animals. 5

Which of these correctly describes a VERY SENSITIVE test? – Q 2 1. Many true positive test results 2. Many false positive test results 3. Few true positive test results 4. Few false positive test results 6

Which type of test produces a lot of false negative test results? – Q 3 1. High sensitivity 2. High specificity 3. Low sensitivity 4. Low specificity 7

Which of these correctly describes a VERY SPECIFIC test? – Q 4 1. Many true positive test results 2. Many true negative test results 3. Few true positive test results 4. Few true negative test results 8

Measuring Sensitivity Produces Two Results: 1. True Positive Results 2. False Negative Results Sensitivity is measured by testing dzed animals High Sensitivity test produces: – Many true positive test results – Few false negative test results Low Sensitivity test produces: – Few true positive test results – Many false negative test results 9

Measuring Specificity Produces Two Results 1. True Negative Results 2. False Positive Results Specificity is measured by testing non-dzed animals High Specificity test produces: – Many true negatives test results – Few false positives test results Low Specificity test produces: – Few true negatives test results – Many false positives test results 10

Two by two tables……MEMORIZE ü Memorize and learn how to use them!!!!! ü Use them to calculate sensitivity and specificity, PPV and NPV. ü In an exam, draw the table and fill in the squares. ü Using a two by two table will prevent you from making mistakes, especially with PPV and NPV. 11

This is a Two by Two Table Learn it, and use it for Se, Sp, PPV, and NPV. Diseased = Healthy = actually has the doesn’t have condition the condition Tests positive True Positive (TP) False Positive (FP) Tests negative False Negative (FN) True Negative (TN) Sensitivity = TPTP + FN × 100% PPV = TPTP + FP × 100% Specificity = TNTN + FP × 100% NPV = TNTN + FN × 100% 12

The columns are the definition of Se and Sp Diseased = Healthy = actually has the doesn’t have condition the condition Tests positive True Positive (TP) False Positive (FP) Tests negative False Negative (FN) True Negative (TN) Se Sp Se = TPTP + FN Sp = TNTN + FP 13

The rows are the definition of PPV and NPV Diseased = Healthy = actually has the doesn’t have condition the condition Tests positive True Positive (TP) False Positive PPV (FP) Tests negative False Negative (FN) True Negative NPV (TN) NPV = TNTN + FN PPV = TPTP + FP 14

Tests for Liver Flukes in Cattle ü Necropsy = Gold Standard for Liver Flukes – 10, 000 cattle tested with necropsy, 20% (2, 000) are positive (truly infected with liver flukes) 1. Coproscopy – 1, 400 of 2, 000 dzed cattle test positive with coproscopy • 600 false negatives – all 8, 000 healthy cattle test negative with coproscopy • 0 false positives – What is the Se and Sp of coproscopy? Example from: Rapsch, et al. 2006. Int J Parasitol. 36: 1153 -1158 15

1, 400 of 2, 000 dzed test positive: 600 false negatives All 8, 000 healthy cattle test negative: 0 false positives Tests positive Tests negative Diseased = actually has the condition True Positives (TP) Healthy = doesn’t have the condition False Positive (FP) Total False Negatives True Negative (FN) (TN) Total Sensitivity = TPTP + FN × 100% Specificity = TNTN + FP × 100% 10, 000 Step One Put in the total number (N) in the corner. The total column must add up to 10, 000. The total row must add up to 10, 000. 16

1, 400 of 2, 000 dzed test positive: 600 false negatives All 8, 000 healthy cattle test negative: 0 false positives Tests positive Tests negative Total Diseased = actually has the condition 1, 400 (TP) Healthy = doesn’t have the condition False Positive (FP) 600 (FN) True Negative (TN) 2, 000 Sensitivity = TPTP + FN × 100% Specificity = TNTN + FP × 100% Total 10, 000 Step Two Fill in the total number of diseased. Then, fill in the diseased column (true positives and false negatives). Make sure they add up to 2, 000. 17

1, 400 of 2, 000 dzed test positive: 600 false negatives All 8, 000 healthy cattle test negative: 0 false positives Tests positive Tests negative Total Diseased = Healthy = actually has the doesn’t have condition the condition 1, 400 0 (TP) (FP) 600 (FN) 8, 000 (TN) 2, 000 8, 000 Sensitivity = TPTP + FN × 100% Specificity = TNTN + FP × 100% Total 10, 000 Step Three Fill in the total number of healthy animals. Make sure the totals add up to N (10, 000). Then, fill in the healthy column (true negatives and false positives). Make sure they add up to 8, 000. 18

1, 400 of 2, 000 dzed test positive: 600 false negatives All 8, 000 healthy cattle test negative: 0 false positives Tests positive Tests negative Total Diseased = Healthy = actually has the doesn’t have condition the condition 1, 400 0 (TP) (FP) Total 1, 400 600 (FN) 8, 000 (TN) 8, 600 2, 000 8, 000 10, 000 Sensitivity = TPTP + FN × 100% Specificity = TNTN + FP × 100% Step Four Sum the test positives and test negatives and add the totals, making sure the totals sum to N (10, 000). 19

1, 400 of 2, 000 dzed test positive: 600 false negatives All 8, 000 healthy cattle test negative: 0 false positives Tests positive Tests negative Total Diseased = Healthy = actually has the doesn’t have condition the condition 1, 400 0 (TP) (FP) Total 1, 400 600 (FN) 8, 000 (TN) 8, 600 2, 000 8, 000 10, 000 Sensitivity = TPTP + FN × 100% Step Five Fill in the Se and Sp equations from the squares in the two by two table. Specificity = TNTN + FP × 100% 20

1, 400 of 2, 000 dzed test positive: 600 false negatives All 8, 000 healthy cattle test negative: 0 false positives Tests positive Tests negative Total Diseased = Healthy = actually has the doesn’t have condition the condition 1, 400 0 (TP) (FP) Total 1, 400 600 (FN) 8, 000 (TN) 8, 600 2, 000 8, 000 10, 000 1400 Sensitivity = TPTP × 100% = 1400 + 600 × 100% = + FN Specificity = TNTN + FP × 100% = 8000 + 0 × 100% = 70% 100% 21

Tests for Liver Flukes in Cattle – Q 5 ü Necropsy = Gold Standard for Liver Flukes – 10, 000 cattle tested with necropsy, 20% (2, 000) are positive (truly infected with liver flukes) 2. Serology (ELISA) – 1, 880 of 2, 000 dzed cattle test positive with the ELISA • 120 false negatives – 7, 680 healthy cattle test negative with the ELISA • 320 false positives ü Sp of the ELISA? – Calculate this showing the table Example from: Rapsch, et al. 2006. Int J Parasitol. 36: 1153 -1158 22

Tests for Liver Flukes in Cattle ü Necropsy = Gold Standard for Liver Flukes – 10, 000 cattle tested with necropsy, 20% (2, 000) are positive (truly infected with liver flukes) 1. Coproscopy – 1, 400 of 2, 000 dzed cattle test positive with coproscopy • 600 false negatives – all 8, 000 healthy cattle test negative with coproscopy • 0 false positives – What is the PPV and NPV of coproscopy? Example from: Rapsch, et al. 2006. Int J Parasitol. 36: 1153 -1158 23

1, 400 of 2, 000 dzed test positive: 600 false negatives All 8, 000 healthy cattle test negative: 0 false positives Tests positive Tests negative Total Diseased = Healthy = actually has the doesn’t have condition the condition 1, 400 0 (TP) (FP) Total 1, 400 600 (FN) 8, 000 (TN) 8, 600 2, 000 8, 000 10, 000 PPV = TP TP + FP × 100% Steps One to Four Are already completed for the Se, Sp calculation NPV = TN TN + FN × 100% Step Five Fill in the PPV and NPV equations from the squares in the two by two table. 24

1, 400 of 2, 000 dzed test positive: 600 false negatives All 8, 000 healthy cattle test negative: 0 false positives Tests positive Tests negative Total Diseased = Healthy = actually has the doesn’t have condition the condition 1, 400 0 (TP) (FP) Total 1, 400 600 (FN) 8, 000 (TN) 8, 600 2, 000 8, 000 10, 000 PPV = TP TP + FP × 100% = NPV = TN TN + FN × 100% 1400 + 0 × 100% = 8000 + 600 × 100% = 100% 93% 25

Tests for Liver Flukes in Cattle – Q 6 ü Necropsy = Gold Standard for Liver Flukes – 10, 000 cattle tested with necropsy, 20% (2, 000) are positive (truly infected with liver flukes) 2. Serology (ELISA) – 1, 880 of 2, 000 dzed cattle test positive with the ELISA • 120 false negatives – 7, 680 healthy cattle test negative with the ELISA • 320 false positives ü What is the PPV and NPV of the ELISA? – Calculate these showing the table Example from: Rapsch, et al. 2006. Int J Parasitol. 36: 1153 -1158 26

Let’s say…… ü You move to a new practice area where the prevalence of liver flukes is much lower (5%). ü What will be the PPV and NPV of coproscopy (Se = 70%, Sp = 100%) in your new practice area? 27

Coproscopy Se = 70%. Sp = 100 % 5% of the population infected Tests positive Tests negative Total Diseased = Healthy = actually has the doesn’t have condition the condition 0 0 (TP) (FP) Total 0 0 (FN) 0 (TN) 0 0 0 1, 000 PPV = TP TP + FP × 100% NPV = TN TN + FN × 100% Step One Start with an empty two by two table. Arbitrarily select an N, but make it fairly large; in this case I selected 1, 000. It’s easier using a large number because there will be no decimals. 28

Coproscopy Se = 70%. Sp = 100 % 5% of the population infected Tests positive Tests negative Total Diseased = Healthy = actually has the doesn’t have condition the condition 0 0 (TP) (FP) Total 0 0 (FN) 0 (TN) 0 50 0 1, 000 PPV = TP TP + FP × 100% NPV = TN TN + FN × 100% Step Two 5% of the population is infected with liver flukes, so, fill in the total number infected. 5% of 1000 (0. 05 X 1000 = 50) 29

Coproscopy Se = 70%. Sp = 100 % 5% of the population infected Tests positive Tests negative Total Diseased = Healthy = actually has the doesn’t have condition the condition 0 0 (TP) (FP) Total 0 0 (FN) 0 (TN) 0 50 950 1, 000 PPV = TP TP + FP × 100% NPV = TN TN + FN × 100% Step Three Fill in the total number of healthy (1, 00050 = 950) animals 30

Coproscopy Se = 70%. Sp = 100 % 5% of the population infected Tests positive Tests negative Total Diseased = Healthy = actually has the doesn’t have condition the condition 35 0 (TP) (FP) Total 0 0 (FN) 0 (TN) 0 50 950 1, 000 PPV = TP TP + FP × 100% NPV = TN TN + FN × 100% Step Four Fill in the TP by multiplying the total number diseased by the Se (50 x. 70 = 35). 31

Coproscopy Se = 70%. Sp = 100 % 5% of the population infected Tests positive Tests negative Total Diseased = Healthy = actually has the doesn’t have condition the condition 35 0 (TP) (FP) Total 0 15 (FN) 0 (TN) 0 50 950 1, 000 PPV = TP TP + FP × 100% NPV = TN TN + FN × 100% Step Five Fill in the FN by subtracting the TP from the total number diseased (50 – 35 = 15). 32

Coproscopy Se = 70%. Sp = 100 % 5% of the population infected Tests positive Tests negative Total Diseased = Healthy = actually has the doesn’t have condition the condition 35 0 (TP) (FP) Total 0 15 (FN) 950 (TN) 0 50 950 1, 000 PPV = TP TP + FP × 100% NPV = TN TN + FN × 100% Step Six Fill in the TN by multiplying the total number of healthy animals by the Sp (950 x 100 = 950). 33

Coproscopy Se = 70%. Sp = 100 % 5% of the population infected Tests positive Tests negative Total Diseased = Healthy = actually has the doesn’t have condition the condition 35 0 (TP) (FP) Total 0 15 (FN) 950 (TN) 0 50 950 1, 000 PPV = TP TP + FP × 100% NPV = TN TN + FN × 100% Step Seven Fill in the FP by subtracting the TN from the total number of healthy animals (950 = 0). 34

Coproscopy Se = 70%. Sp = 100 % 5% of the population infected Tests positive Tests negative Total Diseased = Healthy = actually has the doesn’t have condition the condition 35 0 (TP) (FP) Total 35 15 (FN) 950 (TN) 965 50 950 1, 000 PPV = TP TP + FP × 100% NPV = TN TN + FN × 100% Step Eight Fill the row totals by adding the rows. Total test positives = 35 + 0 = 35. Total test negatives = 15 + 950 = 965. Make sure the totals add up to N. 35

Coproscopy Se = 70%. Sp = 100 % 5% of the population infected Tests positive Tests negative Total Diseased = Healthy = actually has the doesn’t have condition the condition 35 0 (TP) (FP) Total 35 15 (FN) 950 (TN) 965 50 950 1, 000 PPV = TP TP + FP × 100% NPV = TN TN + FN × 100% Step Nine Fill in the PPV and NPV equations from the cells in the two by two table. 36

Coproscopy Se = 70%. Sp = 100 % 5% of the population infected Tests positive Tests negative Total Diseased = Healthy = actually has the doesn’t have condition the condition 35 0 (TP) (FP) Total 35 15 (FN) 950 (TN) 965 50 950 1, 000 PPV = TP TP + FP × 100% = NPV = TN TN + FN × 100% = 35 35 + 0 × 100% = 950 + 15 × 100% = 100% 98% 37

Let’s say…… ü You move to a new practice area where the prevalence of liver flukes is much lower (5%). ü What will be the PPV and NPV of the ELISA in your new practice area? ü Try this on your own, and compare to the next slide. ü NOTE: use the Se and SP that you calculated for the ELISA. 38

Problem Set The SNAP 4 Dx canine test kit produced by IDEXX detects heartworm antigen, Ehrlichia canis antibodies, Anaplasma phagocytophilum antibodies, and Borrelia burgdorferi (Lyme disease) antibodies. This test does not detect anti-vaccine antibodies, but is specific for natural infection. 39

Sensitivity & Specificity ü When independent researchers compared the Idexx SNAP to the gold standard of IFA + Western blotting, the Idexx SNAP assay detected 238 of 252 positive dogs – What is the sensitivity? ü When other researchers tested 987 dogs without Lyme disease with the Idexx SNAP assay, 982 of them tested negative – What is the specificity? 40

According to the Manufacturer ü Idexx performed their own testing in-house, and they advertise a sensitivity of 95. 0% and a specificity of 99. 7%, slightly better than that obtained by outside researchers. ü Lyme disease is most common in the northeastern USA, where the tick vector is often found outside during the summertime. 41

Scenario 1: An Endemic area – Q 7 ü Lyme disease is most common in the northeastern USA, with 20. 63% prevalence of disease in dogs in Massachusetts. ü Let us say you test 5, 000 dogs from MA. ü Given the prevalence of disease, – How many would have Lyme disease? Calculate showing how you got the number. – How many would be healthy? Calculate showing how you got the number. 42

Filling in the Grid Dogs with Healthy Lyme Disease Dogs Totals SNAP-positive SNAP-negative Totals 1, 032 3, 968 5, 000 43

Scenario 1, Continued ü Using the sensitivity and specificity data provided by Idexx, predict how many of the Lyme disease dogs would test positive. – Sensitivity = 95. 0% – So, 95. 0% of the dogs with Lyme disease will test positive using the SNAP test. 95. 0% of 1, 032 = 980 dogs ü How many diseased dogs will test negative? 5. 0% of 1, 032 (all the other infected dogs) = 52 dogs 44

Filling in the Grid Dogs with Healthy Lyme Disease Dogs SNAP-positive 980 SNAP-negative 52 Totals 1, 032 3, 968 Totals 5, 000 45

Scenario 1, Continued ü Using the sensitivity and specificity data provided by Idexx, predict how many of the healthy dogs would test negative? – Specificity = 99. 7% – So, 99. 7% of the healthy dogs will test negative using the SNAP test. 99. 7% of 3, 968 = 3, 956 dogs ü How many healthy dogs will test positive? 0. 3% of 3, 968 (all the other healthy dogs) = 12 dogs 46

The Finished Grid Dogs with Healthy Lyme Disease Dogs Totals SNAP-positive 980 12 992 SNAP-negative 52 3, 956 4, 008 1, 032 3, 968 5, 000 Totals What is the PPV? = 980 ÷ (992) = 98. 8% What is the NPV? = 3, 956 ÷ (4, 008) = 98. 7% 47

Scenario 2: Rare Disease ü The prevalence in Florida is much lower, estimated to be 0. 17%. ü If you test 5, 000 dogs from Florida, given the sensitivity (95. 0%) and specificity (99. 7%) provided by IDEXX, – How many dogs would have Lyme disease? 0. 17% of 5, 000 = 8 dogs – How many dogs would be healthy? All the others! = 4, 992 dogs 48

Scenario 2, Continued ü How many of the Lyme disease dogs would test positive? 95. 0% of 8 = 8 dogs ü How many diseased dogs will test negative? 5. 0% of 8 = 0 dogs ü How many healthy dogs would test negative? 99. 7% of 4, 992 = 4, 977 dogs ü How many healthy dogs would test positive? 0. 3% of 4, 992 = 15 dogs 49

Scenario 2 Dogs with Healthy Lyme Disease Dogs Totals SNAP-positive 8 15 23 SNAP-negative 0 4, 977 Totals 8 4, 992 5, 000 What is the PPV? = 8 ÷ (23) = 35. 6% What is the NPV? = 4, 977 ÷ (4, 977) = 100. 0% 50

The Moral of the Story? One Test, Two States Massachusetts Florida ü 20. 63% prevalence ü 0. 17% prevalence ü PPV = 98. 8% ü NPV = 98. 7% ü PPV = 35. 6% ü NPV = 100% Is it worth the $$ to test all of these dogs? Do you treat the positive animals? 51

Critical Thinking ü How many false positives would you expect from a population of 11, 197 healthy dogs? – Sensitivity 95. 0% ~ Specificity 99. 7% ü In a recent scientific paper, a total of 11, 197 dogs from Louisiana were tested: 9 dogs tested positive with the SNAP test. – Does this mean that Louisiana has Lyme disease? 52

A Little Bit of “Cheating” ü What is a quick, easy indicator that the PPV is going to be VERY LOW? – Look at the % of animals with the disease – Look at the % of false positives • For rare diseases, this approximates “ 100% - specificity” ü If there are more false positives than diseased animals, the PPV will be very low (<50%)! 53

Example: Brucellosis in Kyrgyzstan – Q 8 ü Out of a herd of 10, 000 sheep – Prevalence is actually 2. 0% – So, how many sheep have Brucella? _____ ü We test these sheep with an ELISA that has 98% sensitivity/95% specificity – # of true positives: – # of false negatives: – # of true negatives: – # of false positives: – Calculate PPV and NPV 54

How Can You Compensate for a Low PPV? ü Confirmatory testing! ü PCR has a specificity of 100% – Used to test all 686 seropositive sheep – Will discriminate between the true positives and true negatives, identifying all of the TP ü In the end. . . All of the positives were “truly” positive, and there were only 4/10, 000 false negatives! 55