Curvewise DET Confidence Regions and Pointwise EER Confidence

Curvewise DET Confidence Regions and Pointwise EER Confidence Intervals Michael Schuckers Yordan Minev St. Lawrence University, Canton, NY, US and Center for Identification Technology Research (CITe. R) Andy Adler Carleton University, Ottawa, Canada Funding for this work comes from NSF grants CNS-0325640 and CNS-0520990. CNS-0325640 is cooperatively funded by the National Science Foundation and the United States Department of Homeland Security. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. ICB 2007 - 2007/08/28

Motivation Individual Humans Automated Classifier ICB 2007 - 2007/08/28

Significantly Different? ICB 2007 - 2007/08/28

Background • Goals (ROC Confidence Region-CR) – Truly bivariate in nature (both FAR & FRR) – Deals with multiplicity – Yields inference for EER ICB 2007 - 2007/08/28

Previous Work • Scores and scores in statistics literature, see Pepe (2004) or Zhou et al (2002) for overviews • In biometrics/machine learning – Poh et al (2007? ) – Dass et al (2007) – Macskassy et al (2005) ICB 2007 - 2007/08/28

Background Individual Humans EER Automated Classifier ICB 2007 - 2007/08/28

Significantly Different? EER ICB 2007 - 2007/08/28

Radial Sweep ICB 2007 - 2007/08/28

Background • Goals (ROC/DET Confidence Region(CR)) – Truly bivariate in nature (both FAR & FRR) – Deals with multiplicity – Yields inference for EER ICB 2007 - 2007/08/28

Radial Sweep • Following choices – DET Center (location) cy=cx=1 For EER chose cy=cx – Number of angles and uniformity Too few = poor performance – Number of thresholds and uniformity Original scores ICB 2007 - 2007/08/28

Steps in Radial Sweep 1. Determine thresholds 2. Calculate ROC/DET 3. Convert from (x, y) to (r, q) at each angle ICB 2007 - 2007/08/28

Confidence Region (CR) Algorithm 1. Calculate ROC/DET in polar coordinates and call it r = {rq: q in Q}. 1. Bootstrap both genuine & imposter (healthy & disease) scores 2. Create new ROC/DET in polar coordinates, call it r* for same q’s 3. Repeat Steps 2 & 3 K times 4. Find bounds at each angle such that (1 -a)100% of bootstraps curves fall inside bounds. 6. Use bounds in previous step to make CR ICB 2007 - 2007/08/28

Finding CR bounds Multiple ways to find bounds: 1. 2. 3. 4. 5. 6. rq+/- h rq + g U , r q - g L rq+/- d √ (var(rq)) rq+f. U √ (var(rq)), rq- f. L√(var(rq)) rq+/- bq rq+zqu, rq-zqu We chose #4 with slight adjustment ICB 2007 - 2007/08/28

Finding f. U, f. L • For the kth bootstrapped ROC curve, find all normalized difference of rθ* from rθ at each θ tkθ = • For the kth bootstrapped ROC curve, if |min(t(θ))| > |max(t(θ))| m[k] = min(t(θ)) else m[k] = max(t(θ)) • Repeat this for all K bootstrapped ROC curves ICB 2007 - 2007/08/28

Confidence Region Algorithm ICB 2007 - 2007/08/28

Finding f. U, f. L • Sort all normalized absolute differences m[k], k=1, …, K • To find (1 -α)100% confidence region for the Population ROC curve, we need to ensure that (1 -a)100% of bootstrapped ROC curves fall within the following bounds: – lower bound: h. U = (1 - a/2)th percentile of the m[k]’s – upper bound: h. L = (a/2)th percentile of the m[k]’s Hall adjustment on bounds, using e=10 -4 ICB 2007 - 2007/08/28

Recommendations – At least 100 q’s (angles) – At least 100 thresholds – e~10 -4 edge (axes) effects – Min(sample size)>50 for 95% CR ICB 2007 - 2007/08/28

Examples: Hand Geometry ICB 2007 - 2007/08/28

Example: Fingerprint ICB 2007 - 2007/08/28

Example: Facial Recognition ICB 2007 - 2007/08/28

Table of Pointwise EER’s Modality Lower Bound EER Upper Boundh Hand Geometry 0. 1178 0. 1494 0. 1820 Fingerprint 0. 0458 0. 0715 0. 1000 Face 0. 0303 0. 0575 0. 0766 Note: Hall adjustment to confidence intervals ICB 2007 - 2007/08/28

Final Comments CR for entire curve (ROC or DET), not pointwise CI for EER(angle at 45 degrees) Generalizes most other sweeping methods vertical cy = -1010, cx= 0. 5 horizontal cy = 0. 5, cx= -1010 ICB 2007 - 2007/08/28

Thank You schuckers@stlawu. edu ICB 2007 - 2007/08/28