Recognition System Capacity Joseph A OSullivan Samuel C

Recognition System Capacity Joseph A. O’Sullivan Samuel C. Sachs Professor Electronic Systems and Signals Research Laboratory Department of Electrical and Systems Engineering Washington University in St. Louis (314) 935 -4173; http: //essrl. wustl. edu/~jao jao@wustl. edu Michael D. De. Vore, UVA Naveen Singla Brandon Westover Supported in part by the Office of Naval Research Adaptive Sensing MURI Review, 06/27/06 1

O’Sullivan AS-MURI 2006 Recognition System Capacity • Motivation: – ATR; Network Centric Warfare; Biometrics; Image Understanding • Active Computations • Achievable Rate Regions – Inner and outer bounds – Successive refinement 2

O’Sullivan AS-MURI 2006 Why Theorems? • ONR Perspective: Want Systems That Work – Implementable on projected system architecture – Good performance • Our Perspective: Theorems Provide – Provable performance: bounds and guidelines – Validation and critique of existing system designs – Motivation for recognition system design: system architectures; database design; optimal compression for recognition; communication for recognition; active computations • Growing Awareness of Importance of Information Theory 3

O’Sullivan AS-MURI 2006 Perspective on Image Understanding • “Finding tanks is so World War II. ” Bob Hummel, DARPA program manager, ATR Theory Workshop, Dec. 2004 – What make of car? – What year? – Who is driving? – Where has it been? • Improvised Explosive Devices (IED) • Demand more information from imagery 4

O’Sullivan AS-MURI 2006 Biometrics Must Be • • Universal Permanent Unique Measurable Encoding Uniqueness How unique? Bits. Measurability How measurable? Bits. A. K. Jain, et al. , “Introduction to Biometrics, ” 1999 5 John Daugman, http: //www. cl. cam. ac. uk/users/jgd 1000/

O’Sullivan AS-MURI 2006 Recognition System Capacity • Motivation: – ATR; Network Centric Warfare; Biometrics; Image Understanding • Active Computations • Achievable Rate Regions – Inner and outer bounds – Successive refinement 6

O’Sullivan AS-MURI 2006 Active Computations Concept System Performance Analysis Resource Allocation Network Resources databases, communications, etc. Performance Estimate Algorithm (ATR) Sensors Target Type Estimate Sensor Data § Compute a sequence of inferences and performance estimates (probabilities or reliabilities). § Monitor available resources (time, processors, bandwidth, database, …). § Feed back performance: select next computation; reallocate resources; demand more data. 7

O’Sullivan AS-MURI 2006 Active Computations Concept System Performance Analysis Resource Allocation Network Resources databases, communications, etc. Performance Estimate Algorithm (ATR) Sensors Target Type Estimate Sensor Data § Successively refined inferences § Time or resources to achieve performance goal. § Additional data required to achieve performance goal. 8

O’Sullivan AS-MURI 2006 Total System Performance Resource Consumption Approximations • ATR system performance entails more than just accuracy: – Time to classify a target – Electrical power dissipation – Sensor engagement, CPU cycles, bits communicated, and other “opportunity costs” • Need real-time estimates of total system performance – Enable informed tradeoff of ATR accuracy with throughput and network resource consumption – Dynamically adapt the system as requirements, capabilities, and operational scenarios evolve 9

O’Sullivan AS-MURI 2006 Active Computation • The need to actively manage computations is acute in complex, time-critical environments – Information has a time value – Some information now may be better than a lot of information later, after it is too late to take decisive action – Ideally, we’d like some information now and more later • Static ATR implementations perform the same computations for every image they receive – No tentative answers are available before processing is finished – Availability of more time will not improve the solution accuracy 12

O’Sullivan AS-MURI 2006 Active Computation • Dynamic ATR systems employ active computations to maximize the time-value (or resource-value) of information • Approach: Generate a sequence of increasingly accurate classifications – More resources are consumed at every stage – Continue until accuracy is good enough, resource cap is reached, or the result is no longer relevant – Control the computations to maximize the total information value 13

O’Sullivan AS-MURI 2006 Active Computation • Maximum-likelihood ATR solution is • Solve a sequence of simpler problems – – – The functions get closer to with each stage Each problem is easy to solve given previous solutions Let be the sequence of problems that are chosen up to stage k Let be the error and be the resources used The best strategy at stage K minimizes the total expected cost 14

O’Sullivan AS-MURI 2006 Active Computation • Seek heuristic strategies that do not require prior knowledge of K, but are nearly optimal for all K • For example, maximize the expected future increase in likelihood 15

O’Sullivan AS-MURI 2006 Recognition System Capacity • Motivation: – ATR; Network Centric Warfare; Biometrics; Image Understanding • Active Computations • Achievable Rate Regions – Inner and outer bounds – Successive refinement 16

O’Sullivan AS-MURI 2006 Recognition System Capacity: More Motivation ● Number of bits for recognition ● Number of patterns that can be distinguished ● Number of bits to extract from data ● Size of long term memory ● Data and processing dependence ● Start with simple i. i. d. model 17

O’Sullivan AS-MURI 2006 p(x) Rc Rx memory encoder X 1 X 2. . . XMc f Select one xh p(y|x) p(h) y sensory encoder φ V memory representation Objective: U 1, U 2, …, UMx g Ry sensory representation hˆ Pr{g(V)=h}>1 -ε, s. t. R=(Rc, Rx, Ry) 18

O’Sullivan AS-MURI 2006 Pattern Recognition Codes and Achievable Rates 19

Characterizing O’Sullivan AS-MURI 2006 20

O’Sullivan AS-MURI 2006 V=Y Rc=I(X; Y)-I(X; Y|U) Rc=I(X; Y) H(Y) `Unlimited’ U, V capacity: U=X, Y=V Rc < I(X; Y) Random channel coding On the border, U=X Rc=0 U-X-Y-V , so R*=R**=R. Rc=I(X; Y)-I(X; Y|V) I(Y; V) 0 0 Rc=0 I(X; U) Rx > I(X; U) Ry > I(Y; V) Rc < I(U; V)-I(U; V|X, Y) H(X) Poor memory: U=0 Rc < I(0; V)=0 Poor senses: V=0 Rc<I(U; 0)=0. 21

O’Sullivan AS-MURI 2006 A Related Gap: the distributed source coding problem X f (U, V) p(x, y) Y g φ - Problem: Characterize the achievable (Rx, Ry, Dx, Dy) -Sergio Servetto claimed solution at ITW 2006 -Solution should transfer to our problem 22

O’Sullivan AS-MURI 2006 Related Work 23

24

Naval Impact/Payoff: O’Sullivan AS-MURI 2006 The Sensor to Shooter Problem Terminal ATR Weapon Link Shooter Link Sensor Data Link • Recce Imagery (SAR, IR, Visible) • Intelligent Bandwidth Compression • Wide Area Cueing (ATC) • Strike plan • Target location • ATR parameters Targeting Info Link • Link Target data to TOC (type, location, motion, …) • ATR • Reference Library, Aimpoint • Select Target • Strike Planning • Weapon Selection Ground Recce/Intel Station 25 Strike Planning System

O’Sullivan AS-MURI 2006 Convex Hull Inner Bound Westover and O’Sullivan ISIT 2005 • The convex hull of this inner bound is achievable. • Consider the set of all distributions such that conditioned on a random variable Q, we have U – X – Y – V. Then the achievable region is • For every case examined, this convex hull is achieved by time sharing between a length 4 MC and the (0, 0, 0) point. 26

Successive Refinement, Two-Stage Recognition “Up and to the right” Given a sequence of (Mx 1, My 1, Mc 1, n) pattern recognition codes, design a sequence of (Mx 2, My 2, Mc 2, n) PR codes with the first sequence as subcodes Mx 1≤ Mx 2 My 1 ≤ My 2 Coarse Code: (f 1, Φ 1, g 1)n Mc 1 < Mc 2 Refining Code: (f 2, Φ 2, g 2)n The rate sextuplet (Rx 1, Ry 1, Rc 1, Rx 2, Ry 2, Rc 2) is achievable if there exist sequences of recognition codes (f 1, Φ 1, g 1)n and (f 2, Φ 2, g 2)n such that Comment: two different systems (different patterns) 27

Achievability: Inner Bound Theorem: Two-stage recognition is achievable if there exist auxiliary random variables U 1, V 1, U 2, and V 2 satisfying • Markov conditions: U 1 – X – Y – V 1 and (U 1, U 2) – X – Y – (V 1, V 2). • Rate Bounds: Inner Bound: Proof Sketch. At the coarse stage: • Use the coding strategy for the single-stage pattern recognition system At the refining stage: • Given memory and sensory indices from the coarse stage, generate “refining” codebooks according to the conditional distributions p(u 2|u 1(·)) and p(v 2|v 1(·)). • Encode memory and sensory data with pairs of indices corresponding to coarse and refining stage • Use typical-set decoding to identify pattern 28

O’Sullivan AS-MURI 2006 Inner Bound: Successive Refinement Corollary: Successive refinement is achievable if there exist auxiliary random variables U 1, V 1, U 2, and V 2 satisfying • Markov condition: U 1 – U 2 – X – Y – V 2 – V 1. • Rate bounds: Analogous to the Markov condition for successive refinement for rate-distortion. Equitz and Cover, “Successive refinement of information, ” IEEE Trans. Info. Theory, Mar. 1991. 29

O’Sullivan AS-MURI 2006 Converse: Outer Bound Theorem: If the rate sextuplet (Rx 1, Ry 1, Rc 1, Rx 2, Ry 2, Rc 2) is achievable then there exist auxiliary random variables U 1, V 1, U 2, and V 2 satisfying • Markov conditions: U 1 – X – Y and X – Y – V 1 and (U 1, U 2) – X – Y and X – Y – (V 1, V 2). • Rate Bounds: Corollary: Two length 4 Markov chains follow: U 1 – U 2 – X – Y and X – Y – V 2 – V 1 30

O’Sullivan AS-MURI 2006 Extension: Hierarchical Recognition Based on Random Labels • Extend results so that the codebook is the same for the two stages • Randomly label each Xn(k) with a label L(k), out of exp[n(Rc 2 -Rc 1)] labels • For each label, use a (Mx 1, My 1, Mc 1, n) pattern recognition code • Given Yn, run every decoder (for every label) list of exp[n(Rc 2 -Rc 1)] possible patterns • Use refinement codebooks to determine label and therefore the pattern 31

O’Sullivan AS-MURI 2006 Extension: Hierarchical Recognition Based on Hierarchical Pattern Model • Assume that the patterns are generated by a hierarchical model W X (class identity) • Inner Bound: U 1 – W – Y – V 1 and U 1 – U 2 – X – Y – V 2 – V 1 • Use a (Mx 1, My 1, Mc 1, n) pattern recognition code to obtain Wn(i) (class) p(w) • Use refinement codebooks to determine Xn(i, j) (identity) p(x|w) 32

O’Sullivan AS-MURI 2006 Extensions • Inner bounds for prototypical examples: Gaussian, binary. Convex hull is achievable by successive refinement. • Successive refinement “up and to the right” 33

O’Sullivan AS-MURI 2006 Recognition System Design • • Developing robust ATR algorithms, deriving limits on recognition performance Quantifying recognition performance as a function of system resource measures Developing algorithms and implementations that adapt to dynamically varying resource constraints – Time, availability of processors, communication bandwidth, data storage, sensor image quality Impact: increase efficiency and effectiveness of system implementations – Information latency problem – Recognition systems using visual imagery, SAR, ladar – Increase in ATR performance – Allow more imagery to be screened – Provide systematic tools for analyzing design choices such as processors and network communication Collaborators Washington University Joseph A. O’Sullivan Andrew Li Naveen Singla Po-Hsiang Lai Lee Montagnino Brandon Westover Robert Pless Ronald S. Indeck Natalia A. Schmid (UWVa) Michael D. De. Vore (UVa) Alan Van Nevel 34

O’Sullivan AS-MURI 2006 35

O’Sullivan AS-MURI 2006 Selected Limitations of Existing Systems • “Stovepipe” design – Fixed inputs, processing, database, output – Fixed time – Algorithms are not transparent Seek “any-time” adaptive system design 36

O’Sullivan AS-MURI 2006 Naval Capability Provided “ Network centric warfare is military operations that exploit information and networking technology to integrate widely dispersed human decision makers, situational and targeting sensors, and forces and weapons into a highly adaptive, comprehensive system to achieve unprecedented mission effectiveness. ” Network-Centric Naval Forces, Naval Studies Board, National Research Council, 2000 Active Computations Ø Exploit technology Ø Integrate sensors, resource allocation, decision makers, algorithms Ø Adapt to dynamically varying resources Ø Provide measures of uncertainty as a a function of available resources 37