Imager Design using ObjectSpace Prior Knowledge M A
Imager Design using Object-Space Prior Knowledge M. A. Neifeld University of Arizona OUTLINE 1. The Last Slot 2. Introduction 3. PSF Engineering 4. Feature-Specific Imaging Neifeld IMA 2005
Neifeld IMA 2005 Introduction: objects are not iid pixels. - Conventional cameras are designed to image iid pixels impulse-like point-spread-functions (identity transformation) generic metrics such as resolution, field of view, SNR, etc. - Real objects are not iid pixels so don’t estimate pixels - This keeps the compression guys employed! - (106 pixels)(3 colors/pixel)(8 bits/color) = 2. 4 x 107 bits - (1011 people)(4 x 109 years)(109 images/year) = 4 x 1029 images <100 bits - The set of “interesting” objects is small - Many ways to characterize “interesting” objects: power spectra, principal components, Markov fields, wavelet projections, templates, task-specific models, finite alphabets, etc. Information depends upon task: q Option 1 - this is a random image I = 107 bits q Option 2 – this is a “battlefield” image I = ? bits … how to quantify PDF! q Option 3 – this image either contains a tank or not I = 1 bit … task-specific source model
Neifeld IMA 2005 Introduction: post-processing exploits priors. - Linear Restoration: de-noising and de-blurring exploit noise statistics, object power spectra, principal components, wavelets, … - Nonlinear Restoration: super-resolution uses finite support, positivity, finite alphabet, power spectra, wavelets, principal components, isolated points, … - Recognition: features, templates, image libraries, syntax, invariance, … - Finite Alphabet Post-Processing Examples LADAR largest return rmse = 7. 3 mm Object Viterbi rmse = 0. 6 mm Wiener rmse = 5. 8 mm Axial extent of target = Temporal pulse width = 30 mm. Target feature size = Scan step size = 4. 6 mm Multi-Frame Super-Resolution Object Measurement IBPP – 24% IBP – 28% 2 D 4 - 2% Optical blur = 1. 5 and pixel-blur = 2. Reconstruction from 2 images, σ = 1%
Neifeld IMA 2005 Introduction: plausibility of a single pixel imager. u Measure only what you want to know Source volume ®Fluorescent markers ®Distant “bright” objects: aircraft, missile, stars r 1 Imager r 2 y x r. M M : Number of point sources z Strong Object Model: Equal-intensity monochromatic point sources u Scene is completely specified by sources positions: r 1 r 2 … r. M u Imager Goals: Estimate point source position(s): { r 1 r 2 … r. M } u Conventional image may be formed as a post-processing step u r 1 r 2 … r. M Conventional image
Neifeld IMA 2005 Introduction: information-based design. q Optimize imager based on information metric. q Maximize measurement entropy. q Select detector sizes and positions based on measurement pdf. 1 Source Volume phase mask 40 cm 2 2 1 3 3 1 cm 3 source power = 0. 5 m. W Measurement log-pdf Lens Measurement log-pdf Detector NEP=2 n. W d 1 , h 1 Measurement log-pdf cubic phase random phase
Neifeld IMA 2005 Introduction: single pixel imager results. Single Source in Volume Detector(s) : Imager Type Multiple Sources in Volume Conventional CPM RPM One detector in one aperture 21% 39% 65% Two detectors in one aperture 30% 54% 74% Two detectors in two apertures 36% 74% 89% q Object-space prior knowledge should inform the optical design q Let’s utilize this viewpoint in a more useful problem domain
Neifeld IMA 2005 PSF ENGINEERING
Neifeld IMA 2005 PSF Engineering: Under-Sampled Imagers w Imagers for which pixel size > optical spot size. . w Large pixels result in under-sampling/aliasing. w Sub-pixel shifted measurements to resolve ambiguity. shift camera Frame 1 spatial ambiguity …. . Frame 2 w Optical degrees of freedom not exploited. w We consider engineering optical point spread function. Frame K
Neifeld IMA 2005 Imaging Model Object: f Imaging operator: H Measurements: g …. . N = 512 x 512 Phase-mask Optics details: w Resolution = 0. 2 mrad/1 mm w Field of view = 0. 1 rad w Thickness = 5 mm w Aperture = 2. 75 mm w F/# = 1/1. 8 Sub-pixel shifts Sensor details: w Pixel = 7. 5 mm w Under-sampling = 15 x w Full well capacity = 49 kew Spectral bandwidth = 10 nm w Center wavelength = 550 nm … . . M = 34 x 34 w Single frame signal to noise ratio: SNR = 10 log[sqrt(Ne)] = 23. 3 d. B w SNR can be improved via multi-frame averaging ~ sqrt(K) w Total photon-count is kept constant over multiple-frames.
Neifeld IMA 2005 Linear Reconstruction u Linear imaging model: g = Hf + n (note: n is AWGN) u Block-wise shift-invariant imaging operator H is M x N u Problem: M << N (e. g. , M=N/15) ^ u Linear minimum mean square error (LMMSE) reconstruction: f = Wg u LMMSE operator: W = Rf. Ht(HRf. Ht+Rn)-1 u No Priors = flat PSD u Priors = power law PSD or triangle PSD Example training objects Power Law PSD(f) = 1/f PSD model
Neifeld IMA 2005 Performance Measures RMSE=8. 6% u Root Mean Squared Error: Object Composite Channel Hc u Angular resolution: Point Object f = d(r) Composite Channel Hc n g LMMSE Reconstruction + g Reconstruction to Diffraction-limited sinc 2 =0. 4 mrad ^ f
Conventional/TOMBO Imager Results TOMBO Imager Conventional Imager Shift-sensor RMSE for TOMBO Neifeld IMA 2005 sub-pixel shift Sub-pixel shifted measurements Resolution for TOMBO
Alternate PSF u Consider use of extended point spread function(PSF) Neifeld IMA 2005 impulse-like PSF u Design issue #1: retain full optical bandwidth u Design issue #2: tradeoff SNR for condition number u Pseudo-Random Phase masks for extended PSF Realization of a spatial Gaussian random process. - mask roughness - mask correlation length Pseudo-Random Phase mask Enhanced Lens (PRPEL) Example PSF( =0. 5 , =10 ) Modulation Transfer Function
Neifeld IMA 2005 Resolution Results Resolution for PRPEL and TOMBO u. All designs use optimal roughness. u Note more rapid convergence of PRPEL compared to TOMBO. u Higher resolution achieved by PRPEL at reduced number of frames. u PRPEL achieves 0. 3 mrad resolution at K=5 compared to K=12 for TOMBO.
Neifeld IMA 2005 RMSE Results RMSE for PRPEL and TOMBO u PRPEL makes effective use of prior knowledge at K=1 u Note more rapid convergence of PRPEL. u PRPEL consistently out-performs TOMBO PRPEL K=1 K=2 K=3
Neifeld IMA 2005 PRPEL Summary 4% RMSE requirement RMSE achieved at M=N/4 Imager Type Number of Frames TOMBO PRPEL Imager Type (K=4) TOMBO PRPEL K 5 4 RMSE 4. 2% 3. 9% 0. 3 mrad Resolution requirement Resolution achieved at M=N/4 Imager Type Number of Frames TOMBO PRPEL Imager Type (K=4) TOMBO PRPEL K 12 5 Resolution 0. 60 mrad 0. 35 mrad u PRPEL imager achieves 60% improvement in resolution. u PRPEL imager obtains 22% improvement in RMSE.
PSF Engineering via SPEL Neifeld IMA 2005 u Sine-Phase mask Enhanced Lens(SPEL) : Phase offset Spatial-frequency Amplitude Phase-mask u Pick N=3: yields 12 free parameters for optimization. u Optimization criteria: u RMSE computed over object class using LMMSE operator. u PSF is optimized for each value of K.
Neifeld IMA 2005 Optimized PSF K=1 Observations Note smaller support of SPEL PSF compared to PRPEL PSF. u SPEL PSF also contains subpixel structure. u SPEL PSF has more efficient photon-distribution. u K=2 Observations u PSF support reduces with increasing K. u SPEL PSF is array of delta pulses.
Optimized PSF: System Implications Neifeld IMA 2005 K=16 Observations u SPEL PSF converges to delta pulses as K increases. u In limit K 16 we observe that SPEL PSF to converge to TOMBO-like PSF.
Neifeld IMA 2005 Results RMSE : Power law PSD PRPEL SPEL K=1 K=2 K=3 RMSE for SPEL, PRPEL, and TOMBO u SPEL provides best use of prior knowledge for K=1 u SPEL outperforms TOMBO by 47% in terms of RMSE(K=8). u SPEL improves RMSE by 35% compared to PRPEL (K=8).
Results Angular resolution Neifeld IMA 2005 Resolution for SPEL, PRPEL and TOMBO u Note PSF optimization was performed over RMSE. u SPEL out-performs TOMBO. u SPEL performance compared to PRPEL improves with increasing K. q PSF engineering can exploit weak object prior knowledge to improve performance q Stronger object prior knowledge can enable non-traditional image measurement
Neifeld IMA 2005 FEATURE-SPECIFIC IMAGING
Neifeld IMA 2005 Passive Feature-Specific Imaging: Motivation Conventional imaging system Feature extraction Features Task Restoration, recognition, compression, etc. noisy image noise Feature-specific optics Feature-specific imaging system (FSI) PCA, ICA, Fisher, Wavelet, etc. Features Task noise u Feature-Specific Imaging (FSI) is a way of directly measuring linear features (linear combinations of object pixels). u Attractive solution for tasks that require linear projections of object space u Let’s consider a case for which task = pretty picture
FSI for Reconstruction Neifeld IMA 2005 u PCA features provide optimal measurements in the absence of noise Noise-free reconstruction: PCA solution : General solution : photon count constraint Result using PCA features:
Optimal Features in Noise Neifeld IMA 2005 u PCA features are not optimal in presence of noise Noise-free problem statement: Note: PCA error is no longer monotonic in the number of features trade-off between truncation error and photon count constraint • Object block size = 4 x 4 • Noise = AWGN • We use stochastic tunneling to optimize/search RMSE = 12. 9 RMSE = 124 RMSE = 12 RMSE = 11. 8
Optimal Features in Noise u Error increases as number of feature increases for PCA solution u Reconstructed is improved significantly by using optimal solution u Optical implementation requires non-negative projections Neifeld IMA 2005
Passive FSI Result Summary u Optimal FSI is always superior to conventional imaging u Non-negative solution is a good experimental system candidate Neifeld IMA 2005
Neifeld IMA 2005 Passive FSI for Face Recognition • Face recognition from grayscale image feature measurements • Class of 10 faces, 600 images per face • Training = 3000 faces and testing = 3000 faces • Features: wavelet, PCA, Fisher, … • Recognition algorithms: - k – nearest neighbor based on Euclidean distance metric - 2 -layer neural networks batch trained using back-propagation with momentum FSI Sample images from face database [Each image is 128 x 96] Conventional First Wavelet feature of the above images [Each feature is 8 x 6]
Passive FSI Optical Implementations Neifeld IMA 2005
Active Feature-Specific Imaging: Motivation • What is active illumination ? Neifeld IMA 2005 Object • Project known structure onto scene • Additional degrees of freedom improve imager performance Illumination pattern Projector • Past work on active illumination focused on: • Obtain depth-information for 3 D objects • Enhanced resolution for 2 D objects • Our goals: • Improve object- and/or task-specific performance • Simplify light collection hardware Conventional cameras
Neifeld IMA 2005 FSAI System Flow Diagram • Illumination patterns are eigenvectors (refer as PCA - FSAI) 16 × 16 replication of eigenvector P 1 PM P 2 Light Collection Object G Sequence of illumination patterns 16 × 16 detector 64 × 64 Photodetector noise (AWGN) H (optics operator) • Advantages (Estimate of feature weight) • Small number of detectors • High measurement SNR • Task is to produce object estimate using these values ece Vector of Measurements
Neifeld IMA 2005 FSAI Post-Processing Measurement vector Linear postprocessing Ŵ Ĝ=ŴR ≠ (suboptimal in noise) • Post-processing operator Ŵ is obtained by minimizing J • The MMSE operator is given by: N 2 = number of pixels, M = number of patterns • Metric to evaluate reconstructions : ece
Illumination Using Optimal Patterns Neifeld IMA 2005 • PCA vectors are not optimal in presence of noise • Minimize the residual MMSE (JMMSE) with respect to both Pi’s and Ti’s • Optimal features depend on M, SNR = 26 d. B PCA M=4 optimal PCA M=8 optimal ece
Neifeld IMA 2005 FSAI Results SNR = 26 d. B (LOW NOISE) Original object PCA-FSAI (uniform T) PCA-FSAI (optimal T) Optimal FSAI M=4 M=8 • Minimum from PCA-FSAI RMSE = 0. 0633 • Minimum from optimal FSAI RMSE = 0. 0465 ece
Neifeld IMA 2005 FSAI Results Summary Algorithm Uniform illumination SNR = 26 d. B SNR = 16 d. B 0. 067 (M = 1) 0. 151 (M =1) 0. 063 (M = 4) 0. 0768 (M = 2) 0. 063 (M > 4) 0. 0768 (M > 2 ) PCA – FSAI (uniform T) PCA – FSAI (nonuniform T) Optimal features 0. 0465 (M = 16) 0. 07 31 % 54 % Improvement of optimal FSAI compared to uniform illumination ece (M = 16)
Conclusions q Objects are not iid pixels Pixel-fidelity should not be the goal of an imager Need new non-traditional design metrics q Design should reflect prior knowledge of objects Object-specific imagers (e. g. , SPEL) Joint design of optics and post-processing q Design should reflect prior knowledge of application Task-specific imagers (e. g. , FSI) Neifeld IMA 2005
- Slides: 36