How to Make Printed and Displayed Images Have

How to Make Printed and Displayed Images Have High Visual Quality Brian L. Evans Embedded Signal Processing Laboratory The University of Texas at Austin TX 78712 -1084 http: : //www. ece. utexas. edu/~bevans Ph. D. Graduates: Graduate Student: Support: Niranjan Damera-Venkata (HP Labs) Thomas D. Kite (Audio Precision) Vishal Monga HP Labs National Science Foundation

Outline • Introduction • Grayscale halftoning for printing – – Screening Error diffusion Direct binary search Linear human visual system model • Color halftoning for display – Optimal design – Linear human visual system model • Conclusion UT Center for Perceptual Systems 2

Need for Digital Image Halftoning • Many devices incapable of reproducing grayscale (e. g. eight bits/pixel or gray levels from 0 to 255) – Laser and inkjet printers – Facsimile machines – Low-cost liquid crystal displays • Grayscale imagery binarized for these devices • Halftoning tries to reproduce full range of gray while preserving quality and spatial resolution – Screening methods are fast and simple – Error diffusion gives better results on some media UT Center for Perceptual Systems 3

Digital Halftoning Methods Clutered Dot Screening AM Halftoning Dispersed Dot Screening FM Halftoning Error Diffusion FM Halftoning 1975 Blue-noise Mask FM Halftoning 1993 Green-noise Halftoning AM-FM Halftoning 1992 Direct Binary Search FM Halftoning 1992 UT Center for Perceptual Systems 4

Screening (Masking) Methods • Periodic array of thresholds smaller than image – Spatial resampling leads to aliasing (gridding effect) – Clustered dot screening produces a coarse image that is more resistant to printer defects such as ink spread – Dispersed dot screening has higher spatial resolution – Blue noise masking uses large array of thresholds UT Center for Perceptual Systems 5

Grayscale Error Diffusion • Shape quantization noise into high frequencies • Design of error filter key to quality • Not a screening technique difference threshold current pixel u(m) x(m) + Error Diffusion b(m) _ _ 7/16 + 3/16 5/16 1/16 e(m) shape error UT Center for Perceptual Systems compute error 2 -D sigma-delta modulation weights Spectrum 6

Simple Noise Shaping Example • Two-bit output device and four-bit input words – Going from 4 bits down to 2 increases noise by ~ 12 d. B – Shaping eliminates noise at DC at expense of increased noise at high frequency. Input words 4 2 2 2 To output device Average output = 1/4(10+10+10+11)=1001 4 -bit resolution at DC! Added noise 1 sample delay Assume input = 1001 constant Periodic Time 1 2 3 4 UT Center for Perceptual Systems Input Feedback Sum Output 1001 00 1001 10 1001 01 1010 10 1001 10 1011 10 1001 11 1100 10 12 d. B (2 bits) f If signal is in this band, you are better off 7

Modeling Grayscale Error Diffusion • Sharpening caused by a correlated error image [Knox, 1992] Floyd. Steinberg Jarvis UT Center for Perceptual Systems Error images Halftones 8

Modeling Grayscale Error Diffusion • Apply sigma-delta modulation analysis to 2 -D – Linear gain model for quantizer in 1 -D [Ardalan and Paulos, 1988] – Linear gain model for grayscale image [Kite, Evans, Bovik, 2000] – Signal transfer function (STF), noise transfer function (NTF) Ks us(m) u(m) b(m) Q(. ) { un(m) UT Center for Perceptual Systems Ks n(m) + Signal Path Kn un(m) + n(m) Noise Path 1 – H(z) is highpass so H(z) is lowpass 9

Problems with Error Diffusion • Objectionable artifacts – Scan order affects results – “Worminess” visible in constant graylevel areas • Image sharpening – Error filters due to [Jarvis, Judice & Ninke, 1976] and [Stucki, 1980] reduce worminess and sharpen edges – Sharpening not always desirable: may be adjustable by prefiltering based on linear gain model [Kite, Evans, Bovik, 2000] • Computational complexity – Larger error filters require more operations per pixel – Push towards simple schemes for fast printing UT Center for Perceptual Systems 10

Direct Binary Search • Minimize mean-squared error between lowpass filtered versions of grayscale and halftone images – Lowpass filter is based a linear model of the human visual system (contrast sensitivity function) – Iterative method that gives a practical upper bound on the achievable quality for a halftone of an original image • Each iteration visits every pixel – At each pixel, consider changing state of the pixel (toggle) or swapping it with each of its 8 nearest neighbors that differ in state from it – Terminate when if no pixels are changed in an iteration UT Center for Perceptual Systems 11

Direct Binary Search • Advantages – Significantly improved image quality over screening and error diffusion methods – Quality of final solution is relatively insensitive to the choice of starting point (initial halftone) – Has application in off-line design of threshold arrays for screening methods UT Center for Perceptual Systems • Disadvantages – Computational cost and memory usage is very high in comparison to error diffusion and screening methods – Increase complexity makes it unsuitable for real-time applications such as printing 12

Contrast Sensitivity Function • Contrast needed at a particular spatial frequency for visibility – Angular dependence modeled with a cosine function – Modify it at low frequency to be lowpass, which gives better correlation with psychovisual results – Useful in image quality metrics for optimization and performance evaluation UT Center for Perceptual Systems 13

Color Halftoning by Error Diffusion for Display • Input image has a vector of values (e. g. Red-Green -Blue) at each pixel – Error filter has matrix-valued coefficients – Algorithm for adapting matrix coefficients [Akarun, Yardimci, Cetin 1997] difference threshold u(m) x(m) + b(m) _ t(m) _ + e(m) shape error UT Center for Perceptual Systems compute error 14

Matrix Gain Model for the Quantizer • Replace scalar gain with a matrix u(m) quantizer input b(m) quantizer output – Noise uncorrelated with signal component of quantizer input – Convolution becomes matrix–vector multiplication in frequency domain Noise component of output Signal component of output UT Center for Perceptual Systems 15

Optimum Color Noise Shaping • Vector color error diffusion halftone model – We use the matrix gain model [Damera-Venkata and Evans, 2001] – Predicts signal frequency distortion – Predicts shaped color halftone noise • Visibility of halftone noise depends on – Model predicting noise shaping – Human visual system model (assume linear shift-invariant) • Formulation of design problem – Given HVS model and matrix gain model find the color error filter that minimizes average visible noise power subject to certain diffusion constraints UT Center for Perceptual Systems 16

Linear Color Vision Model • Pattern-Color separable model [Poirson and Wandell, 1993] – Forms the basis for S-CIELab [Zhang and Wandell, 1996] – Pixel-based color transformation B-W R-G B-Y UT Center for Perceptual Systems Opponent Spatial representation filtering 17

Linear Color Vision Model • Undo gamma correction on RGB image • Color separation – – – Measure power spectral distribution of RGB phosphor excitations Measure absorption rates of long, medium, short (LMS) cones Device dependent transformation C from RGB to LMS space Transform LMS to opponent representation using O Color separation may be expressed as T = OC • Spatial filtering is incorporated using matrix filter • Linear color vision model where UT Center for Perceptual Systems is a diagonal matrix 18

Sample Images and optimum coefficients for s. RGB monitor available at: http: //signal. ece. utexas. edu/~damera/col-vec. html Original Image UT Center for Perceptual Systems 19

Floyd-Steinberg UT Center for Perceptual Systems Optimum Filter 20

Conclusions • Design of “optimal” color noise shaping filters – We use the matrix gain model [Damera-Venkata and Evans, 2001] • Predicts shaped color halftone noise – HVS could be modeled as a general LSI system – Solve for best error filter that minimizes visually weighted average color halftone noise energy • Future work – Above optimal solution does not guarantee “optimal” dot distributions • Tone dependent error filters for optimal dot distributions – Improve numerical stability of descent procedure UT Center for Perceptual Systems 21

Backup Slides Designing the Error Filter • Eliminate linear distortion filtering before error diffusion • Optimize error filter h(m) for noise shaping Subject to diffusion constraints where linear model of human visual system matrix-valued convolution UT Center for Perceptual Systems 22

Backup Slides Generalized Optimum Solution • Differentiate scalar objective function for visual noise shaping w/r to matrix-valued coefficients • Write norm as trace and differentiate trace using identities from linear algebra UT Center for Perceptual Systems 23

Backup Slides Generalized Optimum Solution (cont. ) • Differentiating and using linearity of expectation operator give a generalization of the Yule-Walker equations where • Assuming white noise injection • Solve using gradient descent with projection onto constraint set UT Center for Perceptual Systems 24

Backup Slides Implementation of Vector Color Error Diffusion Hgr Hgg + Hgb UT Center for Perceptual Systems 25