ISP Introduction 2007 10 26 Purpose The objective

ISP Introduction 2007 -10 -26

Purpose • The objective of this presentation is to describe the image processing algorithms of Digital Camera. Most of the algorithms along the data path will be described. • The intended audiences include algorithm designers, software programmers, IC engineers, application engineers, and project managers as well.

ISP functional block diagram

Black level with G 1/G 2 filter • This function allows user to adjust the black level so that blacks appear as true blacks. • Formula: • • R’ = R + BL_R_OFFSET G 1’ = G 1 + BL_G 1_OFFSET + α*R G 2’ = G 2 + BL_G 2_OFFSET + β*B B’ = B + BL_B_OFFSET • where BL_R_OFFSET, BL_G 1_OFFSET, BL_G 2_OFFSET, BL_B_OFFSET, α, β are configurable parameters.

DPD/DPC and Noise Reduction • The image from sensor may have noise and dead pixels. This module can do noise cancellation and DPC at the same time. diff 40 = abs(p 4 -p 0); diff 42 = abs(p 4 -p 2); diff 45 = abs(p 4 -p 5); diff 47 = abs(p 4 -p 7); diff 41= abs(p 4 -p 1); diff 43 = abs(p 4 -p 3); diff 46 = abs(p 4 -p 6); diff 48 = abs(p 4 -p 8); if((diff 40>dpc_th) && (diff 41>dpc_th) && (diff 42>dpc_th) && (diff 43>dpc_th) && (diff 44>dpc_th) && (diff 45>dpc_th) && (diff 46>dpc_th) && (diff 47>dpc_th)) Is_dead = 1; else is_dead = 0;

If it is a dead pixel, do dead pixel cancellation (DPC) DV = abs(2*p 4 -p 1 -p 7); DH = abs(2*p 4 -p 3 - DDL = abs(2*p 4 -p 0 -p 8); DDR = abs(2*p 4 -p 2 - p 5); p 6); if((DV<=DH)&&(DV<=DDL)&&(DV<=DDR)) new = (pix 1+pix 7+1)/2; else if((DH<DV)&&(DH<=DDL)&&(DH<=DDR)) new = (pix 3+pix 5+1)/2; else if((DDL<DH)&&(DDL<DV)&&(DDL<=DDR)) new = (pix 0+pix 8+1)/2; else new = (pix 2+pix 6+1)/2;

If current pixel is not dead pixel, do noise cleaning DV 2 = abs(2*p 4 -p 1 -p 7); DH 2 = abs(2*p 4 -p 3 -p 5); DDL 2 = abs(2*p 4 -p 0 -p 8); DDR 2 = abs(2*p 4 -p 2 -p 6); if ((DV 2<=DH 2)&&(DV 2<=DDL 2)&&(DV 2<=DDR 2)) avg = (p 1+p 4+p 7) / 3; var = abs(p 1 -avg) + abs(p 4 -avg) + abs(p 7 -avg); else if((DH 2<DV 2)&&(DH 2<=DDL 2)&&(DH 2<=DDR 2)) avg = (p 3+p 4+p 5) / 3; var = abs(p 3 -avg) + abs(p 4 -avg) + abs(p 5 -avg); else if((DDL 2<DV 2)&&(DDL 2<DH 2)&&(DDL 2<=DDR 2)) avg = (p 0+p 4+p 8) / 3; var = abs(p 0 -avg) + abs(p 4 -avg) + abs(p 8 -avg); else avg = (p 2+p 4+p 6) / 3; var = abs(p 2 -avg) + abs(p 4 -avg) + abs(p 6 -avg); if ( it is noise ) output = avg ; //? else output = p 4 ;

Digital Gain There are two types of digital gain: global digital gain and R/G/B digital gain. The digital gain control is implemented on the input RGB Bayer pattern stream. All the pixel data (R, G and B) are multiplied by a digital global gain. The higher the global gain is, the brighter the image is. The global gain is used to adjust image brightness in Auto Exposure (AE) function. Different from digital global gain, digital RGB gains affect RGB pixel separately. Different color channel uses different gain value, so there are three gain values for R, G and B pixels, respectively. Digital RGB gains are used in Auto White Balance (AWB) function. Implementation: R’ = R * R_gain * Global_gain; G’ = G * G_gain * Global_gain; B’ = B * B_gain * Global_gain;

Lens Shading Correction • This module is used to compensate for the shading effect, which causes the image intensity to get darker on the outer range of the image. We will apply a different gain value for each different (x, y) position in the image. The algorithm works on Bayer image, and works separately on R/G/B channel.

CFA Interpolation In Bayer format, each pixel has only one of R/G/B values. By CFA interpolation, each pixel will have complete R/G/B values. Interpolation of G at R/B location if (DH < DV) else Interpolation of G at B location can be done on the corresponding formula of B values.

CFA Interpolation of G at R/B location Interpolation of G at B location can be done on the corresponding formula of B values.

CFA Interpolation of R at B location Interpolation of B at R location can be done on the corresponding formula of B values.

Color Correction • Color correction is done using a color matrix and it compensates color deviation due to color filtering and sensing circuits.

RGB Gamma Correction • The transfer function of most CRT displays produces an intensity that is proportional to some power (referred to as gamma) of the signal amplitude. As a result, high-intensity ranges are expanded and low-intensity ranges are compressed. ITU-R BT. 709 SMPTE 240 M s. RGB γ=0. 45 γ=1/2. 4 y=4. 5 x x ≤ 0. 018 y=4. 5 x x ≤ 0. 0228 y=12. 92 x x ≤ 0. 0031308 y=1. 1115 xγ 0. 1115 x > 0. 018 y=1. 099 xγ 0. 199 x > 0. 0228 y=1. 055 xγ 0. 055 x > 0. 0031308

RGB Gamma Correction

RGB to YCb. Cr The conversion uses this formula: Formula 3: RGB range: 0 -255; YCb. Cr range: 0 -255 Floating-point: Fixed-point:

Edge Extraction and Sharpness Edge extraction is done on Y channel, and will be used for image sharpening. To calculate one pixel, a 3 x 5 filter is used.

Edge Extraction and Sharpness The edge map (EM) is further modified through a lookup table (EMLUT). EMLUT is constructed as following: The modified edge map is then added back to the Y channel. That is, Y’ = Y + EMLUT(x).

Edge Extraction and Sharpness When –x 1 < x ≤ x 1, we think x is the noise, not the edge. There are two methods to calculate the Y value in this range: one is Y = Y – m 2 x, the other is Y = Avg. DDH = abs(2*Ym, n – Ym, n-1 – Ym, n+1) ; DDV = abs(2*Ym, n – Ym-1, n – Ym+1, n) ; DDL = abs(2*Ym, n – Ym-1, n-1 – Ym+1, n+1) ; DDR = abs(2*Ym, n – Ym-1, n+1 – Ym+1, n-1) ; Find the minimum of then Avg takes the value of that direction.

Edge Extraction and Sharpness To avoid over-shoot on edges, The Y’ is further modified as the following: Find the minimum of the 15 Y’s, denoted by Ymin, and the maximum of the 15 Y’s, denoted by Ymax. if ( Y’ >= Ymax ) Y_out = Ym, n + ( (Y’ – Ymax) >> clip_bits ) else if ( Y’ < Ymin ) Y_out = Ym, n + ( (Y’ – Ymin) >> clip_bits )

False Color Suppression False color suppression is done on U, V channel. Y is not changed, U, V may be changed by false color suppression. Implementation: 1. get YUV 422 data; 2. get edge on Y channel; 3. edge = abs(edge); 4. get temp by clipping the edge into [edge_min, edge_max]; Where edge_min and edge_max are userprogrammable registers.

False Color Suppression 5. calculate u_gain and v_gain: u_gain = K_edge*(edge_max - temp); v_gain = K_edge*(edge_max - temp); u_gain v_gain 1 where K_edge=65536/(edge_max - edge_min) ; 6. do false color suppression on U, V channel when edge is larger than edge_min: 0 edge_min edge_max edge temp_u=(u_gain*(temp_u-128))/65536+128; temp_v=(v_gain*(temp_v-128))/65536+128; if edge is less than or equal to edge_min, just output u, v directly. 7. clip u, v into [0, 255] and output.

Backlighting Detection and Compensation 1. Detection The shape of histogram is very special in backlight condition. Two peaks lie on the two sides of the histogram: one on the low light side, and the other on the high light side. According to this fact, we may detect the backlight condition. To detect Backlighting, we need histogram information , the mean brightness of the whole image (maybe weighted), i. Win. Ymean and the mean brightness of 16 windows Win. Ymean[0] ~ Win. Ymean[15];

Backlighting Detection and Compensation 1. Detection When the following adaptive condition is satisfied, it is regarded as in backlight condition: if(int_Sum_White_Rate*(i. Thres. Hold+int_Sum_Black_Rate)*20>=(Bac k. Light_White_Thres*i. Thres. Hold)&& i. Win. Ymean. Min<Back. Light_Win. Min_Thres) m_i. Back. Light = TRUE;

Backlighting Detection and Compensation 2. Compensation When the backlight is detected, and the following adaptive condition is satisfied, it is regarded as needn’t add ET time or sensor gain or digital gain or all: if(int_Sum_Black_Rate*(i. Thres. Hold+int_Sum_White_Rate)*32<(Co mpensate_Black_Thres*i. Thres. Hold)|| i. Win. Ymean>Compensate_Win. Mean_Thres|| int_Sum_White_Rate>((i. Thres. Hold*Compensate_White_Thres)>>4)| | i. Win. Ymean. Min>Compensate_Win. Mean_Thres) m_i. Bk. LNeed. Add. ET = FALSE;

Brightness Input: Y, U, V in the range of [0, 255], as source image BRIGHTNESS, in the range of [-128, 127], as brightness adjustment Output: Y, U, V in the range of [0, 255] Implementation: define CLIP 255(x) ((x>255)? 255: ((x<0)? 0: x)) Y’ = CLIP 255(Y + BRIGHTNESS); U’ = U; V’ = V; r

Contrast Input: Y, U, V in the range of [0, 255], as source image Output: Y’, U’, V’, in the range of [0, 255] Parameters: CONTRAST, in the range of [1, 63] Y_OFFSET, in the range of [-128, 127] Implementation: define CLIP 255(x) ((x>255)? 255: ((x<0)? 0: x)) Y’ = CLIP 255(((Y*CONTRAST)>>5) + Y_OFFSET); U’ = CLIP 255((((U-128)*CONTRAST)>>5)+128); V’ = CLIP 255((((V-128)*CONTRAST)>>5)+128);

Hue Input: YUV 422 data, Y, U and V are in the range of [0, 255]. Register Hue_sin and Hue_cos are in the range of [-256, 256] Hue angle, within the range of [-45, 45] for case 2. Output: YUV 422 data, Y, U and V are in the range of [0, 255]. Implementation: If U and V fall out of [0, 255], clipping operation is applied as following: Clipping(x) = (x<0) ? 0 : ( (x>255)? 255: x )

Saturation Input: YUV 422 data, Y, U and V are in the range of [0, 255]. Output: YUV 422 data, Y, U and V are in the range of [0, 255]. Parameter: Saturation ratio K, within the range of [0, 2]. Formula: If U and V falls out of [0, 255], clipping operation is applied as following: Clipping(x) = (x<0) ? 0 : ( (x>255)? 255: x )

Brightness, Contrast, Hue and Saturation Control

Auto Expose The luminance of the image changes when the environment varies. In order to keep the luminance in the given range, we must adjust exposure time (ET) of the sensor. If the ET is small, changing ET by one unit (one flickering period) can make the brightness of image change much. In order to make the brightness of image change smoothly, digital global gain is combined with ET adjustment

Auto Expose An image is divided into 16 windows with different weights. Ymean of the frame and Ymean of the 16 windows are calculated as described below. :

Auto White Balance When the white paper moves from one light source to another light source, an image sensor sees different colors under different conditions. Similarly, when a digital camera is moved from outdoors (sunlight) to indoor fluorescent (or incandescent) light conditions, the color in the image shifts. To correct for light source color-temperature changes, the balance among red, green and blue has to be shifted. This “white balancing” is performed by the algorithm below.

Auto White Balance The number (Counter) of the pixels which satisfy the following conditions and their sums (Rsum, Gsum and Bsum)) of red, green and blue are calculated along with image data stream. Ybot. Reg<Y<Ytop. Reg Abs(I)<Itop. Reg Abs(Q)<Qtop. Reg To deal with the issue of big region of uniform color, we will add the following judgment if (Counter>=TH_GRAY_COUNT) //there is enough valid gray pixels To update RGain, BGain to adjust AWB else //there is not enough gray pixels Keep the Rgain, BGain of the last frame.

Auto White Balance The average value (Awb. Rmean, Awb. Gmean and Awb. Bmean) of the tricolor is calculated by the following formulas: Awb. Rmean=Rsum/Counter Awb. Gmean=Gsum/Counter Awb. Bmean=Bsum/Counter RDiff = abs( Awb. Rmean-Awb. Gmean ) BDiff = abs( Awb. Bmean-Awb. Gmean ) Based on these differences, Rgain and Bgain will be updated. AWB range control register (AWB threshold 1, AWB threshold 2, AWB gain step) will control this operation.

Auto White Balance 0 AWB threshold 1 AWB threshold 2 Rstep = 0 Rstep = 1 Rstep = Awb. Step Rgain. Reg – Rstep ( Awb. Rmean>Awb. Gmean) Rgain. Reg + Rstep ( Awb. Rmean<Awb. Gmean) Bgain. Reg – Bstep ( Awb. Bmean>Awb. Gmean) Bgain. Reg + Bstep ( Awb. Bmean<Awb. Gmean) Rgain. New = Bgain. New =

Advance Auto White Balance Data path Stage 1: The task is to adjust the R/G/B balance to an “acceptable” degree. Stage 2: The task is to further fine-tune the R/G/B balance to make it better.

Advance Auto White Balance 1. Stage 1 of the AAWB a) For each frame, IC will collect the statistics of one light source, i. e. , the number of pixels falling into the region of that light source model, denoted by p_data[i] b) p_light_weight[i] = p_data[i]>>COUNTER_SHIFT; b) After every 5 frames, IC will send an interrupt to firmware, saying 5 statistics are ready. With these 5 new statistics, firmware will perform the following smoothing operations, to reduce the unpleasing effect caused by erroneous statistics: for( i=0; i<SOURCE_LIGHT_NUMBER; i++) weighted[i]=(cur_weight*p_light_weight[i] +(8 -cur_weight)*short_pre_weights[i])/8; for(i=0; i<SOURCE_LIGHT_NUMBER; i++) short_pre_weights[i]=weighted[i];

Advance Auto White Balance 1. Stage 1 of the AAWB c) Firmware will apply the smoothed weights on the pre-defined pre_gr_gain and pre_gb_gain to get the weighted gain values: gr_gain and gb_gain. As a by-product, we can also know the light source corresponding to the maximum weight. d) Firmware will combine the weighted gains into color matrix, and write the new color matrix values into IC registers. 6 multiplications will be involved.

Advance Auto White Balance 2. Stage 2 of the AAWB a) Calculate Rmean, Gmean, Bmean of the output from stage 1, satisfying both of the following conditions: rr<=TH_RGB && gg<=TH_RGB && bb<=TH_RGB Y>=Ybot. Reg && Y<=Ytop. Reg && abs(I)<Itop. Reg && abs(Q)<Qtop. Reg At the same time, count the number of pixels satisfying both of the conditions. b) Calculate gain_gr, gain_gb: gain_gr=64*Gmean/Rmean, gain_gb=64*Gmean/Bmean. c) If the count in step a) exceeds a threshold TH_GRAY_COUNT, it means there are enough gray pixels in the image, thus gain_gr and gain_gb are reliable statistics. In this case, we will further smooth them with previous values: pre_gr_gain = (pre_gr_gain*3 + gain_gr + 2)/4; pre_gb_gain = (pre_gb_gain*3 + gain_gb + 2)/4;

Advance Auto White Balance 3. How to get the Statistics Gray model under 4 Light sources 545 540 535 A 530 D 65 525 CWF 520 515 510 505 Indoor 500 510 520 530 540 550 560 570 580 590 600 610 a) Get the luminance Y from RGB: Y = ( 77*r + 150*g + 29*b ) / 256; For those pixels satisfying (Y≥Ybot. Reg) && (Y≤Ytop. Reg) : b) Transform RGB color space to log(g/b), log(r/b) space with an LUT: horizontal axis is log(g/b), and vertical axis is log(r/b); x = log(g) – log(b); y = log(r) – log(b); c) Rotate x_y space by 45 degree. y’=y-x; x’=y+x;

Other ISP Models Anti Flicker Motion Detection Temporal Noise Reduction Dither or Halftone Auto Focus …………

Questions and Thanks