An Introduction to Digital Camera Signal Processor Presented




































- Slides: 36

An Introduction to Digital Camera Signal Processor Presented by Pu-Hua Mei email: p [email protected] ntu. edu. tw mobile: 0920756220

A Typical Image Pipeline for Digital Camera Auto Focus Statistic Raw data Optical Black YUV data White Balance Auto Exposure Auto White Balance Statistic Bad Pixel Correction Color Interpolation Color Space Conversion Gamma Correction Color Correction Edge Enhancement YUV Space Processing Compression RGB data

Image Pipeline Overview n n n From image sensor raw data to final image Targeted at matching human perception Pipeline approach Linear or nonlinear Color depth consideration Calibration, Compensation, Correction, and Concealment

Image Pipeline Step by Step n n n Each step has its purposes. Instead of describing the techniques for each process, we like to focus on their general approaches and their meanings on the image pipeline. We will introduce the general why and how.

Image Sensor Output n CCD (Charge-Coupled Device) u Monitor mode (preview): full width * 2 XX subsampled line u Capture mode (still): full frame output u AF mode: faster frame rate with less lines output n CMOS (Complementary Metal-Oxide. Semiconductor) u Monitor mode: Down sampling output by two or . . . u Capture mode: full frame output u Special mode: window output with panning feature

Optical Black Clamping n n n Compensate image sensors’ dark signal Subtract OB (Optical Black) from pixel signal OB value u Computed by DSP from image sensor’s OB area u Manually set by firmware

Tone Correction n To linearize image sensor’s response curve u Foveon X 3 has linear log response curve. u Analog video source is nonlinear. n n Linearization should be done before further processing A nonlinear example: u TI CCD sensor

White Balance n n n To simulate human eyes white balance Adjustable channel gain for each color channel General approaches Gray world assumption u Perfect reflector assumption u Calibration based approaches u n What if data are nonlinear?

Bad Pixel Correction n n Non-perfect image sensors More than what the spec. claims u Judgment n n standards are different! Must be done in raw data space to prevent bad pixels from polluting neighborhood Considering edge and gradient information

Noise Reduction n What is noise? u Unnatural artifacts: power, readout, flicker. . . u Too many possible noise sources u Focus on removing noise sources first n Should we reduce noise in raw or YUV space? u Raw data space F Prevent noise from going into remaining steps of image pipeline and being magnified further u YUV space F More information to reduce noise correctly.

Color Interpolation n n Also called de-mosaic / raw 2 rgb. . . Guess missing channels for each pixel by the following: u Neighbor pixels u Edge u Gradient u. . . n Avoid zigzag and false color artifacts

Color and Gamma Correction n n Image sensor’s color sensitivity is different from human eyes. A 3 x 3 matrix multiplication and a nonlinear gamma mapping are used to correct color to match TRUE color. However, there is no TRUE color. Color target is used to replace TRUE color. Correction means solving best matrix and gamma. s. RGB assumes 2. 2 gamma correction but. . . u No one really cares. u Perceptive correctness is the truth.

Influence of Color and Gamma Matrix Gamma

Color Space Conversion n RGB <-> YUV Prepare for brightness/contrast/hue/saturation adjustment and JPEG (Joint Photographic Experts Group) compression Typically done by 3 x 3 matrix multiplication

Tone Mapping n n n Map tone curve to get better image Similar to histogram adjustment or Photoshop’s curve function For Y channel only

Edge Enhancement n n A must - all cameras add edges General approaches u Edge filter: Nx. N, 1 x. N+Nx 1 u Edge gain control u Edge detection module n Noise should not be enhanced

Edge Enhancement n Normal and strong edge enhancement

Resizing and Cropping n Preview display u Sensor n Video capture u Sensor n n lines number doesn't match with output. Digital zoom u Crop n lines number doesn't match with LCD. smaller area centered at original center Raw data space or YUV space Performance vs. quality u Dropping or duplication, bilinear interpolation, bicubic interpolation

AE / AF / AWB (I) n Closed loop controls for passive 3 A (active 3 A are totally different) u Data collecting + control mechanism + converging algorithm n AE (auto exposure) u Collect luminance before gamma correction u Control exposure time, analog gain, iris, . . . u Converge luminance to AE target

AE / AF / AWB (II) n AWB (auto white balance) u Collect color statistic after white balancing u Control color gain u Converge color average to white target n AF (auto focus) u Collect focus value before edge enhancement u Control image plane position via AF motor u Find position with maximum focus value

More Processing (I) n To avoid false color u Color clamping u False color suppression u Color noise reduction n n Lens shading correction Lens distortion correction

More Processing (II) n Skin-tone detection

More Processing (III) n Digital flash (local contrast normalization) Improve the contrast on dark area like a flash light u http: //www. ukapical. com/products_DSC_inter. htm u

Image Details Degradation n Image process harms image details u Increasing noise u Accumulating rounding error u Overflow and underflow n Less processing is better

Image Pipeline Noise (8 -bit, =0) White Balance Matrix Gamma

Image Pipeline Noise (8 -bit, =2) White Balance Matrix Gamma

Image Pipeline Noise (8 -bit, =4) White Balance Matrix Gamma

Performance & Resource Consideration n n Memory buffer size DRAM bandwidth Computing complexity Pipelining: overlapping operation

Image Pipeline Fine-tuning n n n From beginning of pipeline Step by step Back and forth

Zoran COACH 6

Nu. CORE Sip-1270 n Work with their AFE NDX-1260

TI TMS 320 DM 270

DM 270 CCD Controller n n n Digital clamping Black level compensation Median filter Gain and offset Output formatter

DM 270 Preview Engine

DM 270 AF/AE/AWB Statistics Engine n AE / AWB u Up to 192 windows u Configurable rows and columns number u Configurable window size and position u Output R, Gr, Gb, B accumulation n AF u Up to 36 windows u Configurable window size and position u Output G accumulation and focus value (result of two filters for green pixels)

Reference n n n n Nu. CORE Technology Inc. , Sip-1270 Clean. Capture. TM Image Processor, 2003. Nu. CORE Technology Inc. , The Clean. Capture. TM NDX-1260 Analog Image Processor - A No-Compromise Approach to Image Quality and Performance, 2003. Nu. CORE Technology Inc. , Sip-1270 Clean. Capture. TM Image Processor - A No-Compromise Approach to Image Quality and Performance, 2003. Texas Instruments, TMS 320 DM 270 System Spec, 2002. Texas Instruments, TMS 320 DM 270 CPU and Peripherals - Technical Reference Manual Version 1. 0, 2003. Texas Instruments, TC 255 P 336 -x 244 -PIXEL CCD IMAGE SENSOR, 2002. Zoran Corporation, COACHTM 6 Digital Camera Processor, 2003.
Asic form 5603
Digital signal processor architecture
Digital signal processor architecture
Digital signal as a composite analog signal
Signal encoding schemes
Advantages of multi camera production
Camera movement
Baseband signal and bandpass signal
Baseband signal and bandpass signal
What is the product of an even signal and odd signal
Digital transparent processor
Pentium processor architecture
Digital camera electronics
Is a digital camera input or output
Camera parts and functions
Video camera basics
Digital camera embedded system
Digital camera processors
Advantages of digital camera
Science fair quotes
Digital camera embedded system
Input-process-output examples in real life
ฮาร์ดดิสก์
Mobile camera parts and functions
Video camera functions
Embedded systems design
Analog vs digital
Digital signal 3
For digital
Digital signal processing
Types of signals
Jatiya kabi kazi nazrul islam university logo
Conversion of continuous awgn channel to vector channel
Digital signal processing
"interlaced"
Digital signal processing
Digital signal processing