Video As A Datatype Ketan MayerPatel Comp 294

Video As A Datatype Ketan Mayer-Patel Comp 294 -9 : : Fall 2003

The mothers of all video. . . • • NTSC - American standard PAL - European standard SECAM - French standard (ugly stepchild) For better or worse, this is where it all starts. Comp 294 -9 : : Fall 2003

Color Spaces • One luminance component • Two chrominance components – Original TV was black and white. – Adding color had to be done in a compatible way. • NTSC: YIQ • PAL: YUV • In general: YUV and YCr. Cb used as terms. Comp 294 -9 : : Fall 2003

YUV vs. RGB • Simple linear transform from one cube to the other. • Specific transform coefficients vary for specific systems, but a common one: – Y = 0. 299 R + 0. 587 G + 0. 114 B – Cr = -0. 147 R - 0. 289 G + 0. 437 B + 0. 5 – Cb = 0. 615 R - 0. 515 G - 0. 100 B + 0. 5 Comp 294 -9 : : Fall 2003

YUV Challenges • YUV is like taking RGB cube, standing it on a corner with (0, 0, 0) on bottom and (1, 1, 1) at top, rotating slightly, and taking the bounding box. • What problems might this incur? – Some RGB colors are illegal. – A lot of YUV colors are illegal. Comp 294 -9 : : Fall 2003

A Note On Color • Previous view of color is EXTREMELY simplified. – Color is complicated • Frequency dependent response for contrast, lightness, etc. – Gamma correction. ` Comp 294 -9 : : Fall 2003

Scanning • Analog video signal is continuous voltage signal that gets scanned along the screen. • The electron stream controlled by two orthogonal sets of magnets. – Horizontal: Beam is moved from left to right and then quickly back. – Vertically: Beam is moved from top to bottom and then quickly back. Comp 294 -9 : : Fall 2003

Voltage Scanning illustrated Time Comp 294 -9 : : Fall 2003

Structure of Video • Blinn’s article. – Macrostructure • Frequency peaks at frame rate. – Microstructure • Frequency peaks at line rate • Adding color is a good first example of sourceaware channel coding. – – What was the problem? What was the solution? Why does it work? What are its drawbacks (i. e. , when does it fail)? Comp 294 -9 : : Fall 2003

Interlacing • Progressive = every scan line done in turn. • Interlaced = every other scan line done. – Creates two fields: odd field and even field. – NTSC: 262. 5 lines per field at 60 fields per sec. – PAL: 312. 5 lines per field at 50 fields per sec. • Fields are separated in time. Comp 294 -9 : : Fall 2003

Analog Bandwidths • True meaning of the bandwidth. • Y, U, and V signals are all continuous along a scan line. – A bit of a hybrid signal: discrete vertically, continuous horizontally. • NTSC: – Y is 4. 2 MHz wide, I is 2 MHz, Q is 1 MHz • PAL: – Y is 6 MHz wide, U is 3 MHz, V is 3 MHz Comp 294 -9 : : Fall 2003

Corresponding Data Rates • How much data can you put in 6 MHz band? – Depends on S/N ratio. – Depends on modulation scheme. • Typical: 27 - 36 Mbs • How many cable channels do you get? – If 50, then 1. 3 - 1. 8 Gbs coming into your home. • The real question for multimedia is: – Why haven’t we found a better use for 1. 3 Gbs than continuous broadcast of Real World. Comp 294 -9 : : Fall 2003

Digital Video Frames • • Almost always progressive 3 planes of pixel values (Y, U, and V) Pixel depth Geometry of each plane: width x height – Chrominance is generally subsampled. • How the planes relate to each other. • Frame rate. Comp 294 -9 : : Fall 2003

CCIR-601 • Standard established for digitizing NTSC and PAL signals. Comp 294 -9 : : Fall 2003

Why 8 -bits for chrominance? • What’s another way to cut chrominance bandwidth in half? – Use 4 -bits per pixel. • Why won’t that work? – Need the dynamic range for color. • But what about when Y is either small or large? – Don’t need the range, but lots of YUV combinations that won’t ever be used. Comp 294 -9 : : Fall 2003

4: 2: 2 • For every 4 luminance samples, take 2 chrominance samples from odd lines and 2 from even lines. • Chrom. planes just as tall, half as wide. • JPEG does this. Comp 294 -9 : : Fall 2003

4: 2: 0 • 2 chrominance samples for every 4 luminance samples, odd lines only. • Chrominance halved in both directions. • MPEG generally does this. Comp 294 -9 : : Fall 2003

4: 1: 1 • What should this be? – 1 chrominance for every 4 luminance for both odd and even lines. – And that is what it is. • But sometimes used to refer to this: Comp 294 -9 : : Fall 2003

De-interlacing • Since analog video fields are separated in time by 1/2 the frame rate, at least half of a digital frame is missing no matter where you sample from. • Normal solution: linear interpolation of even fields to produce matching samples for odd fields. • Even better: linear interpolation of both fields to produce matching samples and thus digital frame rate will equal field rate. Comp 294 -9 : : Fall 2003

Film Frame Rates • Film is a different beast altogether. – Continuous both vertically and horizontally – Projection is simultaneous for all parts of the picture. – Expensive medium. – Combination of all of this motivates 24 fps. • Film to digital is easier than video to digital. – No interlacing, sample where you want to. • Film to video is harder. Comp 294 -9 : : Fall 2003

3: 2 Pulldown • Converts 24 frames to 60 fields. 1 2 3 4 5 1 e 1 o 2 e 2 o 3 e 3 o 4 e 4 o 5 e 5 o Comp 294 -9 : : Fall 2003

Overall sampling lesson • Can’t recover what you don’t have. • Conversion between representations requires estimation of missing samples. • Interpolation causes errors: – Spatially: at the edges. – Temporally: when moving. Comp 294 -9 : : Fall 2003

Common Digital Video Sizes • • • CCIR-601 720 x 480 4: 2: 2, 4: 2: 0 SIF 360 x 240 4: 2: 0 CIF 360 x 288 4: 2: 0 4: 3 HDTV 1440 x 1152 4: 2: 2, 4: 2: 0 9: 16 HDTV 1920 x 1152 4: 2: 2, 4: 2: 0 4 CIF, 16 CIF, QCIF Comp 294 -9 : : Fall 2003

Digital Bitrates • Current television: – 30 fps * 720 * 480 * 1. 5 * 8 = 124 Mb/s • 9: 16 HDTV – 30 fps * 1920 * 1152 * 1. 5 * 8 = 796 Mb/s • This motivates compression. Comp 294 -9 : : Fall 2003