Media Compression NUS SOC CS 5248 2010 Roger

  • Slides: 64
Download presentation
Media Compression NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides

Media Compression NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

You are Here Encoder Decoder Middlebox Receiver Sender Network NUS. SOC. CS 5248 -2010

You are Here Encoder Decoder Middlebox Receiver Sender Network NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Why compress? n “Bandwidth Not Enough” n “Disk Space Not Enough” n Size of

Why compress? n “Bandwidth Not Enough” n “Disk Space Not Enough” n Size of Uncompressed DVD Movie = NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Why compress? n “Bandwidth Not Enough” n “Disk Space Not Enough” n Size of

Why compress? n “Bandwidth Not Enough” n “Disk Space Not Enough” n Size of Uncompressed DVD Movie = (720 x 576) pixels x 3 bytes x 25 fps x 60 sec/min x 120 min = 208. 6 GB n NTSC: 29. 97 fps (30/1. 001); PAL 25 fps NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Optical Disc Formats (1) n CD: ~650 MB n DVD: n 4. 7 (4.

Optical Disc Formats (1) n CD: ~650 MB n DVD: n 4. 7 (4. 38) GB (single layer) n 8. 5 (7. 92) GB (dual layer) n Single and dual sided (up to 18 GB) n 1 X max. read speed: ~10 Mb/s n Video codec: MPEG-2 NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

JPEG Compression NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides

JPEG Compression NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Original Image (1153 KB) 1: 1

Original Image (1153 KB) 1: 1

Original Image (1153 KB) 3. 5: 1

Original Image (1153 KB) 3. 5: 1

Original Image (1153 KB) 17: 1

Original Image (1153 KB) 17: 1

Original Image (1153 KB) 27: 1

Original Image (1153 KB) 27: 1

Original Image (1153 KB) 72: 1

Original Image (1153 KB) 72: 1

Original Image (1153 KB) 192: 1

Original Image (1153 KB) 192: 1

Compression Ratio Quality Size Ratio Raw TIFF 1153 KB 1: 1 Zipped TIFF 982

Compression Ratio Quality Size Ratio Raw TIFF 1153 KB 1: 1 Zipped TIFF 982 KB 1. 2: 1 Q=100 331 KB 3. 5: 1 Q=70 67 KB 17: 1 Q=40 43 KB 27: 1 Q=10 16 KB 72: 1 Q=1 6 KB 192: 1 NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Magic of JPEG n Throw away information we cannot see n Color information n

Magic of JPEG n Throw away information we cannot see n Color information n “High frequency signals” n Rearrange data for good compression n Use standard compression NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Discard color information Y V NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in

Discard color information Y V NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) U

Color Sub-sampling n The subsampling scheme is commonly expressed as a n n three

Color Sub-sampling n The subsampling scheme is commonly expressed as a n n three part ratio (e. g. 4: 2: 2). The parts are (in their respective order): Luma (Y) horizontal sampling reference (originally, as a multiple of 3. 579 MHz in the NTSC television system). Cr (U) horizontal factor (relative to first digit). Cb (V) horizontal factor (relative to first digit), except when zero. Zero indicates that Cb horizontal factor is equal to second digit, and, in addition, both Cr and Cb are subsampled 2: 1 vertically. Zero is chosen for the bandwidth calculation formula to remain correct. To calculate required bandwidth factor relative to 4: 4: 4, one needs to sum all the factors and divide the result by 12. NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Color Sub-sampling 4: 4: 4 4: 2: 0 4: 2: 2 4: 1: 1

Color Sub-sampling 4: 4: 4 4: 2: 0 4: 2: 2 4: 1: 1 NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

4: 2: 2 Sub-sampling Y V NUS. SOC. CS 5248 -2010 Roger Zimmermann (based

4: 2: 2 Sub-sampling Y V NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) U

Original Image (1153 KB) 4: 2: 0

Original Image (1153 KB) 4: 2: 0

Original Image (1153 KB) “ 4: 1: 0”

Original Image (1153 KB) “ 4: 1: 0”

Discrete Cosine Transform Demo NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part

Discrete Cosine Transform Demo NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Quantization DC 242 65 23 5 8 8 8 -54 -10 -4 -2 8

Quantization DC 242 65 23 5 8 8 8 -54 -10 -4 -2 8 8 8 16 13 6 3 5 8 8 16 32 2 1 -1 -2 / 8 8 16 32 64 Quantization Table NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) 30 8 = 2 0 -6 -1 0 0 0 0 0 AC

Differential Coding 30 8 2 0 25 3 1 0 27 3 1 0

Differential Coding 30 8 2 0 25 3 1 0 27 3 1 0 6 -1 0 0 2 1 0 0 0 4 0 1 0 0 0 0 30 8 2 0 -5 3 1 0 2 3 1 0 6 -1 0 0 2 1 0 0 0 4 0 1 0 0 0 0 NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Zig-zag ordering 27 3 1 0 2 1 0 0 4 0 1 0

Zig-zag ordering 27 3 1 0 2 1 0 0 4 0 1 0 0 0 27, 3, 2, 4, 1, 1, 0, 0, 0, 1, 0, 0 NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Run-Length Encoding 27 3 1 0 2 1 0 0 4 0 1 0

Run-Length Encoding 27 3 1 0 2 1 0 0 4 0 1 0 0 0 27, 3, 2, 4, 1, 1, 0, 0, 0, 1, 0, 0 (27, 1) (3, 1) (2, 1), (4, 1), (1, 2), (0, 5), (1, 1), (0, 4 NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Idea: Motion JPEG n Compress every frame in a video as JPEG n DVD-quality

Idea: Motion JPEG n Compress every frame in a video as JPEG n DVD-quality video = 208. 6 GB n Reduction ratio = 27: 1 n Final size = 7. 7 GB NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Video Compression NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides

Video Compression NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Temporal Redundancy NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides

Temporal Redundancy NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Motion Estimation NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides

Motion Estimation NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Bi-directional Prediction NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides

Bi-directional Prediction NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Motion Vectors NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides

Motion Vectors NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

H. 261 P-Frame I-Frame NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part

H. 261 P-Frame I-Frame NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

MPEG-1 B-Frame NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides

MPEG-1 B-Frame NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

MPEG Frame Pattern (1) n HDV GOP example NUS. SOC. CS 5248 -2010 Roger

MPEG Frame Pattern (1) n HDV GOP example NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

MPEG Frame Pattern (2) n Example display sequence: n IBBPBBP … n Example encoding

MPEG Frame Pattern (2) n Example display sequence: n IBBPBBP … n Example encoding sequence: n IPBBPBB NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Compression Ratio Frame Typical Ratio I 10: 1 P 20: 1 B 50: 1

Compression Ratio Frame Typical Ratio I 10: 1 P 20: 1 B 50: 1 NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Sequence sequence header: • width • height • frame rate • bit rate •

Sequence sequence header: • width • height • frame rate • bit rate • : NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

GOP: Group of Picture gop header: • time • : NUS. SOC. CS 5248

GOP: Group of Picture gop header: • time • : NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Picture pic header: • number • type (I, P, B) • : NUS. SOC.

Picture pic header: • number • type (I, P, B) • : NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Picture NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by

Picture NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Slice NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by

Slice NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Macroblock NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by

Macroblock NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Block 1 Macroblock = NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part

Block 1 Macroblock = NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) Y Y U Y Y V

Structure Summary NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides

Structure Summary NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

For I-Frame n Every macroblock is encoded independently (“I-macroblock”) NUS. SOC. CS 5248 -2010

For I-Frame n Every macroblock is encoded independently (“I-macroblock”) NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

For P-Frame n Every macroblock is either n I-macroblock n a motion vector +

For P-Frame n Every macroblock is either n I-macroblock n a motion vector + error terms wrt a prev I/P-frame (“P-macroblock”) NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

For B-Frame n Every macroblock is either n I-macroblock n P-macroblock n a motion

For B-Frame n Every macroblock is either n I-macroblock n P-macroblock n a motion vector + error terms wrt a future I/P-frame n 2 motion vectors + error terms wrt a prev/future I/P-frame NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

MPEG-1/2 File Formats n (Packetized) Elementary streams, ES & PES n Program streams PS

MPEG-1/2 File Formats n (Packetized) Elementary streams, ES & PES n Program streams PS (reliable mediums, e. g. , DVD) n Transport streams TS (for lossy mediums, e. g. , on-air broadcast) Video Source MPEG-2 Elementary Encoder Packetizer MPEG encoded streams Audio Source MPEG-2 Elementary Encoder Data Source NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) PES: *. m 2 v PES: *. m 2 a Packetizer Systems Layer MUX TS: *. ts *. m 2 t *. mpg Transport Stream Packetizer Flow chart © Manish Karir

Review: MPEG structure n ES, PS, TS n Sequence n GOP n Picture n

Review: MPEG structure n ES, PS, TS n Sequence n GOP n Picture n Slice n Macroblock n Block NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

MPEG Decoding (I-Frame) 101000101 Entropy Decoding Dequantize IDCT NUS. SOC. CS 5248 -2010 Roger

MPEG Decoding (I-Frame) 101000101 Entropy Decoding Dequantize IDCT NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

MPEG Decoding (P-Frame) 101000101 Entropy Decoding Dequantize IDCT Prev Frame NUS. SOC. CS 5248

MPEG Decoding (P-Frame) 101000101 Entropy Decoding Dequantize IDCT Prev Frame NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) +

MPEG Decoding (B-Frame) 101000101 Entropy Decoding Future Frame Dequantize IDCT AVG Prev Frame NUS.

MPEG Decoding (B-Frame) 101000101 Entropy Decoding Future Frame Dequantize IDCT AVG Prev Frame NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang) +

There is more… n Half-pel Motion Prediction n Skipped Macroblock n etc. NUS. SOC.

There is more… n Half-pel Motion Prediction n Skipped Macroblock n etc. NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

MPEG in Daily Life MPEG Standards Bit-rate Usage MPEG-1 1. 5 Mbps VCD MPEG-2

MPEG in Daily Life MPEG Standards Bit-rate Usage MPEG-1 1. 5 Mbps VCD MPEG-2 3 -45 Mbps DVD, SVCD, HDTV MPEG-4 Scalable Quick. Time, Div. X, AVCHD NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Camcorders in Daily Life n Different formats used n DV 25 (Mini. DV, DVCAM,

Camcorders in Daily Life n Different formats used n DV 25 (Mini. DV, DVCAM, DVCPRO) n Capacity: 1 hour ~ 13 GB n Speed: 25 Mb/s (user data) n Color sampling: 4: 1: 1 n Compression ratio: ~10: 1 NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Codec Comparison n “M-JPEG” (e. g. , DV) versus “MPEG” Compression Technique “M-JPEG” (I-frames

Codec Comparison n “M-JPEG” (e. g. , DV) versus “MPEG” Compression Technique “M-JPEG” (I-frames only) “MPEG” (Temporal compression) Compression ratio Low (10: 1 to 30: 1) High (>100: 1) Editing (frame-accurate) Easy Difficult Encoding/decoding complexity Symmetric Asymmetric Processing latency Low to Medium High Multi-generation loss Medium High n No “perfect” codec -> application dependent NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

High-Definition n Standard by ATSC n 18 different sub-formats n 720 p and 1080

High-Definition n Standard by ATSC n 18 different sub-formats n 720 p and 1080 i are the most interesting n 1280 x 720 x 60 p, 1920 x 1080 x 60 i (30 p) n 1080 p is non-standard, but available n 1. 4 Gb/s raw bandwidth n 10 – 20 Mb/s compressed (distribution, broadcast) n 100 – 135 Mb/s compressed (pro tapes: DVCPROHD, HDCAM; for editing) NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Consumer HD n HDV: MPEG-2 n 19 (720 p) / 25 Mb/s (1080 i)

Consumer HD n HDV: MPEG-2 n 19 (720 p) / 25 Mb/s (1080 i) n Tape format n http: //www. hdv-info. org n AVCHD: H. 264 n 5 to 25 Mb/s n Hard disk format n http: //www. avchd-info. org/ NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Optical Disc Formats (2) n HD DVD (now dead) n Capacity: 15 GB and

Optical Disc Formats (2) n HD DVD (now dead) n Capacity: 15 GB and 30 GB n 1 X speed: 36 Mb/s n Video codec: VC-1, H. 264, MPEG-2 n Blu-ray n Capacity: 25 GB and 50 GB n 1 X speed: 36 Mb/s n Video codec: VC-1, H. 264, MPEG-2 NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Recent Codec: H. 264 n “Same quality at half the rate” n Encoding complexity:

Recent Codec: H. 264 n “Same quality at half the rate” n Encoding complexity: ~4 X n How: n Variable block size motion compensation n Multiple reference frames n Deblocking filter n. . . n Also called MPEG-4 Part 10 or AVC NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Hands-On n Download source code, compile and play with ffmpeg n mpeg_stat n Video

Hands-On n Download source code, compile and play with ffmpeg n mpeg_stat n Video ‘Surfing_short. m 2 t’ from course web site (98 MB, HDV, transport stream) n n Try different MPEG-1/2 encoding parameter NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)

Impact on Systems Design n n How How : : to to package data

Impact on Systems Design n n How How : : to to package data into packets? deal with packet loss? deal with bursty traffic? predict decoding time? NUS. SOC. CS 5248 -2010 Roger Zimmermann (based in part on slides by Ooi Wei Tsang)