MPEG 2 FGS Implementation ECE 738 Advanced Digital

MPEG 2 FGS Implementation ECE 738 Advanced Digital Image Processing Author: Deshan Yang 05/01/2003

Introduction of FGS l l l FGS = fine granularity scalability For MPEG 2 / MPEG 4 and others Comparing to SNR, temporal, spatial scalability, FGS enhances video quality continuous instead of stepwise. Using bitplane coding for enhancement layer, base layer is still using regular MPEGx coding Enhancement layer bitstream can be truncated. The quality of video stream is proportional to the amount of enhancement bitstream data used.

Project outline l Implement MPEG 2 FGS enhancement into the current MPEG 2 codec project from MPEG MSSG (MPEG Software Simulation Group). l Implementation includes: l • • • Updating the current MPEG 2 codec source code to support FGS Implementing bitplane VL Codec and bit streaming Implementing a new FGS enabled MPEG 2 player Challenges • • Understanding and using the MSSG MPEG 2 codec source code (much more difficult than it seems to be) Bitplane coding / decoding MPEG 2 player on Windows platform Media data to play with

Project outline (cont. )

Source code l MPEG 2 codec source code is from: MPEG Software Simulation Group (MSSG) at http: //www. mpeg. org/MPEG/MSSG/ l l MPEG 2 player source code: searching google. com for mpg 2 w 11 b. zip. My MPEG 2 FGS player is combined from the MPEG 2 player with my MPEG 2 FGS decoder MPEG 2 player source.

MPEG 2 Encoder l l More configurable options to control FGS encoding. Integrating FGS enhancement layer bitplane encoder into the original MPEG 2 encoder • • l Encoding base layer in very low bit rate Calculate the DCT coefficients residues before and after the quantization step. Encoding the DCT coefficients residues with the bitplane VLC Encoded bitplane VLC output data are stored (as binary files) as the enhancement layer bitstream Supporting I-frame and P-frame, not B-frame

MPEG 2 FGS Encoder Architecture

MPEG 2 Decoder l l More configurable options to control FGS decoding Integrating the FGS bitplane VLD into the original MPEG 2 decoder • • • l Accessing the enhancement layer bit stream (from binary files on disk) Decoding DCT residues from the enhancement bit stream Control the degree of enhancement Generating the enhanced DCT coefficients Decoding the enhanced DCT coefficients to get better video over base layer Supporting I-frame and P-frame, not B-frame

MPEG 2 FGS Decoder Architecture

MPEG 2 Player l l Combination of original MPEG 2 player with the new MPEG 2 FGS decoder New features • • • Supporting FGS Replay Stop Stretching to window Supporting wide screen format FGS scale change on the fly Turning FGS on/off Player window resizing Supporting new command line options

DCT Coefficients l Before quantization l After base layer quantization l Difference l Sign bits l Compressed sign bits 88, -14, 0, 0, -3, 0, 0, 0, -40, -2, 0, 0, 0, -7, 8, 3, 0, 0, 0, 5, 0, 0, 3, 0, 0, 0, 0, 0, 0, 4, 0, 0, 2 88, 0, 0, -32, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1 0, -14, 0, 0, -3, 0, 0, 0, -8, -2, 0, 0, 0, -7, 8, 3, 0, 0, 0, 5, 0, 0, 3, 0, 0, 0, 0, 0, 0, 4, 0, 0, 1 x, 1, x, x, x, 1, 1, 1, x, x, x, 1, 0, 0, x, x, x, 1, x, x, 0, x, x, x, x, x, x, 0, x, x, 0 1, 1, 1, 0, 0, 1, 0, 0

Bitplane coding l l DCT coefficient differences: 0, -14, 0, 0, -3, 0, 0, 0, -8, -2, 0, 0, 0, -7, 8, 3, 0, 0, 0, -5, 0, 0, 3, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0 Bit planes: 0: 1: 2: 3: 4: 5: 6: 7: 8: 0000000000000000000000000000000000000000000000000000000000000000 0100000000100000000000000000000000 010000000100000000000000000000100000 01001000011000001010000000100000000000000010000001010000000010000000000000 020020002220000021100000200001000000000000100000 (sign bits) l Compressed sign bits: l Bitplane VLC encoding 1, 1, 1, 0, 0, 1, 0, 0 Planes 0, 1, 2, 3 are all zero Plane 4: (1, 0, 0), (6, 0, 0), (8, 1, 0) Plane 5: (1, 0, 0), (14, 0, 0), (7, 0, 0), (33, 1, 0) Plane 6: (1, 0, 0), (2, 0, 0), (4, 0, 0), (0, 0, 0), (5, 0, 0), (14, 0, 0), (7, 1, 0) Plane 7: (4, 0, 0), (11, 0, 0), (5, 0, 0), (8, 0, 0), (7, 1, 1)

Bitplane coding (cont. ) Bit plane VLC coding Binary code 0 All zero 3 F 1 All zero 3 F 2 All zero 3 F 3 All zero 3 F 4 (1, 0, 0), (6, 0, 0), (8, 1, 0) 01, 06, 48 5 (1, 0, 0), (14, 0, 0), (7, 0, 0), (33, 1, 0) 01, 0 E, 07, 61 6 (1, 0, 0), (2, 0, 0), (4, 0, 0), (0, 0, 0), (5, 0, 0), (14, 0, 0), (7, 1, 0) 01, 02, 04, 00, 05, 01, 0 E, 47 7 (4, 0, 0), (11, 0, 0), (5, 0, 0), (8, 0, 0), (7, 1, 1) 04, 0 B, 01, 05, 08, C 7 Code format: BPnnnnnn B: End of block flag, P: End of plane flag, nnnnnn: number of 0 before 1 3 F=00111111=“All zero plane”, BF=0 x 10111111=“All planes are zero)

Bitplane VL Codec l l VLC • • Splitting DCT coefficients to bitplanes Converting each bitplane to VL codes Converting VL codes to binary codes (Huffman codes) Storing binary codes as enhancement layer bitstream (binary file) VLD • • Loading binary codes from enhancement layer bitstream (binary file) Converting binary codes (Huffman codes) to VL codes Converting VL codes to bitplanes Merging bitplanes to DCT coefficients

Enhancement Layer Bitstream

No FGS 15% FGS 30% FGS 65%FGS 30%

Demo l Run fgs. bat

Issues l l Enhancement layer bitstream size is much bigger than the base layer bitstream size. Motion prediction can not be applied to enhancement layer Current implementation only supports I-frame and Pframe, not B-frame yet Current implementation does not support Huffman VLC coding in the bitplane VLC. I don’t have a good Huffman VLC table, and Huffman VLC will only help to reduce the enhancement bitstream size ¼ anyway.