CSE 489 02 CSE 589 02 Multimedia Processing

CSE 489 -02 & CSE 589 -02 Multimedia Processing Lecture 11 Video Coding Spring 2009 New Mexico Tech 12/3/2020 1

History H. 264/AVC 12/3/2020 2

MC-DCT Coding Framework q. Motion estimation/compensation based on previously decoded frames q. Block-translation model Ø Inter-coding: DCT-based coding of prediction error (residue) Ø Intra-coding: If motion estimation fails or synchronization is desired, macro-block is encoded in intra-mode q. Most international video coding standards are based on this coding framework Ø Video teleconferencing: H. 261, H. 263++, H. 264 Ø Video archive & play-back: MPEG-1, MPEG-2 (in DVDs), MPEG-4 12/3/2020 3

Hybrid MC-DCT Encoder Transform, Quantization, Entropy Coding Input Macro-Block Motion Compensated Prediction Encoded Residual (To Channel) Entropy Decoding, Inverse Q, Inverse Transform Decoder Decoded Input Macro-Block (To Display) Motion Comp. Predictor 12/3/2020 Motion Estimation Frame Buffer (Delay) Motion Vector and Block Mode Data 4 (Side-Info, To Channel)

Inter and Intra Coding p Intra n n p MB is encoded as is without motion compensation DCT followed by Q, zig-zag, run-length, Huffman Inter n n n 12/3/2020 Block-matching motion estimation Predictive motion residue from best-match block is DCT encoded (similarly to intra-mode) Motion vector is differentially encoded 5

Intra-Coding Mode input MB to bit-stream Encoder to motion compensated frame bit-stream to display frame Decoder 12/3/2020 6

Inter-Coding Mode to bit-stream input MB Encoder 12/3/2020 reference frame 7

Video Sequence and Picture Intra 0 p Inter 2 Inter 3 Inter 4 Intra Picture (I-Picture) n n p Inter 1 Encoded without referencing others All MBs are intra coded Inter Picture (P-Picture, B-Picture) Encoded by referencing other pictures n Some MBs are intra coded, and some are inter coded 12/3/2020 8 n Inter 5

Group of Pictures GOP p GOP … GOP I B Group of Pictures (GOP) Video stream I B B P Frame order: 0 1 2 3 4 5 6 Encoding order: 0 2 3 1 5 6 4 12/3/2020 … B B B P …

Coding of I-Slice DCT Original block Bit-stream Transformed block 15 0 -2 -1 -1 -1 0 … Entropy coding 12/3/2020 Zig-zag scan 10 Quantization matrix

Coding of P-Slice - Motion Estimation = Original current frame = Motion Vectors + Frame buffer 12/3/2020 Reconstructed reference frame 11 Motion Compensation Residual

Motion Estimation in 8 H. 261 Macro-block n Luminance: 16 x 16, four 8 x 8 blocks n Chrominance: two 8 x 8 blocks n Motion estimation only performed for luminance component p Motion vector range 12/3/2020 n [ -15, 15] p 8 Y Y Cr Cb 15 15 MB 15 15 Search Area in Reference Frame 12

Coding of Motion Vectors p p p MV has range [-15, 15] Integer pixel ME search only Motion vectors are differentially & separably encoded 11 -bit VLC for MVD Example MV = 2 2 3 5 3 1 -1… MVD = 0 1 2 -2 -2 -2… Binary: 1 010 0011… 12/3/2020 13

Inter/Intra Switching p Based on energy of prediction error n n High energy: scene change, occlusions, uncovered areas… use intra mode Low energy: stationary background, translational motion … use inter mode VAR INTER 64 INTRA MSE 64 12/3/2020 14

Loop Filter p p 12/3/2020 Optional Can be turned on or off for each block, usually go together with MC Advantage n Decreases prediction error by smoothing the prediction frame n Reduces high-frequency artifacts like mosquito effects Disadvantage n Increases complexity & overhead 15

Quantization p p Uniform mid-rise quantizer for intra DC coefficients Uniform mid-tread quantizer with double dead zone for inter DC and all AC coefficients Y ^ =X -2 Q Y -Q 1 X 0 -1 -2 12/3/2020 2 ^ =X 2 For intra DC Q 2 Q 1 -2 Q -Q 0 -1 Q -2 For inter DC and all AC X 2 Q 16

H. 263 p p Standardization effort started Nov 1993 Aim n n p Near-term n p H. 263 and H. 263+: established late 1997 Long-term n p low bit-rate video communications, less than 64 kbps target PSTN and mobile network: 10 -32 kbps H. 26 L, H. 264: still under investigation Main properties H. 261 with many MPEG features optimized for low bit rates n Performance: 3 -4 d. B improvements over H. 261 at less 12/3/2020 than 64 kbps; 30% bit rate saving over MPEG-1 n 17

MPEG p p p Coding and communications of moving pictures and associated audio for digital storage and archival MPEG: Moving Picture Expert Group MPEG family n n p MPEG-1, MPEG-2, MPEG-4, MPEG-7, Nov 1992 Nov 1994 Oct 1998 ongoing work Main features of the MPEG video family n n n 12/3/2020 n Bi-directional MEMC I-frame, P-frame, B-frame Structure: Group of Pictures (GOP), picture, slice, macroblock 18 Coding decisions

MPEG Goals and Applications p MPEG-1 n n n p Optimized for applications that support a continuous transfer bit rate of about 1. 5 Mbps (example, CD-ROM) Target 1. 2 Mbps for video and 250 -300 kbps for audio, around analog VHS quality Does not support interlaced sources Main target source: SIF YCr. Cb 4: 2: 0 360 x 240 x 30 fps VCD MPEG-2 The most commercially successful international coding standard n Wide range of bit rates: 4 – 80 Mbps; optimized for 4 Mbps 12/3/2020 n Target high-resolution, high-quality video broadcast & playback n 19

Requirements p p p p 12/3/2020 Coding of generic video at around 1. 5 Mbps at reasonable quality (VHS) Random access capability, frequent access point Fast forward and fast rewind capability Audio-video synchronization during play and access Simple decoder Flexibility of data format Certain degree of robustness to communication errors Real-time encoder possibility 20

From H. 261 to MPEG-1 p There a few new features in MPEG-1 comparing to the pioneering H. 261 codec n n n n 12/3/2020 Flexible data sizes and frame rates More flexible slice structure to replace the fixed GOB structure Data structure: introducing Group of Picture (GOP) allowing frequent access points Bi-directional motion compensation, B-frames Half-pixel motion compensation More finely tuned VLCs for different purposes Quantization table (like JPEG) replaces single Q step size 21

Bidirectional MC Properties p Advantage n n n p Higher coding efficiency, frame rate can be increased significantly with few bits More accurate motion estimation & compensation No error propagation Disadvantage n n 12/3/2020 More memory buffer for frame storage (minimum of 3) More end-to-end delay 22

H. 264/AVC History p In the early 1990’s, the first video compression standards were introduced: n n p Since then, the technology has advanced rapidly n n n p H. 261 (1990) and H. 263 (1995) from ITU MPEG-1 (1993) and MPEG-2 (1996) from ISO H. 263 was followed by H. 263+, H. 263++, H. 26 L MPEG-1/2 followed by MPEG-4 visual But industry and research coders are still way ahead H. 264/AVC is a joint project of ITU and ISO, to create an up-to-date standard. 12/3/2020 23

Scope and Context p Aimed at providing high-quality compression for various services: n n p Standard defines: n n p IP streaming media (50 -1500 kbps) SDTV and HDTV Broadcast and video-on-demand (1 - 8+ Mbps) DVD Conversational services (<1 Mbps, low latency) Decoder functionality (but not encoder) File and stream structure Final results: 2 -fold improvement in compression Same fidelity, half the size --- Compared to H. 263 and MPEG-2 12/3/2020 24

Video Compression p Motion compensation / prediction n n p Image transform n p Described current frame based on previous frame Output description + residual image Predicted frames are called “inter-frames”. Some frames (intra-frames) are encoded without prediction, as natural images. Concentrate image energy in relatively few numeric coefficients Lossy coding n n 12/3/2020 Compress coefficient values in a lossy manner Try to keep most important information 25

The H. 263 Standard Coder original video Motion Compensation 12/3/2020 compressed video Image Transform Lossy Coding 26

The H. 263 Standard Coder original video H. 263 Motion Compensation compressed video • Image is divided into 16 x 16 macroblocks, • Each macroblock is matched against nearby blocks in previous frame (called reference frame), • “Nearby” = within 15 -pixel horizontal/vertical range Image Lossy Motion • Half-pixel accuracy (with bilinear pixel interpolation) Transform Coding Compensation • Best match is used to predict the macroblock, • The relative displacement, or motion vector, is encoded and transmitted to decoder • Prediction error for all blocks constitute the residual. 12/3/2020 27

Motion Compensation Example 12/3/2020 T=1 (reference) T=2 (current) 28

The H. 263 Standard Coder original video compressed video H. 263 Image Transform • Residual is divided into 8 x 8 blocks, • 8 x 8 2 -d Discrete Cosine Transform (DCT) is applied to each block independently Image Lossy Motion • DCT coefficients describe. Transform spatial frequencies in the block: Coding Compensation • High frequencies correspond to small features and texture • Low frequencies correspond to larger features • Lowest frequency coefficient, called DC, corresponds to the average intensity of the block 12/3/2020 29

8 x 8 DCT Example 12/3/2020 30

8 x 8 DCT Example 12/3/2020 31

8 x 8 DCT Example 12/3/2020 32

The H. 263 Standard Coder original video H. 263 Lossy Coding compressed video • Transform coefficients are quantized: • Some less-significant bits are dropped • Only the remaining bits are encoded • For inter-frames, all coefficients get the same number of bits, except for the DC which gets more. Image Lossy • For. Motion intra-frames, lower-frequency coefficients get more bits Transform Coding Compensation • To preserve larger features better • The actual number of bits used depends on a quantization parameter (QP), whose value depends on the bit-allocation policy • Finally, bits are encoded using entropy (lossless) code • Traditionally Huffman-style code 12/3/2020 33

Changes in Motion Compensation p Quarter-pixel accuracy n p Variable block-size: n n p Every 16 x 16 macroblock can be subdivided Each sub-block gets predicted separately Multiple and arbitrary reference frames n p A gain of 1. 5 -2 d. B across the board over ½pixel Vs. only previous (H. 263) or previous and next (MPEG). Anti-aliasing sub-pixel interpolation n 12/3/2020 Removes some common artifacts in residual 34

Variable Block-Size MC p Motivation: size of moving/stationary objects is variable n n p Many small blocks may take too many bits to encode Few large blocks give lousy prediction In H. 264, each 16 x 16 macroblock may be: n n 12/3/2020 Kept whole, Divided horizontally (vertically) into two subblocks of size 16 x 8 (8 x 16) Divided into 4 sub-blocks In the last case, the 4 sub-blocks may be divided once more into 2 or 4 smaller blocks. 35

H. 264 Variable Block Sizes 12/3/2020 36

Motion Scale Example 12/3/2020 T=1 T=2 37

Motion Scale Example 12/3/2020 T=1 T=2 38

Motion Scale Example 12/3/2020 T=1 T=2 39

H. 264 VBS Example 12/3/2020 T=1 T=2 40

Arbitrary Reference Frames p p p In H. 263, the reference frame for prediction is always the previous frame In MPEG and H. 26 L, some frames are predicted from both the previous and the next frames (biprediction) In H. 264, any one frame may be used as reference: n n p Encoder and decoder maintain synchronized buffers of available frames (previously decoded) Reference frame is specified as index into this buffer In bi-predictive mode, each macroblock may be: Predicted from one of the two references 12/3/2020 n Predicted from both, using weighted mean of predictors 41 n

Intra Prediction p Motivation: intra-frames are natural images, so they exhibit strong spatial correlation n p Macroblocks in intra-coded frames are predicted based on previously-coded ones n n p Implemented to some extent in H. 263++ and MPEG-4, but in transform domain Above and/or to the left of the current block The macroblock may be divided into 16 4 x 4 sub-blocks which are predicted in cascading fashion An encoded parameter specifies which neighbors should be used to predict, and how 12/3/2020 42

Intra-Prediction Example 12/3/2020 43

Intra-Prediction Example Vertical 12/3/2020 44

Intra-Prediction Example Horizontal 12/3/2020 45

Intra-Prediction Example Main Diagonal 12/3/2020 46

H. 264 Image Transform p p p Motivation: n DCT requires real-number operations, which may cause inaccuracies in inversion H. 264 uses a very simple integer 4 x 4 transform n A (pretty crude) approximation to 4 x 4 DCT n Transform matrix contains only +/-1 and +/-2 p Can be computed with only additions, subtractions, and shifts Results show negligible loss in quality (~0. 02 d. B) 12/3/2020 47

Deblocking Filter Non Deblocked Image Courtesy : Images from http: //compression. ru/video/deblocking/ 12/3/2020 48

Entropy Coding p Motivation: traditional coders use fixed, variable-length codes n n n p Essentially Huffman-style codes Non-adaptive Can’t encode symbols with probability > 0. 5 efficiently, since at least one bit required H. 263 Annex E defines an arithmetic coder n n 12/3/2020 Still non-adaptive Uses multiple non-binary alphabets, which results in high computational complexity 49

Entropy Coding: CABAC p p p Context-adaptive binary arithmetic coding (CABAC) framework designed specifically for H. 264 Binarization: all syntax symbols are translated to bit-strings 399 predefined context models, used in groups n E. g. models 14 -20 used to code macroblock type for inter-frames n The model to use next is selected based on previously coded information (the context) 12/3/2020 50

Comparison to MPEG-2, H. 263, MPEG-4 p 2 Tempete CIF 30 Hz Quality Y-PSNR [d. B] 38 37 36 35 34 33 32 31 30 29 28 27 26 25 JVT/H. 264/AVC MPEG-4 Visual MPEG-2 H. 263 0 500 1000 1500 2000 Bit-rate [kbit/s] 2500 3000 3500