Fine Granularity Video Compression and Optimal FEC Assignment

Fine Granularity Video Compression and Optimal FEC Assignment for FG Video Streaming over Burst Error Channel Yih-Ching Su Department of Computer Science and Engineering, National Sun Yat-Sen University

Contents 1. 2. 3. 4. 5. 6. 7. Introduction Gilbert Channel with Loss Rate Feedback Optimal FEC Assignment for FG Video HSDD Motion Estimation Metric HMRME Motion Estimation Algorithm ABEC Embedded Coder Conclusions & Future Works 2

3

Research Focuses n n Optimal FEC assignment scheme for FG video transmission over burst error channel (as wireless Internet) with or without loss rate feedback. Wavelet domain video compression algorithms with high-performance or low-complexity features. Introduction 4

Research Focuses (cont. ) Source Coder Raw Video Motion Estimation Transform HSDD HMRME ABEC Quantization & Entropy Coding Channel Coder Error-Resilient Video Packets FEC Protection Optimal FEC Assignment Introduction 5

Definition of Fine Granularity Video Stream n n Bit stream is scalable (layered). Rate can be precisely controlled. arbitrary enhancement layer length (in bits) R min. (base layer length) max. Introduction 6

Merits of Fine Granularity Video Stream n n Precise rate control Bandwidth adaptation EL BL FG Video Encoder EL BL Media Server FG Client Heterogeneous Internet Environment EL No transcoding! BL Introduction Client 7

Merits of Fine Granularity Video Stream (cont. ) n Content-adaptive error protection EL EL BL BL Equal Error Protection Unequal Error Protection 8

Fine Granularity Video Compression Systems DCT based: n MPEG-4 FGS n n ISO/IEC 14496 -2: 2001/Amd 2: 2002 Base layer plus enhancement layer DWT based: n “Multirate 3 -D subband coding of video”, D. Taubman et al. , 1994. n “ 3 D SPIHT”, B. -J. Kim et al. , 2000. n “HSDD”, Y. -C. Su et al. , 2003. Introduction 9

10

Packet Loss n n Packet loss can severely affect the quality of delay sensitive multimedia applications. FEC (Forward Error Correction) technique can be used when delay time is strictly restricted. BOP len = n pkts data len = k pkts FEC redundancy : the probability of m lost packets within a block of n packets. Gilbert Channel with Loss Rate Feedback 11

Gilbert Channel Model n n The ability of the application to react is enhanced by the availability of simple and efficient loss models. A two state Markov model or Gilbertmodel is often used to simulate burst loss patterns over wired/wireless channel. C. C. Tan, N. C. Beaulieu, ”On first -order Markov modeling for the rayleigh fading channel, ” IEEE Commun. , 2000. Gilbert Channel with Loss Rate Feedback 12

Enhanced Video Transmission over Gilbert Channel n n Feedback loss rate. Decide FEC protection ratio relying on a new probability function which is conditioned on loss rate feedback. mv BOP-b-v+1 m 2 m 1 BOP-b-1 BOP-b feedback delay b Gilbert Channel with Loss Rate Feedback m BOP 0 13

Renewal Error Process n n Packet loss over Gilbertmodel can be modeled with a renewal error process. The lengths of consecutive inter-error intervals (also called gaps) are independently and identically distributed. Gap probabilities: Probability that m-1 packet losses occur in the next n-1 packets following an error: Probability that m packet losses occur within a block of n packets: E. N. Gilbert, "Capacity of a burst-noise channel, " Bell Syst. Tech. J. , vol. 39, pp. 1253 -1265, Sept. 1960. Gilbert Channel with Loss Rate E. O. Elliott, "A model of the switched telephone network Feedback for data communications, " Bell Syst. Tech. J. , 1965. 14

Probability Toolbox n-1 E E#(m-1) E n-1 E E#(m-1) Gilbert Channel with Loss Rate Feedback S 15

Probability Toolbox (cont. ) n-1 S S#(m-1) S n-1 S S#(m-1) E 16

Probability Toolbox (cont. ) n E E n E S n S Gilbert Channel with Loss Rate Feedback E 17

Probability Toolbox (cont. ) n-1 S E#(m) S n-1 S E#(m-1) E n-1 E E#(m) S n-1 E E#(m-1) E 18

Iterative Equation Set BOP-b-v+1 BOP-b-i BOP-b Gilbert Channel with Loss Rate Feedback 19

Initial Conditions Gilbert Channel with Loss Rate Feedback 20

Conditional Probability Function Gilbert Channel with Loss Rate Feedback 21

Validation of Correctness Gilbert Channel with Loss Rate Feedback 22

Performance Evaluation Gilbert Channel with Loss Rate Feedback 23

24

FEC Assignment Schemes n n n B. Hong and A. Nostratinia, "Rateconstrained scalable video transmission over the internet, " Packet Video 2002. Equal error protection Content-adaptive unequal error protection Content-adaptive plus channel-adaptive unequal error protection Y. C. Su, C. S. Yang, and C. W. Lee, "Optimal FEC Assignment for Scalable Video Transmission over Burst Error Channel with Loss Rate Feedback, " Packet Video 2003. Optimal FEC Assignment for FG Video 25

Block of Packets (BOP) Structure Layer 0 Layer 1 Layer i Layer l packet k 0 k 1 ki kl FEC overhead s 0 s 1 si packet size s number of packets n sl 26

Complete Expected Quality Optimal FEC Assignment for FG Video 27

Simplified Expected Quality content adaptive content+channel adaptive Optimal FEC Assignment for FG Video 28

The Optimization Problem Constrained by Optimal FEC Assignment for FG Video 29

Dynamic Programming Optimal FEC Assignment for FG Video 30

Validation of Correctness (i) frame resolution = CIF format (352 x 288) (ii) constant stream rate = 256 Kbps (iii) 1 GOP = 1 intra frame accompanied with 14 inter frames and frame rate = 15 fps (iv) sequence length = 9 GOPs Optimal FEC Assignment for FG Video 31

Performance Discrepancy between Complete & Simplified Models Optimal FEC Assignment for FG Video 32

Performance Evaluation Optimal FEC Assignment for FG Video 33

Performance Evaluation (cont. ) Optimal FEC Assignment for FG Video 34

Performance Evaluation (cont. ) Optimal FEC Assignment for FG Video 35

Performance Evaluation (cont. ) Optimal FEC Assignment for FG Video 36

Performance Evaluation (cont. ) Optimal FEC Assignment for FG Video 37

38

Bit-Plane Coding n The Core of FGS or Embedded Coder Just bit-plane coding! HSDD Motion Estimation Metric 39

Zero-Tree Coding n n n Natural images in general have a low pass spectrum. Large wavelet coefficients are more important than small wavelet coefficients. A zero-tree is a quad-tree of which all nodes are equal to or smaller than the root. HSDD Motion Estimation Metric 40

Hierarchical Sum of Double Difference Metric n n n Zero-tree coding aware Jointly constrain motion vector searching for both temporal and spatial (quad-tree) directions Fewer bits are spent later for describing isolated zeros HSDD Motion Estimation Metric 41

Sum of Absolute Difference Metric Reference Block Current block's pixel (block size nxn) Reference block's pixel within search area (2 p+1)x(2 p+1) 2 p+1 Current Block SAD metric conflicts with the zerotree rule often, because the goal of SAD metric is just to minimize the temporal difference, and it is irrelevant to the magnitude hierarchy of the spatial quad-trees. HSDD Motion Estimation Metric 42

HSDD Metric Calculation Double Difference Hierarchy Sum Current block's pixel (block size nxn) Reference block's pixel within search area (2 p+1)x(2 p+1) Corresponding parent pixel information in the upper level of motion compensation pyramid HSDD Motion Estimation Metric 43

Observations on HSDD Metric n n HSDD value may be negative, but a larger positive one is preferred. Given any parent pixel information, the maximal HSDD(MV) occurs if and only if the perfect SAD matching exists, that is SAD(MV)->0. HSDD Motion Estimation Metric 44

Motion Estimation Applying HSDD Metric HSDD Motion Estimation Metric 45

Layered Magnitude Distributions for HSDD & SAD HSDD Motion Estimation Metric 46

Performance Evaluation HSDD Motion Estimation Metric 47

48

Half-Pixel Multi-Resolution Motion Estimation n Combine transform-adapted half-pixel interpolation with anti-aliasing under complexity constraints. Avoid multiple inverse transforms. Can be united with the conventional wavelet domain motion estimation algorithms. HMRME Motion Estimation Algorithm 49

H-Transform h=H a HMRME Motion Estimation Algorithm 50

Aliasing HMRME Motion Estimation Algorithm 51

Half-Pixel Interpolation HMRME Motion Estimation Algorithm 52

Horizontal Interpolation HMRME Motion Estimation Algorithm 53

Vertical Interpolation HMRME Motion Estimation Algorithm 54

Diagonal Interpolation HMRME Motion Estimation Algorithm 55

Performance Evaluation MRME: Y. Q. Zhang, S. Zafar, “Motion. Compensated Wavelet Transform Coding for Color Video Compression, ” IEEE CSTV, 1992. AMRME: M. K. Mandal, E. Chan, X. Wang and S. Panchanathan, “Multiresolution Motion Estimation Techniques for Video Compression, ” Optical Engineering, 1996 HMRME Motion Estimation Algorithm 56

57

Array-Based Embedded Coder n n Performance similar to SPIHT (Amir Said and William A. Pearlman, ”A New Fast and Efficient Image Codec Based on Set Partitioning in Hierarchical Trees, ” IEEE CSVT, 1996) One pass processing & no link lists Hardware implementation friendly R. O. C. patent no. 141267, 2001 ABEC Embedded Coder 58

ABEC Encoding Flow Raw Image Wavelet Transform Predict Bit Expenditure Remove DC Gain Last Round? NO YES Establish Significance Map ABEC Final Processing Stop ABEC Normal Processing 59

Significance Map Max values ABEC Embedded Coder 60

ABEC Encoder Structure “Zero-tree” Definitions of ABEC Status Bits • P: parent’s significance bit • S: parent’s sign bit • R: parent’s refinement bit • C: children’s significance bit ABEC Embedded Coder 61

62

Conclusions n n Joint optimization for wavelet domain ME & zero-tree coding can raise the compression performance significantly (HSDD). According to the prediction for DC coefficients in wavelet domain, the ideas of fast antialiasing & transform-adapted half-pixel interpolation can be combined (HMRME). Conclusions & Future Works 63

Conclusions (cont. ) n n One pass processing & no link lists; fast & hardware friendly zero-tree coding is possible (ABEC). The loss probability function for Gilbert channel conditioned on past loss rates can be calculated out by an iterative equation set. Conclusions & Future Works 64

Conclusions (cont. ) n n Content-adaptive plus channel-adaptive (loss rate feedback) unequal error protection can further enhance FG video transmission efficiency. Simplified quality prediction formulas can be used with trivial performance degradation while significant speeding up. Conclusions & Future Works 65

Future Works n n n Exploit possible optimal or sub-optimal weighting rules for the two difference terms in HSDD metric. Extend HMRME (by lifting scheme? ) to be available for overlapped transforms. Try to find some other better estimation method for ho in HMRME. Conclusions & Future Works 66

Future Works (cont. ) n n Upgrade to an context-based entropyconstrained version of ABEC coder. Investigate the affection of packet length to FG video transmission over bit -error channel. Conclusions & Future Works 67

68