Deblocking Algorithms in Video and Image Compression Coding

Deblocking Algorithms in Video and Image Compression Coding Speaker: Wei-Yi Wei Advisor: Prof. Jian-Jung Digital Image and Signal Processing Lab GICE, National Taiwan University

Outline n n n n Introduction Observations of Blocking Artifact Four Categories of Deblocking Algorithms Block-Edge Impairment Metric Comparison Simulation Conclusions and Future Work References 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 2

Introduction n What is Blocking Artifacts ? – Discontinuity over the block boundaries – Result from coarse quantization 2009/04/11 Original Image. Algorithms in Video Highly Compressed Image Deblocking and Image Compression Coding 3

A Simple Coarse Quantization Example DCT Q/Q-1 IDCT 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 4

Observations of Blocking Artifacts n Two observations of blocking artifact in block-based transform coding – The blocking artifact are more noticeable in flat areas than in complex areas – The motion compensation propagates the blocking artifacts into the next frames 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 5

Four Categories of Deblocking Algorithms n In-loop Deblocking Filter n Post-processing Deblocking Algorithms n Pre-processing Algorithms n Overlapping Block Methods 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 6

In-loop Deblocking Filter Mainly used in video compression coding n Add deblocking filtering to the coding loop n Not compatible with the existing standards n Case Study n – H. 264/AVC – Optimal Post-Process/In-Loop Filtering 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 7

H. 264/AVC In-loop Filter (1/2) Video Source Intra Inter Transform Quantization Coefficient Scanning Bitstream Motion Estimation Motion Compensation Inverse Quantization Frame Buffer Intra Frame Prediction Inverse Transform Entropy Coding In-Loop Filter 2009/04/11 Motion Vector Deblocking Algorithms in Video and Image Compression Coding 8

H. 264/AVC In-loop Filter (2/2) H. 264/AVC encoder must determine the boundary strength (BS) before filtering. n Apply the filters with different strength based on the boundary strength n q 0 q 1 q 2 q 3 p 2 p 1 p 0 2009/04/11 BS Pixels to be modified 4 p 0, p 1, p 2, q 0, q 1, q 2 3 p 0, p 1, q 0, q 1 2 p 0, p 1, q 0, q 1 1 p 0, p 1, q 0, q 1 0 No filtering is applied Deblocking Algorithms in Video and Image Compression Coding 9

Optimal Post-process/In-loop Filtering (1/3) Buffer Fullness Coder Control Video Source Residue Transform QP Entropy Coding Quantization DCT Coefficients Inverse Quantization Enable Prediction 0 Motion Compensation Motion Vector Motion Estimation Data out Inverse Transform Buffer OPF Frame Buffer Previous Decoded Frame Multiplex ILOF Filter with G Find Optimal Filter Parameters for Kernel G Deblocking Algorithms in Video and Image Compression 2009/04/11 Coding 10

Optimal Post-process/In-loop Filtering (2/3) Input Frame Filter Kernel Reconstructed Frame The relationship between with size W × H with size l × l with size W × H , and can be expressed as The optimal filter coefficient matrix must satisfy Row-stacked form of 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 11

Optimal Post-process/In-loop Filtering (3/3) Row-stacked form of l Window Matrix Solve this equation, then obtain Now we the can equation the optimal filter coefficients Multiply both side of 2009/04/11 l ^ I(i, j) can be expressed as with G , we can obtain Deblocking Algorithms in Video and Image Compression Coding 12

Post-processing Deblocking Algorithms The most popular deblocking methods n Improve the quality of the compressed image and video without the source data n Case Study n – Reduction of Blocking Artifacts in DCT Domain – Deblocking Using Weighted Sums of Symmetrically Aligned Pixels – Deblocking Using Offset and Shift Technique 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 13

Reduction of Blocking Artifacts in DCT Domain (1/2) DCT-domain filtering where Low Activity High Activity 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 14

Reduction of Blocking Artifacts in DCT Domain (2/2) The neighboring DCT coefficients at the same frequency do not vary radically within a small range n Apply lowpass filter to each frequency to filter the high frequency components resulting from blocking effect n 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 15

Deblocking Using Weighted Sums of Symmetrically Aligned Pixels (1/4) n Deblocking Frame 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 16

Deblocking Using Weighted Sums of Symmetrically Aligned Pixels (2/4) The filtered pixel where the filter coefficients are 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 17

Deblocking Using Weighted Sums of Symmetrically Aligned Pixels (3/4) Linear Solution Quadratic Solution where 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 18

Deblocking Using Weighted Sums of Symmetrically Aligned Pixels (4/4) Linear Solution 2009/04/11 Quadratic Solution Deblocking Algorithms in Video and Image Compression Coding 19

Deblocking Using Offset and Shift Technique (1/3) n Estimate the block of activity Horizontal Activity Block boundary Vertical Activity 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 20

Deblocking Using Offset and Shift Technique (2/3) n Filtering for Uniform Deblocking Blocks Block boundary offset p 1 p 2 p 3 p 4 p 5 p 6 Block boundary Offset/2 Offset/4 Offset/8 p 1 p 2 2009/04/11 p 3 Offset/8 Offset/4 Offset/2 p 4 p 5 p 6 Deblocking Algorithms in Video and Image Compression Coding 21

Deblocking Using Offset and Shift Technique (3/3) n Filtering for Directional Deblocking Blocks n Filtering for Complex Deblocking Blocks 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 22

Pre-processing Algorithms (1/2) n Achieve deblocking and enhance the coding efficiency q 3 q 0 p 3 2009/04/11 p 2 p 1 q 2 Encoding Decoding Deblocking Algorithms in Video and Image Compression Coding 23

Pre-processing Algorithms (2/2) n Our objective is to minimize the difference between I and V and obtain the coefficients Original (I) Pre-processed Solve this equation, (R) then we can obtain Filtered (V) the optimal filter coefficients The difference between I and V Taking derivative 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 24

Overlapped Block Methods n Source of Blocking Artifact – Block-based transform does not take the correlation across the block boundary – Motion Estimation in video coding will propagate the blocking artifact to the next frame n How to solve these two problems – Lapped Orthogonal Transform – Overlapped Block Motion Compensation 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 25

Lapped Orthogonal Transform (1/3) n n The blocks overlap slightly and the redundant information is transmitted for the samples in the block boundaries Reduce the blocking artifact Assume the 1 D discrete time signal is a sequence of MN samples Transform Coefficients Segments the signal into M blocks and each block with size N 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 26

Lapped Orthogonal Transform (2/3) Energy Compaction Measurement Where is the i-th diagonal entry of the matrix where 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 27

Lapped Orthogonal Transform (3/3) 1 2 2 4 3 4 DCT 5 3 6 5 7 7 0 0 1 2 2 4 3 4 DCT Block 1 DCT 6 1 5 3 6 5 7 7 2009/04/11 1/2 2 Y(0 ) Y(2 ) Y(4 ) Y(6 ) - O Block 2 DCT Y(1 ) Y(3 ) Y(5 ) Y(7 ) DCT Block M - E O 1/2 - E 1/2 O - 1/2 2 LOT of Block 2 1/2 - E LOT of Block 1 … 0 - 1/2 Deblocking Algorithms in Video and Image Compression Coding LOT of Block M 28

Overlapped Block Motion Compensation (1/2) Reference blocks Window Function Reference Frame Overlapped Block Evaluation Target Block Current Frame 2009/04/11 Motion Vector Output Deblocking Algorithms in Video and Image Compression Coding 29

Overlapped Block Motion Compensation (2/2) Motion Vectors Reference Frame Overlapped Block Window Function Prediction Frame 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 30

Block-Edge Impairment Metric (1/2) n The disadvantages of PSNR – It does not always reveal the real quality perceived by the HVS – It can not real the blockiness n Several block-edge impairment metrics have been proposed – GBIM – It can reveal the real blockiness and evaluate the performance of deblocking algorithms 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 31

Block-Edge Impairment Metric (2/2) Reconstructed Image after deblocking =3. 11 =1. 05 PSNR=21. 75 d. BAlgorithms in Video and. PSNR=21. 86 d. B Deblocking Image Compression 2009/04/11 Coding 32

Comparison (1/2) n In-loop Filter – Achieve better improvement because it can refer to the new input video frame – Not compatible with the existing standards n Post-processing Deblocking Algorithms – Good potential to be integrated into existing standards – Blind to the target image and video 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 33

Comparison (2/2) n Pre-processing Deblocking Algorithms – Modify the source image and video in advance and reduces the bit rate – Achieve the quality close to the direct compressed image and video n Overlapped Block Methods – Prevents the blocking artifacts from happening in advanced 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 34

Simulation (1/3) Original Image Highly Compressed Image DCT Domain Offset and Shift Deblocking Algorithms. WSSAP in Video and Image Compression Coding 2009/04/11 35

Simulation (2/3) n LOT and DCT PSNR under different quantization strength LOT with Q=10 DCT with Q=10 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 36

Simulation (3/3) n Overlapped Block Motion Compensation 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 37

Conclusions and Future Work Many conventional methods do not exploit the characteristics of the blocking artifacts n The new algorithms apply different filter with different strength to different areas n Adopting the blockiness metric to measure the blockiness will be the future trend n 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 38

References (1/2) In-loop Filter n n n [1] Lain E. G. Richardson, “H. 264 and MPEG-4 Video Compression: Video Coding for Next-generation Multimedia”, John Wiley & Sons, Jan. 2004. [2] Lee, Jae-Beom, Kalva and Hari, ”The VC-1 and H. 264 Video Compression Standards for Broadband Video Services”, Springer, 2008. [3] “Text of ISO/IEC FDIS 14496 -10/Draft ITU-T H. 264: Information Technology – Coding of Audio. Visual Objects: Advanced Video Coding”, International Organization for Standardization, 2003. [4] Dong-Hwan Kim, Hwa-Yong Oh, O˘guzhan Urhan, Sarp Ertürk and Tae-Gyu Chang, “Optimal Post. Process/In-Loop Filtering for Improved Video Compression Performance”, IEEE Trans. on Consumer Electronics, vol. 53, no. 4, Nov. 2007. [5] S. Romero, and L. F. Romero, “An Optimized Preconditioned Conjugate Gradient Algorithm”, Technicah Report No: UMA-DAC-02/11, University of Malaga, Sept. 2002. Post-Filtering n n [6] Tao Chen, Hong Ren Wu and Bin Qiu, “Adaptive Postfiltering of Transform Coefficients for the Reduction of Blocking Artifacts”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 11, no. 5, Dec. 2001. [7] Shizhong Liu and Alan C. Bovik, “Efficient DCT-Domain Blind Measurement and Reduction of Blocking Artifacts”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 12, no. 12, May 2002. [8] Ci Wang, Wen-Jun Zhang and Xiang-Zhong Fang, “Adaptive Reduction of Blocking Artifacts in DCT Domain for Highly Compressed Images”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 50, no. 2, May 2004. [9] A. Z. Averbuch, A. Schclar and D. L. Donoho, “Deblocking of block-transform compressed images using weighted sums of symmetrically aligned pixels, ” IEEE Trans. on Circuits Syst. Video Technology, vol. 14, pp. 200 -212, Feb. Deblocking 2005. Algorithms in Video and Image Compression 2009/04/11 Coding 39

References (2/2) n n n [10] Jongho Kim, Minseok Choi, and Jechang Jeong, “Reduction of Blocking Artifact for HDTV using Offset-and-Shift Techniques”, IEEE Trans. on Consumer Electronics, vol. 53, no. 4, November 2007. [11] Zixiang Xiong, Michael T. Orchard, and Ya-Qin Zhang, “A Deblocking Algorithm for JPEGCompressed Images Using Overcomplete Wavelet Representations”, IEEE Trans. on Circuits and Systems for Video Technology, vol. 7, no. 2, April 1997. [12] Jongho Kim and Jechang Jeong, “Adaptive Deblocking Technique for Mobile Video”, IEEE Trans. on Consumer Electronics, vol. 53, no. 4, Nov. 2007. Pre-Filtering n [13] Sheng-Ho Wang, Sung-Wen Wang, Yi-Shin Tung, Ja-Ling Wu, Jau-Hsiung Huang, "Pre-Process for maximizing the effect of in-loop deblocking filtering in H. 264/AVC encoding, " 5 th EURASIP Conference focused on Speech and Image Processing, Multimedia Communications and Services (EC-SIP-M 2005), Smolenice, Slovak, 29 June - 2 July, 2005. Overlapped Block Methods n n [14] H. S. Malvar and D. H. Staelin, “The LOT: transform coding without blocking effects”, IEEE Trans. on Acoustic. , Speech, Signal Processing, vol. 37, no. 4, pp. 553 -559, April 1989. [15] M. T. Orchard and G. J. Sullivan, “Overlapped block motion compensation: an estimation-theoretic approach”, IEEE Trans. on Image Processing, vol. 3, no. 4, pp. 693 -699, Sep. 1994. Block-Edge Impairment Metric n n [16] H. R. Wu and M. Yuen, “A Generalized Block-Edge Impairment Metric for Video Coding”, IEEE Signal Processing Letters, vol. 4, no. 11, Nov. 1997. [17] Athanasios Leontaris, Pamela C. Cosman and Amy R. Reibman, “Quality Evaluation of Motion-Compensated Edge Artifacts in Compressed Video”, IEEE Trans. on Image Processing, vol. 16, no. 4, April 2007. 2009/04/11 Deblocking Algorithms in Video and Image Compression Coding 40