Modern Transform Design Trn Duy Trc ECE Department

Modern Transform Design Trần Duy Trác ECE Department The Johns Hopkins University Baltimore, MD 21218

Acknowledgements u Former Ph. D students § Jie Liang, Assistant Professor, Simon Fraser Univ. § Chengjie Tu, Senior Software Design Engineer, Microsoft Corporation u Funding support § NSF: CCF-0093262, CCF-0430869, CCF-0728893

Outline u Transformation fundamentals § § u Linear mapping Popular transforms: WHT, DFT, FFT, DCT Desirable transform properties in image/video coding Concentration: integer-coefficient design Direct scaling design § H. 264 16 -bit 4 x 4 integer transform u Lifting-based design § § Upper- or lower-triangular matrix components bin. DCT: binary-friendly DCT Lossless integer transforms JPEG-XR integer POT & PCT

Image/Video Compression Paradigm Quantization original signal compressed bit-stream reconstructed signal Transform Prediction MEMC VLC Huffman Arithmetic Run-length

Next-Generation Coding Standards u H. 264/MPEG-4 Advanced Video Coding (AVC) § International video coding standard § ISO/IEC 14496 -10 MPEG-4 Part 10 § Broad application range: Internet streaming, broadcast, Blu-ray… u VC-1 in Microsoft Windows Media 9 § SMPTE 421 M video coding standard, 04/03/2006 § Mandatory supported standard for HD-DVD, Blu-ray § Part of Windows Media 9 u JPEG-XR, formerly HD Photo or Windows Media Photo § Flexible high-dynamic-range image compression standard § ISO/IEC Standard 29199 -2, 06/19/2009 § “Twice” better than JPEG at similar complexity, only requires

Transform Timeline Haar Transform 1910 Fourier Transform 1807 Gibbs 1899 Gabor, STFT 1946 bin. DCT 1999 FFT 1965 Hadamard 1893 DCT 1974 FB 1976 H. 264 4 x 4 2001 2000 1800 Walsh 1923 1900 Pyramid 1983 Early wavelet Orthogonal FB 1984 MDCT 1987 Daubechies wavelets 1988 Multi-resolution, Fast DWT 1989 Lapped transform 1989 Lifting 1994 Wavelet-based JPEG 2000 JPEG XR LT 2006

Linear Signal Representation transform coefficient input signal basis function Analysis Synthesis dual basis function aim to use a sparse subset Approximation

Transform Fundamentals Analysis Synthesis Q u u 1 D Analysis Transform 1 D Synthesis Transform

Invertibility & Unitary u Invertibility § perfect reconstruction, bi-orthogonal, reversible u Unitarity § orthogonal, orthonormal same analysis & synthesis basis functions

DFT & WHT u Discrete Fourier Transform (DFT) u Walsh-Hadamard Transform (WHT)

Discrete Cosine Transforms u Type III u Type IV

4 x 4 Type-II DCT capturing smooth homogeneous regions representing edges & textures Houston, we have a problem!!!!

Direct Scaling Design u Approximation philosophy § Closest fixed-point realization of a well-known transform with irrational coefficients § Try to retain as many desirable properties as possible § High coding gain § Symmetry good § Orthogonality irrational-coef § Most popular: DCT approximation transform integer transform scaling factor

Direct Scaling Design: Example I u Early 4 x 4 integer transform in H. 264

Direct Scaling Design: Example II u u Interesting fact: Current H. 264 4 x 4 integer transform Fast Implementation 2 2

Lifting u Lifting – an elementary matrix operation – a shear X 1 P P X 2 u X 1 Y 2 X 2 Perfect reconstruction is guaranteed by the structure u u u P can be any scalar value P actually can be any polynomial (wavelet) P can even be any non-linear operator

Plane-Rotation and Lifting Structure u Rotation R(θ) = 3 lifting steps (shears) = u Rotation R(θ) = 2 lifting steps + 2 scaling factors =

Lifting Properties u Integer-to-integer mapping: u Enable lossless compression Ŷ 1 X 2 u [. ] [P] -[P] Ŷ 2 X 2 Fast and robust implementation u u u X 1 Multiplierless operations, lower computation complexity Robust to quantization Efficient transform design u Efficient structures with fewer degrees of freedom

Multiplierless In-place Implementation 1/2 3/4 1/2

General Lifting Parameterization u Via LDU decomposition (Gauss-Jordan Elimination)

Example: 4 x 4 bin. DCT u Just a matter of parameter choices! § u=7/16; p=3/8: Our proposal to H. 264 § u=1/2; p=2/5: Current H. 264/MPEG-4 transform § Last 2 basis functions just off by {2, 5/2} scaling § u=1/2; p=1/2: Current JPEG-XR base transform PCT § Same complexity, same coding gain as H. 264 § Lower dynamic range & allows lossless coding!

Example: 8 x 8 bin. DCT

Example: 16 x 16 bin. DCT

Block Transform Drawback u u Blocking artifacts at low bit rates Like tiling your kitchen floor! Compression ratio: 100: 1

Observation: Local or Global? u u Coding contest: can JPEG outperform JPEG 2000? Can we turn a gradient signal into a collection of piecewise-constant blocks?

Toy Example … … … Before Post-Filtering … After Post-Filtering post-filtering operator

Pre- and Post-Filtering Framework C C Pre-Filtering Block-transform Codec Post-Filtering

Pre-Filtering Effect Clockwise: original, borrow 1, 2, 3, 4. pre-filter = flattening operator post-filter = de-blocking operator high-frequency time shifting: creating piece-wise smooth signal

JPEG-XR Transform u PCT: base transform C s s u POT: overlapped operator 1/s P 1/s

Pre-/Post-Filter in VC-1 & WMV-9 u u WMV-9/VC-1 adopts a 4 x 4 pre-/post-filtering scheme VC-1: Mandatory standard for HD-DVD, Blu-ray and for Society of Motion Picture and Television Engineers (SMPTE)

Lapped Transform Connection Global Matrix Representation u LT Inverse LT E u Perfect Reconstruction Condition: G E = I G

Summary u u Review history of transform design in image/video coding applications Present a simple systematic approach to design current & future integer-coefficients transforms § Incorporate as many desirable features into the structure as possible § Lifting step is an example of such a structure § Multiplier-less solutions: shift and add operations only § Lossless integers-to-integers mapping § In-place computation

2 D Separable Transformation u 2 D Analysis Nx. N u transforming rows transforming columns 2 D Synthesis Nx. N u 2 D Orthogonal Synthesis

DCT Type-II DCT basis 8 x 8 block DC • • • orthogonal real coefficients symmetry near-optimal fast algorithms horizontal edges w lo cy n ue q fre le d id m vertical edges cy n ue q fre gh i h cy n ue q fre

Desirable Transform Properties I u Symmetry § Linear-phase basis functions § Very important in image/video processing u Energy compaction: coding gain variance of the i-th transform coefficients u Orthogonality § Mathematically elegant! § Practical codecs often emply near-orthogonal transforms

Desirable Transform Properties II allow lossless as well as lossy compression u u Integer or dyadic-rational coefficients Integer-to-integer mapping with exact invertibility Limited dynamic range extension Other practical considerations § Hardware & software friendly § Low latency, small bus width § Structure regularity

Transform Design Approach u How to construct such a transform? § Obviously try to achieve as many desirable properties as possible § Sometimes, a slight relaxation of a tight constraint can yield a remarkable coding improvement in practice (orthogonality for instance) § Trial-and-error approach: Direct scaling design § Structural approach: § § Capture structurally as many desirable properties as possible Two prominent structure types: lattice and lifting Optimize the parameters of the structure for high coding gain Approximate optimal parameters by dyadic rationals