Introduction to JPEG and MPEG Ingemar J Cox

Introduction to JPEG and MPEG Ingemar J. Cox University College London

Outline q Elementary information theory q Lossless compression UCL Adastral Park Postgraduate Campus q Quantization q Fundamentals of images q Discrete Cosine Transform (DCT) q JPEG q MPEG-1, MPEG-2 Nov 27 th 2006 Ingemar J. Cox 2

Bibliography q D. Mac. Kay, “Information Theory, Inference and learning Algorithms”, Cambridge University Press, 2003. http: //www. inference. phy. cam. ac. uk/itprnn/book. html UCL Adastral Park Postgraduate Campus q W. B. Pennebaker and J. L. Mitchell, “JPEG Still Image Data Compression Standard”, Chapman Hall, 1993 (ISBN 0 -442 -01272 -1). q G. K. Wallace, “The JPEG Still-Picture Compression Standard”, IEEE Trans. On Consumer Electronics, 38, 1, 18 -34, 1992. q http: //en. wikipedia. org/wiki/JPEG Nov 27 th 2006 Ingemar J. Cox 3

Bibliography q http: //en. wikipedia. org/wiki/MPEG-2 Nov 27 th 2006 Ingemar J. Cox UCL Adastral Park Postgraduate Campus q T. Sikora, “MPEG Digital Video-Coding Standards”, IEEE Signal Processing Magazine, 82 -100, September 1997 4

Elementary Information Theory

Elementary Information Theory q How much information does a symbol convey? UCL Adastral Park Postgraduate Campus q Intuitively, the more unpredictable or surprising it is, the more information is conveyed. q Conversely, if we strongly expected something, and it occurs, we have not learnt very much Nov 27 th 2006 Ingemar J. Cox 6

Elementary Information Theory q If p is the probability that a symbol will occur UCL Adastral Park Postgraduate Campus q Then the amount of information, I, conveyed is: q The information, I, is measured in bits q It is the optimum code length for the symbol Nov 27 th 2006 Ingemar J. Cox 7

Elementary Information Theory UCL Adastral Park Postgraduate Campus q The entropy, H, is the average information per symbol q Provides a lower bound on the compression that can be achieved Nov 27 th 2006 Ingemar J. Cox 8

Elementary Information theory q A simple example. Suppose we need to transmit four possible weather conditions: Sunny Cloudy Rainy Snowy UCL Adastral Park Postgraduate Campus 1. 2. 3. 4. q If all conditions are equally likely, p(s)=0. 25, and H=2 Ø i. e. we need a minimum of 2 bits per symbol Nov 27 th 2006 Ingemar J. Cox 9

Elementary information theory q Suppose instead that it is: Sunny 0. 5 of the time Cloudy 0. 25 of the time Rainy 0. 125 of the time, and Snowy 0. 125 of the time UCL Adastral Park Postgraduate Campus 1. 2. 3. 4. q Then the entropy is Nov 27 th 2006 Ingemar J. Cox 10

Elementary Information Theory q Variable length codewords q Huffman code – integer code lengths Nov 27 th 2006 Ingemar J. Cox UCL Adastral Park Postgraduate Campus q Arithmetic codes – non-integer code lengths 11

Elementary Information Theory q Huffman code Nov 27 th 2006 Probability Information Integer code Sunny 0. 5 1 0 Cloudy 0. 25 2 10 Rainy 0. 125 3 110 Snowy 0. 125 3 111 Ingemar J. Cox UCL Adastral Park Postgraduate Campus Weather 12

Elementary Information Theory q Previous illustration is an example of a lossless code UCL Adastral Park Postgraduate Campus Ø I. e. we are able to recover the information exactly Nov 27 th 2006 Ingemar J. Cox 13

Elementary Information Theory q Note that we have assumed that each symbol is independent of the other symbols Nov 27 th 2006 Ingemar J. Cox UCL Adastral Park Postgraduate Campus Ø I. e. the current symbol provides no information regarding the next symbol 14

Quantization UCL Adastral Park Postgraduate Campus q Quantization is the process of approximating a continuous (or range of values) by a (much) smaller range of values q Where Round(y) rounds y to the nearest integer q is the quantization stepsize Nov 27 th 2006 Ingemar J. Cox 15

Quantization q Example: =2 Nov 27 th 2006 -4 -3 -2 -1 0 1 2 3 4 -2 -1 0 1 2 -4 -2 0 2 4 Ingemar J. Cox 5 UCL Adastral Park Postgraduate Campus -5 16

Quantization q Quantization plays an important role in lossy compression UCL Adastral Park Postgraduate Campus Ø This is where the loss happens Nov 27 th 2006 Ingemar J. Cox 17

Fundamentals of Images

Fundamentals of images q An image consists of pixels (picture elements) q Each pixel represents luminance (and colour) Nov 27 th 2006 Ingemar J. Cox UCL Adastral Park Postgraduate Campus Ø Typically, 8 -bits per pixel 19

Fundamentals of images q Colour Ø Colour spaces (representations) • • • UCL Adastral Park Postgraduate Campus v RGB (red-green-blue) v CMY (cyan-magenta-yellow) v YUV Y = 0. 3 R+0. 6 G+0. 1 B (luminance) U=R-Y V=B-Y q Greyscale q Binary Nov 27 th 2006 Ingemar J. Cox 20

Fundamentals of images q A TV frame is about 640 x 480 pixels UCL Adastral Park Postgraduate Campus q If each pixels is represented by 8 -bits for each colour, then the total image size is Ø 640× 480*3=921, 600 bytes or 7. 4 Mbits q At 30 frames per second, this would be Ø 220 Mbits/second Nov 27 th 2006 Ingemar J. Cox 21

Fundamentals of images UCL Adastral Park Postgraduate Campus q Do we need all these bits? Nov 27 th 2006 Ingemar J. Cox 22

Fundamentals of images UCL Adastral Park Postgraduate Campus q Here is an image represented with 8 -bits per pixel Nov 27 th 2006 Ingemar J. Cox 23

Fundamentals of images UCL Adastral Park Postgraduate Campus q Here is the same image at 7 -bits per pixel Nov 27 th 2006 Ingemar J. Cox 24

Fundamentals of images UCL Adastral Park Postgraduate Campus q And at 6 -bits per pixel Nov 27 th 2006 Ingemar J. Cox 25

Fundamentals of images UCL Adastral Park Postgraduate Campus q And at 5 -bits per pixel Nov 27 th 2006 Ingemar J. Cox 26

Fundamentals of images UCL Adastral Park Postgraduate Campus q And at 4 -bits per pixel Nov 27 th 2006 Ingemar J. Cox 27

Fundamentals of images q Do we need all these bits? Ø No! UCL Adastral Park Postgraduate Campus q The previous example illustrated the eye’s sensitivity to luminance q We can build a perceptual model Ø Only code what is important to the human visual system (HVS) v Usually a function of spatial frequency Nov 27 th 2006 Ingemar J. Cox 28

Fundamentals of Images q Just as audio has temporal frequencies q Images have spatial frequencies Ø Ø Nov 27 th 2006 UCL Adastral Park Postgraduate Campus q Transforms Fourier transform Discrete cosine transform Wavelet transform Hadamard transform Ingemar J. Cox 29

Discrete cosine transform UCL Adastral Park Postgraduate Campus q Forward DCT q Inverse DCT Nov 27 th 2006 Ingemar J. Cox 30

Basis functions UCL Adastral Park Postgraduate Campus q DC term Nov 27 th 2006 Ingemar J. Cox 31

Basis functions UCL Adastral Park Postgraduate Campus q First term Nov 27 th 2006 Ingemar J. Cox 32

Basis functions UCL Adastral Park Postgraduate Campus q Second term Nov 27 th 2006 Ingemar J. Cox 33

Basis functions UCL Adastral Park Postgraduate Campus q Third term Nov 27 th 2006 Ingemar J. Cox 34

Basis functions UCL Adastral Park Postgraduate Campus q Fourth term Nov 27 th 2006 Ingemar J. Cox 35

Basis functions UCL Adastral Park Postgraduate Campus q Fifth term Nov 27 th 2006 Ingemar J. Cox 36

Basis functions UCL Adastral Park Postgraduate Campus q Sixth term Nov 27 th 2006 Ingemar J. Cox 37

Basis functions UCL Adastral Park Postgraduate Campus q Seventh term Nov 27 th 2006 Ingemar J. Cox 38

DCT Example

Example UCL Adastral Park Postgraduate Campus q Signal Nov 27 th 2006 Ingemar J. Cox 40

Example q DCT coefficients are: Nov 27 th 2006 4. 2426 0 -3. 1543 0 0 0 -0. 2242 0 UCL Adastral Park Postgraduate Campus Ø Ø Ø Ø Ingemar J. Cox 41

Example: DCT decomposition UCL Adastral Park Postgraduate Campus q DC term Nov 27 th 2006 Ingemar J. Cox 42

Example: DCT decomposition UCL Adastral Park Postgraduate Campus q 2 nd AC term Nov 27 th 2006 Ingemar J. Cox 43

Example: DCT decomposition UCL Adastral Park Postgraduate Campus q 6 th AC term Nov 27 th 2006 Ingemar J. Cox 44

Example: summation of DCT terms UCL Adastral Park Postgraduate Campus q First two non-zero coefficients Nov 27 th 2006 Ingemar J. Cox 45

Example: summation of DCT terms UCL Adastral Park Postgraduate Campus q All 3 non-zero coefficients Nov 27 th 2006 Ingemar J. Cox 46

Example q What if we quantize DCT coefficients? Ø =1 q Quantized DCT coefficients are: Nov 27 th 2006 4 0 -3 0 0 0 UCL Adastral Park Postgraduate Campus Ø Ø Ø Ø Ingemar J. Cox 47

Example UCL Adastral Park Postgraduate Campus q Approximate reconstruction Nov 27 th 2006 Ingemar J. Cox 48

Example UCL Adastral Park Postgraduate Campus q Exact reconstruction Nov 27 th 2006 Ingemar J. Cox 49

2 -D DCT Transform q Let i(x, y) represent an image with N rows and M columns UCL Adastral Park Postgraduate Campus q Its DCT I(u, v) is given by q where Nov 27 th 2006 Ingemar J. Cox 50

Fundamentals of images q Discrete cosine transform Ø Coefficients are approximately uncorrelated UCL Adastral Park Postgraduate Campus v Except DC term v C. f. original 8× 8 pixel block Ø Concentrates more power in the low frequency coefficients Ø Computationally efficient q Block-based DCT Ø Compute DCT on 8× 8 blocks of pixels Nov 27 th 2006 Ingemar J. Cox 51

Fundamentals of images UCL Adastral Park Postgraduate Campus q Basis functions for the 8× 8 DCT (courtesy Wikipedia) Nov 27 th 2006 Ingemar J. Cox 52

Fundamentals of JPEG

Fundamentals of JPEG Encoder Quantizer Entropy coder UCL Adastral Park Postgraduate Campus DCT Compressed image data IDCT Dequantizer Entropy decoder Decoder Nov 27 th 2006 Ingemar J. Cox 54

Fundamentals of JPEG q JPEG works on 8× 8 blocks q Extract 8× 8 block of pixels UCL Adastral Park Postgraduate Campus q Convert to DCT domain q Quantize each coefficient Ø Different stepsize for each coefficient v Based on sensitivity of human visual system q Order coefficients in zig-zag order q Entropy code the quantized values Nov 27 th 2006 Ingemar J. Cox 55

Fundamentals of JPEG Nov 27 th 2006 16 11 10 16 24 40 51 61 12 12 14 19 26 58 60 55 14 13 16 24 40 57 69 56 14 17 22 29 51 87 80 62 18 22 37 56 68 109 103 77 24 35 55 64 81 104 113 92 49 64 78 87 103 121 120 101 72 92 95 98 112 100 103 99 Ingemar J. Cox UCL Adastral Park Postgraduate Campus q A common quantization table is 56

Fundamentals of JPEG Nov 27 th 2006 0 1 5 6 14 15 27 28 2 4 7 13 16 26 29 42 3 8 12 17 25 30 41 43 9 11 18 24 31 40 44 53 10 19 23 32 39 45 52 54 20 22 33 38 46 51 55 60 21 34 37 47 50 56 59 61 35 36 48 49 57 58 62 63 Ingemar J. Cox UCL Adastral Park Postgraduate Campus q Zig-zag ordering 57

Fundamentals of JPEG q Entropy coding Ø Run length encoding followed by UCL Adastral Park Postgraduate Campus v Huffman v Arithmetic q DC term treated separately Ø Differential Pulse Code Modulation (DPCM) q 2 -step process 1. Convert zig-zag sequence to a symbol sequence 2. Convert symbols to a data stream Nov 27 th 2006 Ingemar J. Cox 58

Fundamentals of JPEG q Modes Ø Sequential Ø Progressive • UCL Adastral Park Postgraduate Campus v Spectral selection Send lower frequency coefficients first v Successive approximation • Send lower precision first, and subsequently refine Ø Lossless Ø Hierarchical v Send low resolution image first Nov 27 th 2006 Ingemar J. Cox 59

Fundamentals of MPEG-1/2

Fundamentals of MPEG q A sequence of 2 D images UCL Adastral Park Postgraduate Campus q Temporal correlation as well as spatial correlation q TV broadcast Ø Frame-based Ø Field-based Nov 27 th 2006 Ingemar J. Cox 61

MPEG q Moving Picture Experts Group q Standard for video compression Nov 27 th 2006 Ingemar J. Cox UCL Adastral Park Postgraduate Campus q Similarities with JPEG 62

MPEG q Design is a compromise between Nov 27 th 2006 Ingemar J. Cox UCL Adastral Park Postgraduate Campus Ø Bit rate Ø Encoder/decoder complexity Ø Random access capability 63

MPEG q Images UCL Adastral Park Postgraduate Campus Ø Spatial redundancy Ø Perceptual redundancy q Video Ø Spatial redundancy v Intraframe coding Ø Temporal redundancy v Interframe coding Ø Perceptual redundancy Nov 27 th 2006 Ingemar J. Cox 64

MPEG q Consider a sequence of n frames of video. q It consists of: UCL Adastral Park Postgraduate Campus Ø I-frames Ø P-frames Ø B-frames q A sequence of one I-frame followed by P- and B -frames is known as a GOP Ø Group of Pictures Ø E. g. IBBPBBP Nov 27 th 2006 Ingemar J. Cox 65

MPEG q I-frames Ø Intraframe coded v No motion compensation UCL Adastral Park Postgraduate Campus q P-frames Ø Interframe coded v Motion compensation • Based on past frames only q B-frames Ø Interframe coded v Motion compensation • Nov 27 th 2006 Based on past and future frames Ingemar J. Cox 66

MPEG q Motion-compensated prediction Nov 27 th 2006 Ingemar J. Cox UCL Adastral Park Postgraduate Campus Ø Divide current frame, i, into disjoint 16× 16 macroblocks Ø Search a window in previous frame, i-1, for closest match Ø Calculate the prediction error Ø For each of the four 8× 8 blocks in the macroblock, perform DCT-based coding Ø Transmit motion vector + entropy coded prediction error (lossy coding) 67

MPEG q Like JPEG, the DC term is treated separately Ø DPCM UCL Adastral Park Postgraduate Campus q B-frame compression high Ø Need buffer and delay Nov 27 th 2006 Ingemar J. Cox 68