Towards Practical Distributed Coding Bernd Girod Information Systems

Towards Practical Distributed Coding Bernd Girod Information Systems Laboratory Stanford University

Outline q Distributed lossless compression – – q Simple examples Slepian-Wolf Theorem Slepian-Wolf coding vs. systematic channel coding Turbo codes for compression Lossy compression with side information – Wyner-Ziv Theorem – Optimal quantizer design with Lloyd algorithm q Selected applications – Sensor networks – Low complexity video coding – Error-resilient video transmission B. Girod: Towards Practical Distributed Coding 2

Simple Example Source A Source B Source statistics exploited in the encoder. Different statistics Different code. B. Girod: Towards Practical Distributed Coding 3

Simple Example - Revisited Source A Source B Different statistics Same code. Source statistics exploited in the decoder. “Lossless” compression with residual error rate. B. Girod: Towards Practical Distributed Coding 4

Compression with Side Information Source A/B Encoder B. Girod: Towards Practical Distributed Coding Decoder 5

Distributed Compression of Dependent Sources Source X Encoder X Joint Decoder Source Y X Y Encoder Y B. Girod: Towards Practical Distributed Coding 6

Achievable Rates for Distributed Coding Example Separate encoding and decoding of X and Y Separate encoding and joint decoding of X and Y B. Girod: Towards Practical Distributed Coding 7

General Dependent i. i. d. Sequences [Slepian, Wolf, 1973] No errors Vanishing error probability for long sequences B. Girod: Towards Practical Distributed Coding 8

Distributed Compression and Channel Coding Source X|Y Decoder Encoder P Idea q Interpret Y as a “noisy” version of X with “channel errors” D q Encoder generates “parity bits” P to protect against errors D q Decoder concatenates Y and P and performs error-correcting decoding B. Girod: Towards Practical Distributed Coding 9

Practical Slepian-Wolf Encoding q q Coset codes [Pradhan and Ramchandran, 1999] Trellis codes [Wang and Orchard, 2001] Turbo codes [Garcia-Frias and Zhao, 2001] [Bajcsy and Mitran, 2001] [Aaron and Girod, 2002] LDPC codes [Liveris, Xiong, and Georghiades, 2002] B. Girod: Towards Practical Distributed Coding 10

Compression by Turbo Coding L bits in Systematic Convolutional Encoder Rate Interleaver length L Systematic Convolutional Encoder Rate L bits L bits [Aaron, Girod, DCC 2002] B. Girod: Towards Practical Distributed Coding 11

Turbo Decoder Pchannel bits in “Channel” probabilities calculations SISO Pa posteriori Decoder Pa priori Pextrinsic Deinterleaver length L bits in “Channel” probabilities calculations Interleaver length L Decision L bits out Pextrinsic Pa priori SISO Decoder Pchannel Interleaver length L Deinterleaver length L Pa posteriori [Aaron, Girod, DCC 2002] B. Girod: Towards Practical Distributed Coding 12

Results for Compression of Binary Sequences q q q X, Y dependent binary sequences with symmetric cross-over probabilities Rate 4/5 constituent convolutional codes; RX=0. 5 bit per input bit 0. 15 bit [Aaron, Girod, DCC 2002] B. Girod: Towards Practical Distributed Coding 13

Results with Puncturing [Aaron, Girod, DCC 2002] B. Girod: Towards Practical Distributed Coding 14

Channel Coding vs. Slepian-Wolf Coding Systematic Channel Coding q Systematic bits and parity bits subject to bit-errors Mostly memoryless BSC channel q Low bit-error rate of channel q Rate savings relative to systematic bits q B. Girod: Towards Practical Distributed Coding Slepian-Wolf Coding q No bit errors in parity bits q General statistics incl. memory q Whole range of “error” probabilities q Rate savings relative to parity bits q Might have to compete with conventional compression (e. g. , arithmetic coding) 15

Distributed Lossy Compression of Dependent Sources Source X Encoder X Achievable rate region Source Y Joint Decoder X’ Y’ Encoder Y B. Girod: Towards Practical Distributed Coding 16

Lossy Compression with Side Information Source Encoder Decoder [Wyner, Ziv, 1976] [Zamir, 1996] B. Girod: Towards Practical Distributed Coding 17

Practical Wyner-Ziv Encoder and Decoder Wyner-Ziv Encoder Quantizer Slepian. Wolf Encoder B. Girod: Towards Practical Distributed Coding Wyner-Ziv Decoder Slepian. Wolf Decoder Minimum Distortion Reconstruction 18

Non-Connected Quantization Regions q Example: Non-connected intervals for scalar quantization x q Decoder: Minimum mean-squared error reconstruction with side information x B. Girod: Towards Practical Distributed Coding 19

Quantizer Reconstruction Function B. Girod: Towards Practical Distributed Coding 20

Finding Quantization Regions 1 3 4 2 q= 1 2 3 4 1 2 B. Girod: Towards Practical Distributed Coding x 21

Lloyd Algorithm for Wyner-Ziv Quantizers Choose initial quantizers Find best reconstruction functions for current quantizers Update rate measure for current quantizers [Fleming, Zhao, Effros, unpublished] [Rebollo-Monedero, Girod, DCC 2003] Lagrangian cost for current quantizers, reconstructor and rate measure Convergence N Find best quantizers for current reconstruction and rate measure B. Girod: Towards Practical Distributed Coding Y End 22

Which Rate Measure? Wyner-Ziv Decoder Wyner-Ziv Encoder Conditional entropy coder H(Q|Y) Quantizer Slepian. Wolf Encoder Slepian. Wolf Decoder Minimum Distortion Reconstruction [Rebollo-Monedero, Girod, DCC 2003] B. Girod: Towards Practical Distributed Coding 23

Example Data set: Video sequence Carphone, 100 luminance frames, QCIF X: pixel values in even frames Y: motion-compensated interpolation from two adjacent odd frames B. Girod: Towards Practical Distributed Coding 24

Example (cont. ) Quantizer w/ rate constraint H(Q) PSNR=37. 4 d. B H(Q)=1. 87 bit H(Q|Y)=0. 54 bit B. Girod: Towards Practical Distributed Coding Quantizer w/ rate constraint H(Q|Y) PSNR=39 d. B H(Q)=3. 05 bit H(Q|Y)=0. 54 bit 25

Example (cont. ) Quantizer w/ rate constraint H(Q|Y) PSNR[d. B] Quantizer w/ rate constraint H(Q) Rate [bit] B. Girod: Towards Practical Distributed Coding Rate [bit] 26

Wyner-Ziv Quantizers: Lessons Learnt q q Typically no quantizer index reuse for rate constraint H(Q|Y) and high rates: Slepian-Wolf code provides more efficient many-to-one mapping in very high dimensional space. Uniform quantizers close to minimum m. s. e. , when combined with efficient Slepian-Wolf code Quantizer index reuse required for rate constraint H(Q) and for fixed-length coding Important to decouple dimension of quantizer (i. e. scalar) and Slepian-Wolf code (very large) B. Girod: Towards Practical Distributed Coding 27

Sensor Networks Remote Sensor Central Unit Side Information Local Sensor Remote Sensor B. Girod: Towards Practical Distributed Coding [Pradhan, Ramchandran, DCC 2000] [Kusuma, Doherty, Ramchandran, ICIP 2001] [Pradhan, Kusuma, Ramchandran, SP Mag. , 2002] [Chou, Perovic, Ramchandran, Asilomar 2002] 28

Video Compression with Simple Encoder Interframe Decoder Intraframe Encoder Video frame Scalar Quantizer Turbo Encoder Buffer Turbo Decoder Request bits X Slepian-Wolf Codec B. Girod: Towards Practical Distributed Coding X’ Y previous [Aaron, Zhang, Girod, Asilomar 2002] [Aaron, Rane, Zhang, Girod, DCC 2003] Reconstruction Key frames Interpolation next 29

Video Compression with Simple Encoder Decoder Side information B. Girod: Towards Practical Distributed Coding After Wyner-Ziv Decoding 16 -level quantization (~1 bpp) 30

Video Compression with Simple Encoder Decoder Side information B. Girod: Towards Practical Distributed Coding After Wyner-Ziv Decoding 16 -level quantization (~1 bpp) 31

Video Compression with Simple Encoder Decoder Side information B. Girod: Towards Practical Distributed Coding After Wyner-Ziv Decoding 16 -level quantization (~1 bpp) 32

Performance of Simple Wyner-Ziv Video Coder 7 d. B B. Girod: Towards Practical Distributed Coding 33

Digitally Enhanced Analog Transmission Analog Channel Side info Wyner. Ziv Encoder q q q Digital Channel Wyner. Ziv Decoder Forward error protection of the signal waveform Information-theoretic bounds [Shamai, Verdu, Zamir, 1998] “Systematic lossy source-channel coding” B. Girod: Towards Practical Distributed Coding 34

Forward Error Protection for MPEG Video Broadcasting MPEG Decoder with Error Concealment MPEG Encoder Wyner-Ziv Encoder A Error-Prone channel S Wyner-Ziv Encoder B S’ Wyner-Ziv Decoder A S* Wyner-Ziv Decoder B S** Graceful degradation without a layered signal representation B. Girod: Towards Practical Distributed Coding 35

Error-resilient Video Transmission with Embedded Wyner-Ziv Codec Carphone CIF, 50 frames @ 30 fps, 1 Mbps, 1% Random Macroblock loss [Aaron, Rane, Rebollo-Monedero, Girod, ICIP 2003] B. Girod: Towards Practical Distributed Coding 36

Towards Practical Distributed Coding: Why Should We Care? q Chance to reinvent compression from scratch – – – q Entropy coding Quantization Signal transforms Adaptive coding Rate control. . . Enables new compression applications – – – Sensor networks Very low complexity encoders Error-resilient transmission of signal waveforms Digitally enhanced analog transmission Unequal error protection without layered coding. . . B. Girod: Towards Practical Distributed Coding 37

The End