Theory of Information Lecture 6 Instantaneous Codes and

Theory of Information Lecture 6 Instantaneous Codes and Kraft’s Theorem (Section 1. 4) 1

Instantaneous Codes DEFINITION A code is said to be instantaneous if, whenever any sequence of codewords is transmitted, each codeword can be interpreted as soon as it is received. Example. {0, 01} {0, 10} 2

Prefix and Suffix Properties DEFINITION 1) A code is said to have the prefix property if no codeword is a prefix of any other codeword. 2) A code is said to have the suffix property if no code word is a suffix of any other codeword. Fact: Having the suffix or prefix property is sufficient to be uniquely Decipherable. But not vice versa. Does {0, 01} have the prefix property? suffix property? Is it uniquely decipherable? Does {00, 001011, 11} have the prefix property? suffix property? Is it uniquely decipherable? 3

Prefix Property = Instantaneousity THEOREM 1. 4. 1 A code is instantaneous if and only if it has the prefix property. Comma code: {0, 110, 1110} has the prefix property and is instantaneous. How about {0, 011, 0111}: Prefix property? Instantaneous? Suffix property? Uniquely decipherable? 4

Kraft’s Theorem THEOREM 1. 4. 2 1. There exists an instantaneous r-ary code with codeword lengths a 1, a 2, …, aq if and only if Kraft’s inequality is satisfied: 1/ra 1 + … + 1/raq 1. 2. Let C be an instantaneous r-ary code. Then C is maximal instantaneous, i. e. C is not contained in any strictly larger instantaneous code, if and only if equality holds in Kraft’s inequality. 3. Suppose that C is an instantaneous code with maximum codeword Length m. If C is not maximal, then it is possible to add a word of length m to C without destroying its property of being instantaneous. 5

Kraft’s Theorem THEOREM 1. 4. 2 1. There exists an instantaneous r-ary code with codeword lengths a 1, a 2, …, aq if and only if Kraft’s inequality is satisfied: 1/ra 1 + … + 1/raq 1. Note: That a given code C satisfies Kraft’s inequality does not necessarily mean that C is instantaneous; maybe another code D --- with the same codeword lengths --- is instantaneous rather than C itself. Example: C={0, 11, 100, 110} Does C satisfy Kraft’s inequality? Is C instantaneous? However, D={0, 11, 100} has the same codeword lengths and Is instantaneous. 6

The Utility of Kraft’s Theorem Kraft’s theorem allows us to find an instantaneous code (if such exists) with given codelengths a 1 a 2 … aq as follows: For i=1 to i=q do: Pick any codeword of length ai such that no earlier codeword is its prefix Example: Construct a ternary code {c 1, c 2, c 3, c 4, c 5, c 6} with codelengths 1, 1, 2, 4, 4, 5 Kraft’s test to see if such a code exists: c 1= c 2= c 3= c 4= c 5= c 6= 7

Reasonable Uniquely Decipherable Codes Are Instantaneous THEOREM 1. 4. 3. If a uniquely decipherable code exists with codeword lengths l 1, l 2, …, ln, then an instantaneous code must also exist with these same codeword lengths. Proof: Suppose the above uniquely decipherable code exists. Then, by Mc. Millan’s Theorem, Kraft’s inequality should be satisfied for those lengths. But then, by Kraft’s Theorem, there must also exist an Instantaneous code with the same codelengths. 8

Homework Exercises 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14 of Section 1. 4. 9