Theory of Information Lecture 1 Introduction and preliminaries

Theory of Information Lecture 1 Introduction and preliminaries (Chapter 0) 1

Introduction Subject: Encoding data for transmission Transmission: • “from here to there” (i. e. , sending a message from one place to another) • “from now to later” (i. e. , storing data for later retrieval) 2

Introduction Data encoding – representing data with strings of symbols of another alphabet (mostly, {0, 1}) 3 reasons to encode data: • for efficiency (information theory) • for error detection and/or correction (coding theory) • for secrecy (cryptology) Such encodings are usually done separately: compression → encryption → adding bits for error correction 3

Introduction Example of compression. How many bits would it take to encode a 100 -character English word? Now assume character “a” appears 50% of the time. How many bits would it take to encode a 100 -character English word? 4

Introduction Example of error detection. parity bit 00100011001011011 1 If errors occur, what is the probability of noticing them? How about if we used 2 parity bits? How about if we used 10 parity bits? 5

Sets: {aa, bab}, {2, 4, 6, …} etc. Zn= {0, 1, …, n-1} , or {} --- the empty set |S| --- the size (number of elements) of S b S --- b is an element of S S T --- S is a subset of T. Also, T is a superset of S. Sc --- the complement of S S T --- the union of S and T S T --- the intersection of S and T When S T= , we say that S and T are disjoint Set notation examples: {a | a is an even integer} {2 k | k is an integer} 6

Summation Notation If s 1, …, sn are numbers or algebraic expressions, their sum is denoted by n sk k=1 3 2 k = k=1 21+22+23=14 3 2 k = k=0 20+21+22+23=15 4 2 k = k=2 22+23+24=28 7

Permutations A permutation of set S has two meanings: • A bijective function from S to S, or • An ordered arrangement of S Example: Let S={a, b, c, d} A functional permutation of S: f(a)=c, f(b)=b, f(c)=d, f(d)=a An arrangemental permutation of S: c, b, d, a, or just cbda 8

Permutations How many permutations do the following sets have? {a} {a, b, c} How many permutations does {1, 2, 3, 4, 5} have? possibilities for the 1 st element, possibilities for the 2 nd element, possibilities for the 3 rd element, possibilities for the 4 th element, possibility for the 5 th element Generally, an n-element set has permutations 9

Permutations A permutation of size k of S = = a permutation of a k-element subset of S How many permutations of size 3 does {1, 2, 3, 4, 5, 6, 7} have? possibilities for the 1 st element, possibilities for the 2 nd element, possibilities for the 3 rd element, Generally, an n-element set has permutations of size k 10

Binomial Coefficients Let 0 k n. The binomial coefficient, read as “n choose k”, is defined by: () n k n! = -------k!(n-k)! THEOREM 0. 1. 1 A set S of size n has precisely (n ) subsets of size k. k I. e. (nk ) is the number of ways of choosing k elements from n elements. Why? Fact: Why? () n k =(n-k) n 11

Homework 1. 1. What are three reasons for encoding data? What are the names of the corresponding three disciplines/theories? 1. 2. Write all of the subsets of the set {a, b, c}. 1. 3. True or false: a) b {a, b, c} b) b {a, b, c} c) {a, c} {a, b, c} d) {a} e) {b, a} {a, b} 1. 4. a) {a, b, c} {b, d}= b) {a, b, c} {b, d}= c) Are {a, b, c} and {b, d} disjoint? 4 d) (3 k+1) = k=2 12

Homework 1. 5. Write all the permutations of {x, y, z, t}. 1. 6. How many permutations does the set {0, 1, …, 9} have? 1. 7. Write all permutations of size 2 of the set {a, b, c, d}. 1. 8. How many permutations of size 6 does the set {0, 1, …, 9} have? 1. 9. Imagine a language that uses the 26 English characters, and where no word can contain the same character more than once. At most how many 4 -character words could such a language have? 1. 10. By definition, ( )= n k 1. 11. Your class has 20 students. The teacher needs to choose 5 of them for a special assignment. How many possibilities does the teacher have? 13