ICS 804 Theory of Computation Ibrahim Otieno iotienouonbi

: : ICS 804: : Theory of Computation - Ibrahim Otieno iotieno@uonbi. ac. ke +254 -0722 -429297 SCI/ICT Building Rm. G 15

ICS 804

ICS 804 - Delivery ± 60 contact hours 4. 5 hours a week Tuesday 7: 00 8: 30 Thursday 5: 30 8: 30 Assessment CAT Assignment Exams Theoretical course, but Interactivity Exercises (reflect exam questions) Laboratory sessions

ICS 804 - Delivery Online access to course materials: ◦ Notes ◦ Slides ◦ Assignments http: //learning. uonbi. ac. ke Login: ◦ Username: registration number (P 58/10202/2010) ◦ Password: msc

ICS 804 – Course Text

Introduction Computation from the point of view of mathematics (discrete mathematics) Introduction to computability and complexity theory What kind of problems are inherently computable and which options are there? Regular languages and finite-state automata Very recent field started in the 1930 s

Course Outline Mathematical Preliminaries Turing Machines Recursion Theory Markov Algorithms Register Machines Regular Languages and finite-state automata Aspects of Computability

Course Outline Mathematical Preliminaries Turing Machines Recursion Theory Markov Algorithms Register Machines Regular Languages and finite-state automata Aspects of Computability

Discrete Mathematics Theory of computability = new branch of mathematics Presupposes background in discrete mathematics Topics: ◦ Set-theoretic concepts ◦ Formal language theory ◦ Functions ◦ Big-O notation ◦ Propositional logic ◦ Proof Techniques ◦ Number-theoretic predicates

Sets

Sets Elements of a set: may be physical objects Set: never physical – is an abstract concept a S – a member of set S Members of sets may be sets S = {0, 1, {0, 1}} Cardinality: card(S) = 3 Singleton: S = {a} – set containing single element Sets lists (unordered, elements are unique) Set = family = class

Specifying (Naming) Sets Finite sets {a, e, i, o, u} – elements can be listed Infinite sets: come from mathematics ◦ enumerate elements in an extended sense ◦ ◦ Natural numbers N = { 0, 1, 2, …} Integers Z = {…, -2, -1, 0, 1, 2, …} Set abstraction ◦ ◦ ◦ {n|n is prime} or {n: n is prime} {n|…n…} or {n: …n…}: abstraction operator: the set of all n such that n… {n|n N & 2|n}: all even natural numbers

Specifying (Naming) Sets … By convention: ◦ Use variables n, m, k, j, …: natural numbers / integers ◦ Use variables x, y, z, …: rational/real numbers ◦ Set variables: uppercase letters (A, B, C, …S) ◦ Universal set: U – all elements in current domain U by set abstraction {x|x=x} ◦ Empty Set: - with no element by set abstraction {x|x x}

Subsets A = {0, 1, 2, 3, 4} A is subset of N or A N A is a subset of B ◦ If for any x, x A implies x B Consequences: ◦ ◦ Every set is a subset of itself i. e. A A, for any set A “proper subset”: non-identical subset {1, 2, 3} {1, 2, 3, 4} Power set – all subsets of a set ◦ ◦ ◦ For a set S: P(S) = { A|A S } What is the power set of S = {1, 2, 3} ? Card(P(S) = 2 card(s) for a finite set S

Tuples ◦ ◦ ◦ A tuple is an ordered list of elements Ordered pair: <a, b> = <c, d> iff a = c and b = d defined Ordered triple: <a, b, c> 0 -tuple: A = {<>} 1 -tuple: A = <a> For any two sets A and B: are said to be disjoint if A B = Card(A B) Card(A) + Card(B) Card(A B) = Card(A) + Card(B) iff A B = Compliment of B relative to A denoted AB is defined as {x| x A and x B} We write BC and speak of compliment of B for the set UB Symmetric difference denoted by A B is as {x| (x A and x B) or (x B and x A) }

Some set properties A ∪ U = U A ∩ = C = U UC = Idempotent Law: For any set A, ◦ A∪A=A ◦ A∩A=A Identity Law: For any set A, ◦ A∪ =A ◦ A∩U=A

Some set properties Commutative Law: For any two sets A and B ◦ A∪B=B∪A ◦ A∩B=B∩A Associative Law: For any three sets A, B and C ◦ (A ∪ B) ∪ C = A ∪ (B ∪ C) ◦ A ∩ (B ∩ C) = (A ∩ B) ∩ C Distributive Law: For any three sets A, B and C ◦ A ∪ (B ∩ C) = (A ∪ B) ∩ (A ∪ C) ◦ A ∩ (B ∪ C) = (A ∩ B) ∪ (A ∩ C) De Morgan’s Law: For any two sets A and B ◦ (A ∪ B)' = A' ∩ B' ◦ (A ∩ B)' = A' ∪ B'

Special Sets N: set of natural numbers closed under addition and multiplication not closed under subtraction nor division Z: set of integers closed under addition, subtraction and multiplication Q: set of rational numbers Q = {x|x = p/q for some p, q Z with q 0} closed under addition, subtraction and multiplication Q{0} is closed under division R: set of real numbers closed under addition, subtraction and multiplication R{0} closed under division We define a partition of a set A to be a family of mutually disjoint subsets such that A = A 1 ∪ A 2 …. An

Exercises Exercise 1 If A and B are two sets, then A ∩ (A ∪ B) equals ◦ (A) A (B) (C) (D) none of these Exercise 2 The set (A ∪ B ∪ C) ∩ (A ∩ B' ∩ C')' ∩ C' is equal to ◦ (A) B ∩ C' these (B) A ∩ C (C) B' ∩C' (D) None of

Formal Language Theory

Formal Language Formal language: specific kind of set Formal = not natural language, programming language, … but: NLP try to formalize natural language Ignoring semantics: ◦ possible to regard natural languages like Spanish and programming languages such as PROLOG as formal

Formal Language Alphabet : finite set of symbols, e. g. = {a, b} Word over alphabet : finite string of elements of allowing repetition Length of word w = |w| = number of symbol occurrences in w Card( ) = m then there exist mn distinct words over of length n ◦ For alphabet = {a, b} we have 2 n distinct words over of length n

Formal Language Special superscript notation: anbm word consisting of n times a and m times b Example: a 3 b 4 a 2 aaabbbbaa Distinct occurrences: na(aba)=2 nb(aba)=1

Formal Language Null word: word of length 0 ◦ ◦ ◦ Zero occurrences of a and b = a 0 na( ) = 0 and nb( ) = 0 Universal Language: * set of all words over Language over : any subset of * ◦ ◦ ◦ Empty language: - containing no word Universal language: * - containing all words Unit language L = {w} – containing word, w, only Alphabet = {a, b} ◦ {w *: |w| = 3} = aaa, aab, aba, abb, baa, bab, bba, bbb}

Operations on words Concatenation: ww’ of words w and w’ w is aba and w’ is bba ww’ = ababba Reverse of a word: w. R abb. R = bba Palindromes over is the language {w *|where w=w. R}

Palindrome Examples: Radar – level – mum – gig “Madam, in Eden, I’m Adam” “A man, a plan, a canal: Panama” “Doc, note. I dissent. A fast never prevents a fatness. I diet on cod. ” (Peter Hilton) Longest 1 -word English palindrome: detartrated - (to remove tartrates) Longest 1 -word palindrome in the world (Finnish) saippuakivikauppias - (lye dealer) Swahili palindromes: amesema, ataufuata, akatutaka Longest palindrome: 31358 words

Operations on words A word w’ is a prefix of a word w if w can be written w’w’’ for some word w’’ ◦ , a, abb, abba are prefixes of the word abba (first 4 are proper) A word w’’ is a suffix of a word w if w can be written w’w’’ for some word w’ ◦ , a, bba, abba are suffixes of the word abba (first 4 are proper)

Operations on words A word w’’ is a substring of word w if w can be written as w’w’’w’’’ for some words w’ and w’’ abba: a ab bb ba abb bba abba

Exercises � � � 1 = {a, b, c} 2 = {a, b, ca} 3 = {a, b, Ab} Determine to which * each word belongs aba b. Ab cba cab caab ba. Ab

Solution 1 = {a, b, c} 2 = {a, b, ca} 3 = {a, b, Ab} Determine to which * each word belongs aba 1, 2, 3 b. Ab 3 cba 1 cab 1, 2 caab 1, 2 ba. Ab 3

Exercises Calculate how many words there are in the following languages. List 3 elements in each of them. Which language contains ◦ ◦ *, where = {a, b, c} {w *| |w| 3}, where = {a, b} {w *| |w| =4}, where = {a, b} {anb|n is prime}

Solution *, where = {a, b, c} Infinite number of words { , abba, ccccba, aaccccc, …} {w *| |w| 3}, where = {a, b} 15 words (23+22+21+20) = { , a, b, ab, ba, bb, aaa, baa, aba, aab, bba, bab, abb, bbb} {w *| |w| =4}, where = {a, b} 16 words (24) = {aaaa, aaab, aaba, abaa, baaa, aabb, abba, bbaa, baba, abab, baab, abbb, babb, bbab, bbba, bbbb} {anb|n is prime} Infinite number of words {aab, aaaaab, …}

Exercises Let = {a, b, c, d}. List as many words as you can that are words in the language {w *| |w| = 3 and w is a word in the English language}

Solution Let = {a, b, c, d}. List as many words as you can that are words in the language {w *| |w| = 3 and w is a word in the English language} dad cab bad dab add cad baa

Exercises Suppose w is ab and w’ is bab. Identify each of the following words: ww’ w. R (ww’)R ww’R

Solution Suppose w is ab and w’ is bab. Identify each of the following words: ww’ abbab w. R ba (ww’)R babba ww’R abbab

Exercise Which of the following languages are palindrome languages ◦ ◦ ◦ ww. R for any w * w. R for any palindrome w * ww for any palindrome w * waw. R for any w * ww’w. R for any w * and any palindrome w’ *

Solution Which of the following languages are palindrome languages ◦ ww. R for any w * yes ◦ w. R for any palindrome w * yes ◦ ww for any palindrome w * yes ◦ waw. R for any w * yes ◦ ww’w. R for any w * and any palindrome w’ * yes ◦ yes

Mapping and Functions

Mapping and Functions Mapping: association of members of one set with members of another (not necessarily different) set First: domain Second: codomain or range Members of domain serve as: arguments Members of codomain serve as: values Mapping f between Dom(f)=A and Cod(f)=B f: A B A mapping f is a function if each member of Dom(f) is mapped to one and only one member of Cod(f)

Function = single valued mapping Complete description of a mapping f consists of: a) specification of Dom(f) and Cod(f) b) description of values of f for any member of Dom(f) taken as arguments Examples f: N N f(n) = n + 1 Unary constant-7 function – returns 7 for all n C 17: N N C 17(n) = 7 f: * * f(w) = w. R

Functions Examples …continued Floor function f: R Z f(x) = [x] = greatest integer ≤ x Ceiling function f: R Z f(x) = [x] = least integer ≥ x Multi-valued mapping ◦ f: Z Z f(n) = n ± 3 f(n) = n 3 Not a function

Functions Image of a function f defined as Image(f) is subset of Cod(f) consisting precisely of the values of f for members of Dom(f) Precisely in set language: Image(f) = {y Cod(f)|y=f(x) for some x Dom(f)}

Number-theoretic Functions Number-theoretic functions: map natural numbers to natural numbers: f: N N Unary number-theoretic functions f(n) = n + 1 Binary number-theoretic functions f: N 2 N f(n, m) = n + m K-ary number-theoretic functions f: N k N 0 -ary number theoretic functions C 02: N 0 N C 02() = 2

Injective (1 -to-1) Functions A function f is 1 -to-1 or injective if no two distinct elements of Dom(f) are mapped to one and the same member of the Cod(f) Examples: Successor function f: N N f(n) = n + 1 Word reversal function f: * * f(w) = w. R Ceiling, floor and constant-7 functions not injective

Surjective (onto) Functions A function f is onto or surjective provided that every element of Cod(f) is a value of f for at least one element of the Dom(f) Cod(f) = Image (f) Examples: Word reversal function f: * * f(w) = w. R Ceiling function f: R Z f(x) = [x] = least integer ≥ x Successor function not surjective – 0 not successor A function that is both injective and surjective is termed as 1 -1 correspondence or a bijective function

Functions forming operations

Inverse For two functions g and f: g: N N g(n) = n/2 f: N N f(n) = 2 n g is the inverse of f g = f-1 The inverse operation (though not always defined) -1 is; Unary function forming operation

Composition For two functions g and f: g: * N where ={a, b} g(w) = na(w) f: N N f(n) = 3 n + 2 h: * N h(w) = f(g(w)) h: composition of g and f (h = g f) The composition operation is: Binary function forming operation

Partial Functions For a function g: g: N 2 N g(n, m) = n – m The function g is undefined for pairs <n, m> where n < m Number-theoretic functions that are undefined for some n N are Partial functions.

Partial Functions Let f(n) and g(n) be partial number-theoretic functions We can define a new partial number-theoretic function h(n) = max{ f(n), g(n) } This means that h(n) is defined just in case both f(n) and g(n) are defined for any n Dom(h) = Dom(f) Dom(g), we have h(n) is the maximum of f(n) and g(n)

Polynomial Functions We restrict ourselves to unary polynomial functions Polynomial in n of degree k is any expression of form: aknk + ak-1 nk-1 + ak-2 nk-2 + …. . + a 1 n 1 + a 0 n 0 where k is any natural constant and coefficients ak , ak-1 , ak-2 , …. . , a 1 + a 0 are integer constants with ak ≠ 0 Polynomials of degree 0 are numerals Polynomials in n are not themselves natural numbers or integers; they are expressions

Polynomial Functions For example: 2 n 5 + 2 n 3 + 4 n 2 +16 n + 7 is a polynomial of degree 5 The unary polynomial function is defined by: f: N N f(n) = 2 n 5 + 2 n 3 + 4 n 2 +16 n + 7 f(2) = ? Polynomial functions with negative coefficients may be partial Sometimes we write p(n) for a unary polynomial function of n For p 1(n) and p 2(n) => p 2[p 1(n)]) – result of replacing every occurrence of n in p 2(n) with p 1(n)

Big-O Notation

Big-O (micron) notation provides standard means of comparing growth of number-theoretic functions Analyze time and space requirements of algorithms Typical calculation: growth of a function as argument n goes to infinity Monotone increasing: for all n 1, n 2 N, we have n 1< n 2 implies f(n 1) f(n 2) Strictly monotone increasing: f(n 1)<f(n 2)

Big-O Four partial number-theoretic functions: k(n) = 2 n h(n) = n 2 + 1 g(n) = 3 n f(n) = 7[log 2 n]

Big-O

Big-O Which are (strictly) monotone increasing ?

Big-O Four partial number-theoretic functions k(n) = 2 n h(n) = n 2 + 1 g(n) = 3 n f(n) = 7[log 2 n] Value 2: k(2) = 4 h(2)=5 g(2) = 6 Value 5: k(5) = 32 h(5)=26 g(2) = 15 f(2)=7 f(2)=14

Big-O Fastest growing function? Slowest?

Big-O For small numbers f is higher, but it is k that grows the fastest for large numbers. Big-O expresses how fast a function grows relative to another Function f(n) is O(g(n)) ◦ g(n) is at least as great as f(n); ultimately for all sufficiently large n Similarly; g(n) is O(h(n)) and h(n) is O(k(n)) Transitivity holds: ◦ If f(n) is O(g(n)) and g(n) is O(h(n)) then f(n) is O(h(n))

Big-O Definition: Let f(n) and g(n) be partial number-theoretic functions. Then function f(n) is O(g(n)) if there exist natural number constants C and n 0 such that for any n n 0 we have f(n) C. g(n) whenever both f(n) and g(n) are defined. e. g. Function f(n) is O(g(n)): with C=1 and n 0=5 4 5=n 0 7 16 f(n) = 7[log 2 n] 14 14 14 28 C. g(n) = 1. 3 n 12 15 21 48

Big-O • • f 1 is O((f 2)) f 3 is O((f 4)) f 5 is O((f 6)) …. . f 9 is O((f 10)) • Growth rate of a function ftime is O(f 72(n)) for instance In other words function ftime is O(n 2) •

Big-O

Big-O Function f(0) is O(0) Function f(n) that is O(1) iff f(n) is bounded above by a constant for sufficiently large arguments n Function f(n) that is O(n) has linear growth Function f(n) is O(n 2) or O(n 3): polynomial growth Function f(n) is O(2 n) or O(3 n): exponential growth

Exercises (a) (b) Characterize these functions as ◦ monotone increasing ◦ strictly monotone increasing ◦ neither (c) (e) (d) (f)

Solution (a) (c) (e) (b) (d) (f) • • • Strictly monotone Neither Strictly monotone Monotone Neither Monotone

Exercises Which of the following functions are O(1) ◦ ◦ ◦ f(n) = [log 2 n] g(n) = [log 10 n] h(n) = min(n, 100) j(n) = 4 n 2 + 7 n + 3 k(n) = 5 n 5 + n 3 + n + 68

Solution Which of the following functions are O(1) ◦ f(n) = [log 2 n] ◦ g(n) = [log 10 n] ◦ h(n) = min(n, 100) only this one is O(1) ◦ j(n) = 4 n 2 + 7 n + 3 ◦ k(n) = 5 n 5 + n 3 + n + 68

Propositional Logic

Proposition: any statement with a truth value Any statement that is either true or false Examples: ◦ For all n in P, n 3 – n is divisible by 6 ◦ N 2 is O(n 3) ◦ 6 is a prime number ◦ Nairobi is located in Kenya ◦ Goldbach’s Conjecture: every even integer strictly greater than 4 is the sum of two primes ◦ Don’t forget to vote ◦ Is Russian beer the best beer in the world? ◦ We define the degree of a vertex v within an undirected graph G to be the number of edges in G incident upon v.

Notation PL: infinite symbol set of ◦ lower-case letters of the Roman alphabet, p, q, r, p’, q’, r’, p’’, q’’, r’’, … which stand for propositions: sentence letters ◦ Propositional connectives: & ◦ Aggregation signs - (), [], {} Language of propositional logic- set of strings over alphabet PL

Negation Symbol: p p T F F T p: John is tall p: John is not tall Unary connective

Conjunction Binary connective: symbol - & p q p&q T T T F T F F For conjuncts p and q p: John is tall q: Andrew is short p&q: John is tall and Andrew is short

Disjunction Binary connective: symbol - p T F q T T F F p q T T T F Inclusive or For disjuncts p and q p: John is tall q: Andrew is short p q: Either John is tall or Andrew is short

Conditional Used to express conditional statements Binary connective: symbol p q T T T F F F F T p: It is raining q: The streets are wet Antecedent p and consequent q False wherever antecedent is true and consequent is false

Conditional Binary connective: Not quite the same semantics as in natural language If Kenya is in Europe, then 2+2=4 = false

Biconditional Binary connective: symbol p q T T T F T F F T If and only if p: 2+2 = 4 q: 1+1 = 2 True if component sentence letters have same truth values

Semantic Issues Sentences of propositional logic classified as: Tautologous ◦ True under all assignments of component sentence letters ◦ p p Contradictory ◦ False under all assignments of component sentence letters ◦ p& p Contingent ◦ Sentences that are neither tautologous or contradictory ◦ p & q and p q

Semantic Issues Satisfiable ◦ A sentence is satisfiable if for at least one assignment of the component letters the sentence holds true ◦ Not contradictory ◦ Truth tables: decision procedure for satisfiability ◦ The satisfiability problem for propositional logic is decidable Equivalence Two sentences S 1 and s 2 are (logically) equivalent if for any given assignment of truth values to component sentence letters, the resulting truth value of S 1 is identical with value of S 2 Idempotence, Associativity, Commutativity, Distributivity and De. Morgans Laws also hold

PROOF TECHNIQUES

Proof Techniques Conditional Proof p q assume truth of the antecedent reason to the truth of consequent q Example 1 If I drink too much alcohol, I can’t drive a car • When I drink alcohol, the alcohol gets absorbed in my blood. My blood reaches my brain, where it will distort its functionality, reducing my coordination abilities. To drive a car I need coordination.

Proof Techniques Conditional Proof Example 2 If Jones is elected, then the treasury will be bankrupted • Given that Jones is lavish and a spendthrift • He will spend vast amounts on annual banquets • Given his slackness in supervising his subordinates, there will be widespread corruption Therefore, if elected then the treasury will be bankrupted

Proof Techniques Indirect Proof p q Proof by contradiction • 11 is not a prime number if it is not a prime number, it must have a non-trivial factor. But it is not 2, 3, 4, 5, 6, 7, 8, 9, 10 and it can’t be larger than n. This is a contradiction, so 11 is a prime after all • Appropriate when p is unstructured. • Derive a contradiction of the form p & p. • Having assumed p is true, continue to show that p & p is true. • Conclude then that p is not true, thereby showing p is true.

Number-theoretic Predicates

Number-theoretic Predicates • Properties of objects, relations, holding between pairs of objects, relations, holding among triples of objects are expressed using predicates. • If the objects are members of N, then one speaks of number-theoretic predicates • e. g. prime(n) is a unary number-theoretic predicate that is satisfied by 2 and 3, but not by 4 or 6 • e. g. predicate n|m hold just in case m is a multiple of n so that 2|6 and 2|0 hold

Number-theoretic Predicates • The extension of a k-ary number-theoretic predicate is the set of all and only those k-tuples of natural numbers satisfying that predicate extension of the predicate prime(n) is the set {n N|prime(n)} extension of cube_of(n, m) is the infinite set of ordered pairs {<0, 0>, <1, 1>, <8, 2>, <27, 3>, <64, 4>, …}

Number-theoretic Predicates • Predicates can be combined using the five propositional connectives: & C 1(n 1, n 2, …nk) & C 2(n 1, n 2, …nk) is true if k-tuple <n 1, n 2, …, nk> satisfies both C 1(n 1, n 2, …nk) and C 2(n 1, n 2, …nk) • • A binary predicate C(n, n) is said to be reflexive if, for any and all n N, we have C(n, n) A binary predicate C(n 1, n 2) is said to be symmetric if, for any n, m N; C(n, m) implies C(m, n)

Notation Quantifier: may play a role in specification of predicates ◦ Universal quantifier ( n)(C 1(n) C 1(n): prime(n) Given any natural number n, either n is prime or it is not Given any natural number n is prime and n> 2 is odd ? ◦ Existential quantifier ( n)(C 1(n) & C 2(n)) C 1(n): prime(n) C 2(n): odd(n) There exists an n such that n is prime and n is not odd

Exercise Given: the following predicates C 1(n) C 2(n, m, k) C 3(n, m, k) with n, m, k N C 1(n 1) n 1 is prime C 2(n 1, n 2, n 3) n 1 = n 2 + n 3 C 3(n 1, n 2, n 3) n 1 = n 2. n 3 For each of the predicates below, determine whether it is satisfied by the pair <4, 12> n=4 m=12 � ( k)C 2(n, m, k) � ( k)[C 2(k, n, m) & C 1(k)] � ( k) C 3(n, m, k) � ( k)C 3(k, n, m) � ( k)[C 3(k, n, m)&C 1(k)] � ( k n)[C 3(m, k, n) & C 1(k)]

Exercise Given: the following predicates C 1(n) C 2(n, m, k) C 3(n, m, k) with n, m, k N C 1(n 1) n 1 is prime C 2(n 1, n 2, n 3) n 1 = n 2 + n 3 C 3(n 1, n 2, n 3) n 1 = n 2. n 3 For each of the predicates below, determine whether it is satisfied by the pair <4, 12> n=4 m=12 � ( k)C 2(n, m, k) no � ( k)[C 2(k, n, m) & C 1(k)] no � ( k) C 3(n, m, k) yes � ( k)C 3(k, n, m) yes � ( k)[C 3(k, n, m)&C 1(k)] no � ( k n)[C 3(m, k, n) & C 1(k)] yes