Divide Conquer Themes Reasoning about code correctness and

Divide & Conquer Themes Reasoning about code (correctness and cost) recursion, induction, and recurrence relations Divide and Conquer Examples sorting (insertion sort & merge sort) computing powers

Sorting Given a list L = (L 1 … Ln) of elements from a totally ordered set (e. g. integers with ≤) return a list M = (M 1 … Mn) such that (sorted M), i. e. M 1 ≤ M 2 ≤ ≤ Mn M = (permutation L), i. e. Mi = L (i) where is a bijection from {1, …, n} to {1, …, n} Use divide and conquer to derive several recursive algorithms and the principle of balance to obtain an efficient algorithm

sortedp Predicate to test to see if a list is sorted Should check to see that L is a list of integers (x 1 x 2 … xn) with x 1 ≤ x 2 ≤ ≤ xn xi ≤ xj for 1 ≤ i < j ≤ n (defun sortedp (L) (cond ((equal L nil) t) ((equal (length L) 1) t) ((and (<= (first L) (second L)) (sortedp (rest L))))))

Correctness of sortedp L = (x 1 x 2 … xn) By induction on len(L) Base cases L nil or of length 1 Show xi ≤ xj for 1 ≤ i < j ≤ n x 1 ≤ x 2 and by induction xi ≤ xj for 2 ≤ i < j ≤ n x 1 ≤ x 2 ≤ xj x 1 ≤ xj for 2 ≤ j ≤ n

Precise Proof is Difficult We take advantage of informal reasoning and names which provide intuitive meaning L = (x 1 x 2 … xn) Prove (fn 123 L) = t if xi ≤ xj for 1 ≤ i < j ≤ n and nil otherwise using equational properties of fn 13, fn 14, fn 15, fn 17, fn 81, fn 100 and fn 150 (defun fn 123 (L) (fn 13 ((fn 81 L nil) t) ((fn 81 (fn 150 L) 1) t) ((fn 82 (fn 100 (fn 14 L) (fn 17 L)) (fn 123 (fn 15 L))))))

Insertion Sort To sort x 1, …, xn, recursively sort x 2, …, xn insert x 1 into x 2, …, xn (see code for details) Prove correctness by induction

Insertion Sort (defun insertionsort (L) (if (equal L nil) nil (insert (first L) (insertionsort (rest L))))) (defun insert (x L) (cond ((equal L nil) (list x)) ((<= x (first L)) (cons x L)) ((cons (first L) (insert x (rest L))))))

Insertion Sort (Example) (insertionsort ‘(7, 6, 5, 4, 3, 2, 1, 0)) recursive call returns (0, 1, 2, 3, 4, 5, 6) (insert 7 ‘(0, 1, 2, 3, 4, 5, 6)) (0, 1, 2, 3, 4, 5, 6, 7) Number of comparisons to sort inputs that are in reverse sorted order (worst case) C(n) = C(n-1) + (n-1) C(1) = 0

Correctness of insertionsort L = (L 1 … Ln) 1. (insertionsort L) terminates and returns a list of length n 2. (insertionsort L) is sorted 3. (insertionsort L) is a permutation of L Proof by induction on n 1. For n 0, recursive call with smaller n 2. Recursive call produces a sorted list of (rest L) by induction and (insert (first L) (insertionsort (rest L))) returns a sorted list (by induction) 3. Recursive call returns a permutation of (rest L) and insert returns a list containing (first L) and elements of (rest L)

Solve Recurrence for C(n)

Merge Sort To sort x 1, …, xn, recursively sort x 1, …, xa and xa+1, …, xn, where a = n/2 merge two sorted lists Insertion sort is a special case where a=1

Mergesort (defun mergesort (L) (cond ((equal L nil) ((equal (length L) 1) L) ((mergesort (take (floor (length L) 2) L)) (mergesort (drop (floor (length L) 2) L))))))

Merge (defun merge (L M) (cond ((equal L nil) M) ((equal M nil) L) ((<= (first L) (first M)) (cons (first L) (merge (rest L) M))) ((cons (first M) (merge L (rest M))))))

Take and Drop (defun take (n L) (cond ((equal L nil) ((equal n 0) nil) ((cons (first L) (take (- n 1) (rest L)))))) (defun drop (n L) (cond ((equal L nil) ((equal n 0) L) ((drop (- n 1) (rest L)))))

Merge Sort (Example) (7, 6, 5, 4, 3, 2, 1, 0) after recursive calls (4, 5, 6, 7) and (0, 1, 2, 3) Number of comparisons needed to sort, worst case (the merged lists are perfectly interleaved) M(n) = 2 M(n/2) + n-1 M(1) = 0 What is the best case (all in one list > other list)? M(n) = 2 M(n/2) + n/2

Comparison of Insertion and Merge Sort Count the number of comparisons for different n=2 k M(n)/C(n) 0 as n increases C(n) grows faster than M(n) C(2 n)/C(n) 4 So, apparently quadratic. C(n) = Θ(n 2) M(2 n)/M(n) 2 as n increases n M(n) C(n) M(n)/C(n) 2 1 1 1 4 4 6 0. 667 8 12 28 0. 429 16 32 120 0. 267 32 80 496 0. 161 64 192 2016 0. 095 128 4481 8128 0. 055

Solve Recurrence for M(n)

Mergesort What is wrong with the following? (defun mergesort (L) (if (equal L nil) nil (mergesort (mytake (floor (length L) 2) L)) (mergesort (drop (floor (length L) 2) L)))))

Correctness of mergesort L = (L 1 … Ln) 1. (mergesort L) terminates and returns a list of length n 2. (mergesort L) is sorted 3. (mergesort L) is a permutation of L 4. Proof by induction on n 1. For n 2, recursive calls with smaller n 2. Recursive calls produce sorted lists by induction and merge produces a sorted list 3. Recursive calls contain lists whose concatenation is L and hence by induction return lists whose concatenation is L. merge returns a permutation of L

Properties of nth L = (L 1 … Llen(L)) 1. 0 < n ≤ len(L), then (nth n L) = Ln Proof by induction on n 2. n > len(L), then (nth n L) = nil Proof by induction on len(L) (defun nth (n L) (if (equal L nil) nil (if (equal n 1) (first L) (nth (- n 1) (rest L)))))

Properties of nth L = (L 1 … Llen(L)) 1. 0 < n ≤ len(L), then (nth n L) = Ln Proof by induction on n 1. Base case. n = 1 return (first L) 2. By induction (nth (- n 1) (rest L)) returns the (n-1)st element of (rest L) which is the nth element of L (n > 1)

Properties of nth L = (L 1 … Llen(L)) 1. n > len(L), then (nth n L) = nil Proof by induction on len(L) 1. Base case. Len(L) = 0. L = nil and return nil 2. n > len(L) n-1 > len((rest L)), therefore by induction (nth (- n 1) (rest L)) returns nil

Properties of take and drop L = (L 1 … Llen(L)) 1. 0 ≤ n ≤ len(L), then len((take n L)) = n Proof by induction on n 2. n > len(L), then (take n L) = L Proof by induction on len(L) 3. 0 < k ≤ n ≤ len(L), then (nth k (take n L)) = Lk Proof by property of nth and induction on k 4. 0 ≤ n ≤ len(L), then len((drop n L)) = len(L) – n 5. n > len(L), then (drop n L) = nil 6. 0 ≤ n ≤ len(L), 0 < k ≤ len(L)-n, then (nth k (drop n L)) = Ln+k 7. 0 ≤ n ≤ len(L), (append (take n L) (drop n L)) = L

Properties of merge L = (L 1 … Lm) and M = (M 1 … Mn) • Assume L is sorted and M is sorted • (merge L M) is a list of length m+n • (merge L M) is sorted • (merge L M) is a permutation of (L 1 … Lm M 1 … Mn)

Exercise Use check= to verify properties of take and drop for several examples Use induction to prove the properties

Solution Property 1 L = (L 1 … Llen(L)) 1. 0 ≤ n ≤ len(L), then len((take n L)) = n Proof by induction on n 1. Base case. n = 0 return nil and len(nil) = 0. 2. Assume n > 0 and show that len((take n L)) = n. Assume that take returns a list of length k for k < n [Inductive hypothesis] 3. In this case take returns (cons (first L) (take (- n 1) (rest L))) and since n-1 ≤ len((rest L)) (take (- n 1) (rest L)) by induction is a list of length n-1. Therefore (cons (first L) (take (- n 1) (rest L))) has length (n-1) + 1 = n.

Solution Property 2 L = (L 1 … Llen(L)) 1. n > len(L), then (take n L) = L Proof by induction on len(L) 1. Base case. len(L) = 0 and take returns L = nil independent of n. 2. Assume n > len(L) = and show that (take n L) = L. Assume that take returns M when len(M) < [Inductive hypothesis] 3. In this case take returns (cons (first L) (take (- n 1) (rest L)). By induction, (take (- n 1) (rest L)) returns (rest L) and therefore (take n L) returns (cons (first L) (rest L)) = L.

Solution Property 3 L = (L 1 … Llen(L)) 1. 0 < k ≤ n ≤ len(L), then (nth k (take n L)) = Lk Proof by property of nth and induction on k 1. Base case. k = 1 and by the first property of nth, (nth 1 (take n L)) = the first element of (take n L) which by the definition of take is the first element of L. 2. Assume the jth element of (take n L) = Lj for j < k [Inductive hypothesis] and show that the kth element of (take n L) = Lk. 3. By the first property of nth (nth k (take n L)) is the kth element of (take n L) and by the definition of take, that is the (k-1)st element of (take (- n 1) (rest L)) which by induction is the (k-1)st element of (rest L) which is the kth element of L.

Solution of Properties 4 -7 L = (L 1 … Llen(L)) Properties 4 -6 for drop are analogous to properties 1 -3 for take and their proofs are similar to those for properties 1 -3. Property 7 follows from properties 3 and 6.

Computing Powers Recursive definition: an = a an-1, n > 0 a 0 = 1 Number of multiplications M(n) = M(n-1) + 1, M(0) = 0 M(n) = n

Exercise Write a recursive ACL program to compute the power of a number using the recursive definition on the previous slide Prove by induction on n that (power a n) = an

Solution ; inputs a rational number a and a natural number n ; output a rational number equal to a^n (defun power (a n) (if (= n 0) 1 (* a (power a (- n 1))))) (check= (power 2 0) 1) (check= (power 2 10) 1024)

Solution Continued (defun power (a n) (if (= n 0) 1 (* a (power a (- n 1))))) Base case n = 0 (power a 0) = 1 = a 0 Assume n > 0 and (power a (- n 1)) = an-1 [Inductive Hypothesis] (power a n) = (* a (power a (- n 1)) = a×an-1 [by induction] = an

Binary Powering (Recursive) Binary powering x 16 = (((x 2)2)2)2, 16 = (10000)2 x 23 = (((x 2)2 x)2 x)2 x, 23 = (10111)2 Recursive (right-left) xn = (x n/2 )2 x(n % 2) M(n) = M( n/2 ) + [n % 2] + 1

Exercise Write a recursive ACL program to compute the power of a number using the binary powering algorithm Prove by induction on n that (fastpower a n) = an

Solution ; inputs a rational number a and a natural number n ; output a rational number equal to a^n (defun fastpower (a n) (if (= n 0) 1 (let ((b (fastpower a (floor n 2)))) (if (evenp n) (* b b a))))) (check= (power 2 0) 1) (check= (fastpower 2 16) 65536) (check= (fastpower 2 15) 32768)

Solution Continued Base case n = 0 (fastpower a 0) = 1 = a 0 Assume n > 0 and (fastpower a k) = ak for k < n [Inductive Hypothesis] Even case n = 2 k (fastpower a n) = (* (fastpower a k)) = ak×ak [by induction] = a 2 k = an Odd case n = 2 k+1 (fastpower a n) = (* (fastpower a k) a) = ak×ak×a [by induction] = a 2 k+1 = an

Binary Powering Number of multiplications Let (n) = number of 1 bits in binary representation of n num bits = lg(n) +1 M(n) = lg(n) + (n) ≤ lg(n) => M(n) ≤ 2 lg(n)

Asymptotic Growth Rates f(n) = O(g(n)) [grows at the same rate or slower] There exists positive constants c and n 0 such that f(n) c g(n) for all n n 0 Ignore constants and low order terms

Asymptotic Growth Rates f(n) = o(g(n)) [grows slower] f(n) = O(g(n)) and g(n) O(f(n)) limn f(n)/g(n) = 0 f(n) = (g(n)) [grows at the same rate] f(n) = O(g(n)) and g(n) = O(f(n))

Asymptotic Growth Rates (log(n)) – logarithmic [log(2 n)/log(n) = 1 + log(2)/log(n)] (n) – linear [double input double output] (n 2) – quadratic [double input quadruple output] (n 3) – cubit [double input output increases by factor of 8] (nk) – polynomial of degree k (cn) – exponential [double input square output]

Merge Sort and Insertion Sort TI(n) = TI(n-1) + (n) = (n 2) [worst case] TI(n) = TI(n-1) + (1) = (n) [best case] Merge Sort TM(n) = 2 TM(n/2) + (n) = (nlogn) [worst case] TM(n) = 2 TM(n/2) + (n) = (nlogn) [best case]

Divide & Conquer Recurrence Assume T(n) = a. T(n/b) + (n) T(n) = (n) [a < b] T(n) = (nlog(n)) [a = b] T(n) = (nlogb(a)) [a > b]

Karatsuba’s Algorithm Using the classical pen and paper algorithm two n digit integers can be multiplied in O(n 2) operations. Karatsuba came up with a faster algorithm. Let A and B be two integers with A = A 110 k + A 0, A 0 < 10 k B = B 110 k + B 0, B 0 < 10 k C = A*B = (A 110 k + A 0)(B 110 k + B 0) = A 1 B 1102 k + (A 1 B 0 + A 0 B 1)10 k + A 0 B 0 Instead this can be computed with 3 multiplications T 0 = A 0 B 0 T 1 = (A 1 + A 0)(B 1 + B 0) T 2 = A 1 B 1 C = T 2102 k + (T 1 - T 0 - T 2)10 k + T 0

Complexity of Karatsuba’s Algorithm Let T(n) be the time to compute the product of two n-digit numbers using Karatsuba’s algorithm. Assume n = 2 k. T(n) = (nlg(3)), lg(3) 1. 58 T(n) 3 T(n/2) + cn 3(3 T(n/4) + c(n/2)) + cn = 32 T(n/22) + cn(3/2 + 1) 32(3 T(n/23) + c(n/4)) + cn(3/2 + 1) = 33 T(n/23) + cn(32/22 + 3/2 + 1) … 3 i. T(n/2 i) + cn(3 i-1/2 i-1 + … + 3/2 + 1). . . cn[((3/2)k - 1)/(3/2 -1)] --- Assuming T(1) c 2 c(3 k - 2 k) 2 c 3 lg(n) = 2 cnlg(3)

Identities for Sums

Some Common Sums

Arithmetic Series Sum of consecutive integers (written slightly differently):

Arithmetic Series (Proof)

Geometric Series = 1 + x 2 + … + xn-1 =

Geometric Series (Proof)

Floor and Ceiling Let x be a real number The floor of x, x , is the largest integer less than or equal to x If an integer k satisfies k x < k+1, k = x E. G. 3. 7 = 3, 3 = 3 The ceiling of x, x , is the smallest integer greater than or equal to x If an integer k satisfies k-1 < x k, k = x E. G. 3. 7 = 4, 3 = 3

logarithm y = logb(x) by = x Two important cases ln(x) = loge(x) lg(x) = log 2(x) [frequently occurs in CS] Properties log(cd) = log(c) + log(d) log(cx) = xlog(c) logb(bx) = x = blogb(x) d ln(x)/dx = 1/x

logarithm 2 k x < 2 k+1 k = lg(x) E. G. 16 25 < 32 4 lg(25) < 5 lg(25) 4. 64 Change of base logc(x) = logb(x) / logb(c) Proof. y = logc(x) cy = x ylogb(c) = logb(x) y = logb(x) / logb(c)