Analysis of Algorithms Chapter 02 Recurrences 1 Methods

Analysis of Algorithms Chapter - 02 Recurrences 1

Methods Ø Definition: A recurrence is an equation or inequality that describes a function in terms of its value on smaller inputs. Ø The worst-case running time T(n) of an algorithm could be described as by the recurrence T(n) = a. T(n/b) + f(n) where a≥ 1, b>1, and f(n) is a given function. Ø Three methods will be discussed to solve the recurrences obtaining asymptotic bounds on the solution. Ø Substitution method: Ø We guess a bound and then use mathematical induction to prove that our guess is correct. Ø Recursion-tree method: Ø It converts the recurrence into a tree whose nodes represent the cost incurred at various levels of the recursion Ø We use techniques for bounding summations to solve the recurrence. Ø Master method: Ø It provides bounds for recurrences of the above form. Ø It requires memorization of three cases. Ø Once you memorize the cases, determining asymptotic bounds for many recurrences is easy. 2

Induction Ø Definition of Proof: The process or an instance of establishing the validity of a statement especially by derivation from other statements in accordance with principles of reasoning. Ø Proof by Induction: Let Pn be a statement for all the positive integers (n = 1, 2, 3, . . . ). If the following two properties hold: 1. P 1 is true. 2. Pk+1 is true if Pk is true for each positive integer k. Then Pn is true for all n. Ø Why Induction Works? Ø Let S be the set of all numbers n for which Pn is false. Ø Let k be the minimum number in S. Ø k > 1 since by the first property of the induction definition, P 1 is true. Ø By the minimality of k, Pk− 1 is true and Pk is false. Ø A contradiction to the second property of the induction definition. 3

A Summation Problem Ø Prove that for any integer n ≥ 1: 1 + 2 + 3 + · · · + n = n(n+1)/2. ØDefine: − L(n) = 1 + 2 + 3 + · · · + n. − R(n) = n(n+1)/2. Ø Prove that L(n) = R(n) for n ≥ 1. Ø Verifying the Claim Ø First check for small values of n: n 1 L(n) 1=1 R(n) 1. 2/2=1 2 3 1+2=3 1+2+3=6 2. 3/2=3 3. 4/2=6 : 10 : 1+2+3+…+10=55 : 10. 11/2=55 4

A Direct Proof Ø Idea: Compute the value of 2 L(n). Example: 2(1 + 2 + 3) = (1 + 2 + 3) + (3 + 2 + 1) = (1 + 3) + (2 + 2) + (3 + 1) =3· 4 = 12 = 2(3· 4/2) Hence 2·L(n) = (1 + · · · + n) + (n + · · · + 1) = (1 + n) + (2 + (n − 1)) + · · · + (n + 1) = (n + 1) + · · · + (n + 1) = n(n + 1) This implies that: L(n) = n(n+1)/2 = R(n). 5

A Proof by Induction Ø For n=1, L(1) = 1 and R(1) = (1. 2)/2 = 1 So, for n=1, it is true. ØLet us assume that it is true for n=k, that is, L(k) = R(k) =>1+2+3+…. . +k = k(k+1)/2 ØNow, if we can prove that it is true for n=k+1 also, then it can be said that it is true for all n. ØFor n=k+1, L(k + 1) = 1 + 2 + · · · + k + (k + 1) = L(k) + (k + 1) = R(k) + (k + 1) = k(k + 1)/2 + (k + 1) = (k + 1)(k/2 + 1) = (k + 1)(k + 2)/2 = R(k + 1) 6 Ø Hence L(n) = R(n), for all integer n.

Substitution Method 7

Recurrence -1 Ø The method for solving recurrences entails two steps: − Guess the form of the solution − Use mathematical induction to find the constants and show that the solution works. Ø This method is powerful, but it obviously can be applied only in cases when it is easy to guess the form of the solution. Ø This method can be used to establish either upper or lower bounds on a recurrence. Ø Problem: Let us determine an upper bound on the following recurrence: T(1) = 0. T(n) = 2 T(n/2) + n, for n>1. Ø Solution: Compute the solution for small powers of 2: T(2) = 2 T(1) + 2 = 2. T(4) = 2 T(2) + 4 = 8. T(8) = 2 T(4) + 8 = 24. T(16) = 2 T(8) + 16 = 64. T(32) = 2 T(16) + 32 = 160. 8

Recurrence -1 (Contd. ) ØGuessing the Solution - Guess T(n) = n log 2 n for n a power of 2. Ø A Proof by Induction For n=1, T(1)=0, that is T(n)=nlog 2 n So, for n=1, it is true. Let us assume that it is true for n=k/2, that is, T(k/2)=(k/2) log 2 (k/2) Now, if we can prove that it is true for n=k also, then it can be said that it is true for all n. For n=k, T(k) = 2 T(k/2) + k = 2(k/2) log 2(k/2) + k = k(log 2 k − 1) + k = k log 2 k Ø Hence T(n) = n log 2 n, for all integer n. 9

Recurrence -2 Ø Problem: Let us determine an upper bound on the following recurrence: T(1) = a. T(n) = 2 T(n/2) + bn. For some constants a, b (independent of n). Ø Solution: Compute the solution for small powers of 2: T(2) = 2 T(1) + 2 b = 2 b + 2 a. T(4) = 2 T(2) + 4 b = 8 b + 4 a. T(8) = 2 T(4) + 8 b = 24 b + 8 a. T(16) = 2 T(8) + 16 b = 64 b + 16 a. T(32) = 2 T(16) + 32 b = 160 b + 32 a. ØGuessing the Solution - Guess T(n) = bn log 2 n + an, for n a power of 2 10

Ø Verify the guess for small numbers: b · 1 log 2 1 + a · 1 = a. b · 2 log 2 2 + a · 2 = 2 b + 2 a. b · 4 log 2 4 + a · 4 = 8 b + 4 a. b · 8 log 2 8 + a · 8 = 24 b + 8 a. b · 16 log 2 16 + a · 16 = 64 b + 16 a. b · 32 log 2 32 + a · 32 = 160 b + 32 a. Ø A Proof by Induction For n=1, T(1)=a, that is T(n)=b. nlog 2 n + a. n So, for n=1, it is true. Let us assume that it is true for n=k/2, that is, T(k/2)=b. (k/2) log 2 (k/2) + a. (k/2) Now, if we can prove that it is true for n=k also, then it can be said that it is true for all n. For n=k, T(k) = 2 T(k/2) + bk = 2(b(k/2) log 2(k/2) + a(k/2)) + bk = bk(log 2 k − 1) + ak + bk = bk log 2 k + ak 11 Ø Hence T(n) = b. n log 2 n + a. n, for all integer n

Recurrence -3 Ø Problem: Let us determine an upper bound on the following recurrence: T(1) = a. T(n) = T(n/2) + b. For some constants a, b (independent of n). Ø Solution: Compute the solution for small powers of 2: T(2) = T(1) + b = b + a. T(4) = T(2) + b = 2 b + a. T(8) = T(4) + b = 3 b + a. T(16) = T(8) + b = 4 b + a. T(32) = T(16) + b = 5 b + a. Ø Guessing the Solution - Guess T(n) = b log 2 n + a, for n a power of 2 12

Ø Verify the guess for small numbers: b · log 2 1 + a = a. b · log 2 2 + a = b + a. b · log 2 4 + a = 2 b + a. b · log 2 8 + a = 3 b + a. b · log 2 16 + a = 4 b + a. b · log 2 32 + a = 5 b + a. Ø A Proof by Induction For n=1, T(1)=a, that is T(n)=b log 2 n + a So, for n=1, it is true. Let us assume that it is true for n=k/2, that is, T(k/2)=b log 2 (k/2) + a Now, if we can prove that it is true for n=k also, then it can be said that it is true for all n. For n=k, T(k) = T(k/2) + b = (b log 2(k/2) + a) + b = b(log 2 k − 1) + a + b = b log 2 k + a 13 Ø Hence T(n) = b log 2 n + a, for all integer n

Recurrence -4 T(n) = 2 T(n/2) + n = O(n lg n) m Thus, we need to show that T(n) c n lg n with an appropriate choice of c q Inductive hypothesis: assume T(n/2) c (n/2) lg (n/2) q Substitute back into recurrence to show that T(n) c n lg n follows, when c 1 Ø T(n) = 2 T(n/2) + n 2 (c (n/2) lg (n/2)) + n = cn lg(n/2) + n = cn lg n – cn lg 2 + n = cn lg n – cn + n cn lg n for c 1 = O(n lg n) for c 1

Iteration Method m Iteration method: q Expand the recurrence k times q Work some algebra to express as a summation q Evaluate the summation

Iteration Method – Example T(n) = n + 2 T(n/2) = n + 2(n/2 + 2 T(n/4)) = n + 4 T(n/4) = n + 4(n/4 + 2 T(n/8)) = n + n + 8 T(n/8) … = in + 2 i. T(n/2 i) = kn + 2 k. T(1) = nlgn + n. T(1) = Θ(nlgn) Assume: n = 2 k Taking lg on both sides lg n = lg (2 k ) lg n = klg 2 lg n = k x 1 lg n = k

m m T(n) = c + T(n-1) = 2 c + T(n-2) = 3 c + T(n-3) = ck + T(n-k) So far for n k we have q T(n) = ck + T(n-k) = = … c + T(n-2) 2 c + T(n-3) kc + T(n-k) m To stop the recursion, we should have q n-k=0 k=n q T(n) = cn + T(0) = cn m Thus in general T (n ) = O (n )

m T(n) = n + T(n-1) = n + n-1 + T(n-2) = n + n-1 + n-2 + T(n-3) = n + n-1 + n-2 + n-3 + T(n-4) =… = n + n-1 + n-2 + n-3 + … + (n-k+1) + T(n-k) = m for n k To stop the recursion, we should have n - k = 0 k = n

The Master Method w Based on the Master theorem. w T(n) = a. T(n/b) + f(n) • a 1, b > 1 are constants. • f(n) is asymptotically positive. w Requires memorization of three cases.

The Master Theorem: Let a 1 and b > 1 be constants, let f(n) be a function, and Let T(n) be defined on nonnegative integers by the recurrence T(n) = a. T(n/b) + f(n), where we can replace n/b by n/b or n/b. T(n) can be bounded asymptotically in three cases: 1. If f(n) = O(nlogba– ) for some constant > 0, then T(n) = (nlogba). 2. If f(n) = (nlogba), then T(n) = (nlogbalg n). 3. If f(n) = (nlogba+ ) for some constant > 0, and if, for some constant c < 1 and all sufficiently large n, we have a·f(n/b) c f(n), then T(n) = (f(n)).

The Master Theorem m Given: a divide and conquer algorithm q An algorithm that divides the problem of size n into a subproblems, each of size n/b q Let the cost of each stage (i. e. , the work to divide the problem + combine solved subproblems) be described by the function f(n)

The Master Theorem m m if T(n) = a. T(n/b) + f(n) then where a ≥ 1 & b > 1

Understanding Master Theorem m In each of the three cases, we are comparing f(n) with nlogba , the solution to the recurrence is determined by the larger of the two functions. m In case 1, if the function nlogba is the larger, then the solution T(n) = Θ(nlogba). m In case 3, if the function f(n) is the larger, then the solution is T(n) = (f(n)). m In case 2, if the two functions are the same size, then the solution is T(n) = Θ(nlogba lg n) = Θ(f(n) lg n).

Understanding Master Theorem m In case 1, not only must f(n) be smaller than nlogba, it must be polynomially smaller. That is f(n) must be asymptotically smaller than nlogba by a factor of nε for some constant ε > 0. m In case 3, not only must f(n) be larger than nlogba , it must be polynomially larger and in addition satisfy the “regularity” condition that: a f(n/b) ≤ c f(n).

Understanding Master Theorem m It is important to realize that the three cases do not cover all the possibilities for f(n). m There is a gap between cases 1 and 2 when f(n) is smaller than nlogba but not polynomially smaller. m There is a gap between cases 2 and 3 when f(n) is larger than nlogba but not polynomially larger. m If f(n) falls into one of these gaps, or if the regularity condition in case 3 fails to hold, the master method cannot be used to solve the recurrence.

Using The Master Method Case 1 m T(n) = 9 T(n/3) + n q a=9, b=3, f(n) = n q nlog a = nlog 9 = (n 2) >> from log 332 q Since f(n) = O(nlog 9 - ) = O(n 2 -0. 5) = O(n 1. 5) q where =0. 5 q case 1 applies: b 3 3 q Thus the solution is Ø T(n) = (n 2)

Using The Master Method Case 2 m T(n) = T(2 n/3) + 1 q a=1, b=3/2, f(n) = 1 q nlog a = nlog 1 = n 0 = 1 q Since f(n) = (nlog a) = (1) q case 2 applies: b 3/2 b q Thus the solution is Ø T(n) = (lg n)

Using The Master Method Case 3 m T(n) = 3 T(n/4) + n lg n q a=3, b=4, f(n) = n lg n q nlog a = nlog 3 = n 0. 793 = n 0. 8 q Since f(n) = (nlog 3+ ) = (n 0. 8+0. 2)= (n) q where 0. 2, and for sufficiently large n, b 4 4 Ø a. f(n/b) = 3(n/4) lg(n/4) < (3/4) n lg n for c = 3/4 q q case 3 applies: Thus the solution is Ø T(n) = (n lg n)

Recursion-tree Method 29

The method Ø Although the substitution method can provide a concise proof that a solution to a recurrence is correct, it is sometimes difficult to come up with a good guess. Ø Drawing out a recursion-tree is a straight forward way to devise a good guess. Ø In a recursion-tree, − each node represents the cost of a single sub-problem somewhere in the set of recursive function invocations. − We sum the costs within each level of the tree to obtain a set of per-level costs. − Then we sum all the per-level cost to determine the total cost of all levels of the recursion. Ø Recursion trees are particularly useful when the recurrence describes the running time of a divide-and-conquer algorithm. Ø A recursion tree is best used to generate a good guess, which is then verified by the substitution method. Ø In this section, we will use recursion-trees to generate good guess. 30

Example-1 Ø Let us see a recursion-tree would provide a good guess for the recurrence: Ø We start by focusing on finding an upper bound for the solution. Following figure shows the derivation of the recursion-tree, assuming n as exact power of 4. Ø We create a recursion tree for T(n) = 3 T(n/4) + cn 2, c>0. 31

Explanation Ø Part (a) of the figure shows T(n), which is expanded in Ø Part (b) into an equivalent tree representing the recurrences. The cn 2 term at the root represents the cost at the top level of recursion, and the three subtrees of the root represent the cost incurred by the subproblems of size n/4. Ø Part (c) shows this process carried one step further by expanding each node with cost T(n/4) from part (b). The cost for each of the three children of the root is c(n/4)2. Ø We continue expanding each node in the tree by breaking it into its constituent parts as determined by the recurrence. Ø Since the subproblem sizes decrease as we get further from the root, we eventually must reach a boundary condition. Ø How far from the root do we reach one? Ø The subproblem size for a node at depth i is n/4 i. Thus the subproblem size hits n=1, when n/4 i=1, that is, i=log 4 n. Ø Thus the tree has levels (0, 1, 2, ……, log 4 n). 32

Explanation (Contd. ) Ø Now, we determine the cost at each level of the tree. Ø Each level has three times more nodes than the level above, So the number of nodes at depth i is 3 i. Ø Each node at depth i has a cost of c(n/4 i), for i=0, 1, 2, …. . , log 4 n -1. Ø Multiplying, we get that total cost over all nodes at depth i is 3 ic(n/4 i)2 = (3/16)i cn 2. Ø The last level at, depth log 4 n, has 3 log 4 n = nlog 43 nodes, each contributing cost T(1), for a total cost of nlog 43. T(1) , which is Θ(nlog 43). Ø Now add up the costs over all levels to determine the cost for the entire tree. 33

Explanation (Contd. ) Ø Now, we can use the substitution method to verify that our guess was correct, that is T(n)=O(n 2) is an upper bound for the recurrence Ø We want to show that T(n) ≤ dn 2, for some d>0. Ø Using the same constant c>0 as before, we have 34

Example-2 ØLet us see a recursion-tree would provide a good guess for the recurrence: T(n) = T(n/3) + T(2 n/3) + O(n). ØLet c>0 be the constant factor for the term O(n). 35

Explanation (Contd. ) Ø Now, when we add the values across the levels of the recursion-tree, we get a value of cn for every level. Ø The longest path from root to a leaf is n→(2/3)n →(2/3)2 n →…. . → 1. Ø Since (2/3)kn =1, when k = log 3/2 n, Ø The height of the tree is log 3/2 n. Ø Intuitively, we expect the solution to the recurrence be O(cn. log 3/2 n) = O(n. lg n) Ø We can use the substitution method to verify that our guess was correct, that is T(n)=O(n. lg n) is an upper bound for the given recurrence. Ø We want to show that T(n) ≤ dn. lg n, for some d>0. Ø Using the same constant c>0 as before, we have 36