Chapter 2 Algorithm Analysis Objectives How to estimating

Chapter (2) - Algorithm Analysis Objectives: • How to estimating the time required for a program to solve a problem. • How to reduce the running time. • Observe the result of careless use of recursion. • Introducing an efficient algorithm to raise a number to power. • Computing the greatest common divisor (GCD) of two numbers. Questions: How do we determine how good a program is without writing it? What is the quantitative measure of the performance of a program? An algorithm is a clearly set of instructions to be followed to solve a problem. An algorithm is consider to be poor if: It takes a long time to solve the problem, and/or It takes a large part of main memory for the calculations. 1

Mathematical Background Definition: T(n) = O(f(n)) if there are constants c and n 0 such that: Definition: If there are constants c and n 0 such that : Definition: if and only if Definition: if 2

Definition: T(n) = O(f(n)) if there are constants c and n 0 such that: There is some point n 0 past which c. f(n) is always at least as large as T(n), so that if constant factors are ignored, f(n) is at least as big as T(n). Example: 1000 n > n 2 for small values of n: 1000 > 1, 2000 > 2, 10, 000 > 100, … , 999, 000 > 998001. 3

How about for n = 1000? The answer is no. 1, 000 is not larger than 1, 000. This will tell us that n 2 grow much faster than 1000 n. Going back to the definition, in our case: T(n) = 1, 000 n, f(n)=n 2, n 0=1, 000, and c = 1, or we could use n 0 = 10 and c = 100. Thus, we will say that 1, 000 n = O(n 2) (order of n-square). This means that we are guaranteed that 1000 n does not grow faster than n 2. In other words, n 2 is the upper bound on 1000 n. Since this implies that f(n)= (T(n)), we will say that T(n) is a lower bound on f(n). This notation O( ), is known as Big-Oh, and o( ) is as little-oh. 4

Example: n 3 grows faster than n 2, so we can say: f(n) = n 2 and g(n) = 2 n 2 grow at the same rate. Thus, the above relation is true: When two functions grow at the same rate, then the decision on whether or not to signify this with can depend on the particular context. Example: if g(n) = 2 n 2, then g(n) = O(n 4), g(n)=O(n 3), and g(n)=O(n 2) are all technically correct. But, the last one is the most accurate and is the best answer. Note: writing, says that g(n) = O(n 2) and as good (tight) answer as possible. 5

Important Rules: 6

Proof of Rule 1(a): 7

Proof of Rule 1(b): 8

Typical Growth Rates: 9

Important Notes: • Do not include constants or lower terms inside a Big-Oh, i. e. T(n) = O(2 n 2) or T(n)=O(n 2+n) are not correct. The correct form for both is: O(n 2). • The relative growth rates of two functions f(n) and g(n) can be computed using: known as the L’Hopital’s rule states that: Where, f’(n) and g’(n) are the derivatives of f(n) and g(n), respectively. 10

The limit can have 4 possible values: 1. The limit is 0: This means that f(n) = o(g(n)), i. e. g(n) is the upper -bound to f(n). 2. The limit is a not zero value: This means that are growing with the same rates. Both 3. The limit is : This means that g(n) = o(f(n)), i. e. , g(n) is the lower-bound to f(n). 4. The limit oscillates: There is no relation (this will not happen in out case). 11

Model A model of computation is needed for the analysis of algorithms in a formal framework. Our model is a normal computer in which instructions are executed sequentially. This model will do the simple instructions such as +, -, /, =, …, but unlike real computers, it takes exactly one time unit to do anything (simple). What do we analyze? The most important thing is the running time. Several factors affect the running time: • Effect of compilers and computer used will not be discussed here. • Effect of algorithm used and the input to the algorithm will be discussed here. Typically, the size of the input is the main consideration. The worst-case running time, is the quantity required in the analysis. 12

Maximum Subsequence Sum Problem: Given (possibly negative) integers a 1, a 2, a 3, …, an, find the maximum value of. (For convenience, the maximum subsequence sum is 0 if all the integers are negative). Example: For input: -2, 11, -4, 13, -5, -2, the answer is 20: (11+(-4)+13 = 20, a 2 through a 4). There are many algorithms to solve this problem. Here we will discuss 4 of these methods. Running Time Calculations There are several ways to estimate the running time of a program: • Based on empirical data: write the code, run and measure the time it takes to perform a particular calculation. • Based on the Analysis of the program: this method defines the bottlenecks. It gives an upper bound to the running time, i. e. , the Big. Oh running time. 13

14

15

16

Sample Analysis Count Line 4: 1 Line 5: 1+ n+1{<=}+ n {i++} = 2 n+2 Line 6: n times [3 {2 “s” and 1 “+”}] = 3 n Line 7: 1 Total count = 5 n+4 which gives the function O(n). 17

General Rules: RULE 1: Loops: The running time of a loop is at most the running time of the statements inside the loop (including tests) times the number of iterations. Example: /* 1 */ for(int i = 0; i < N; i++) /* 2 */ c = a*a*a+2; inside we have 2 (*) and 1(+), i. e. , running time of 3. Total = 3*N (N is number of iterations). 18

RULE 2: Nested Loops: Analyze these inside out. The total running time of a statement inside a group of nested loops is the running time of the statement multiplied by the product of the sizes of all the loops. Example: /* 1 */ for(int i = 0; i < N; i++) /* 2 */ /* 3 */ for(int j = 0; j < M; j++) k++; k++ = k+1, i. e. , running time of 1. Total = 1*M*N = M*N. 19

RULE 3: Consecutive Statements: just add (the maximum is the one that counts highest among all, recall: Example: 2 n+ 2 /* 1 */ for(int i = 0; i < N; i++) nested loop: 2(N*N) /* 2 */ a[i] = 0; /* 3 */ for(i = 0; i < N; i++) /* 4 */ for(int j = 0; j < N; j++) /* 5 */ A[i] += i + j; O(n) O(n 2) inside we have 2 (+) and 1(+), i. e. , running time of 2. Total = 2*N (N is number of iterations). 20

RULE 4: If-Then-Else: The running time of an if-then-else statement is never more than the running time of the test plus the larger of the running times of S 1 and S 2. Example: /* 1 */ if ( cond) /* 2 */ S 1; /* 3 */ else /* 4 */ S 2; Of course, there are cases for which we over estimate, but we are sure we never under estimate. 21

Factorial unsigned long Fact(int N) { if(N ==0 ) return 1; else return Fact(N-1)*N; } if(N == 0) return 1; O(n) if(N == 0) … else 2 return Fact(1 -1)*1; if(N == 0) … else return Fact(5 -1)*5; 2 if(N == 0) … else 2 return Fact(4 -1)*4; if(N == 0) … else return Fact(3 -1)*3; if(N == 0) … else return Fact(2 -1)*2; 22

unsigned long Fact(int N) { if(N < 0){ cout << “No Factorial of negative n"; return 0; if(N < 0){ } … else if(N == 0) else if(N ==0 ) … return 1; else return Fact(4 -1)*4; return Fact(N-1)*N; 3 } if(N < 0){ O(n) … else if(N == 0) … else return Fact(3 -1)*3; … else if(N == 0) … else return Fact(2 -1)*2; if(N < 0){ … else if(N == 0) … else return Fact(5 -1)*5; 3 if(N < 0){ … else if(N == 0) … else return Fact(1 -1)*1; if(N < 0){ … else if(N == 0) return 1; 2 23

Example: (recursion) An inefficient approach to find Fibonacci numbers {1 1 2 3 5 8 13 21 …. } /* 1 */ /* 2 */ /* 3 */ /* 4 */ /* 5 */ /* 6 */ /* 7 */ /* 8 */ /* 9 */ // Compute Fibonacci numbers unsigned long int Fib( const unsigned int N) { if( N <= 1) return 1; else return Fib(N-1) + Fib(N-2); } 24

If n = 0 or n =1, then from line 5: we will have T(0) = T(1) =1. For n = 2, we have line 5 and line 8. Line 8 consists of two function calls and one addition. Function calls are not easy to analyze and must be analyzed by themselves. Fib(N-1) requires T(N-1) units of time and Fib(N-2) requires T(N-2) units of time. The total time is: T(n) = T(n-1) + T(n-2) + 2 (2 = one for line 5 + one for the addition on line 8) 25

Using induction prove that: Base: T(0) = F(0) = 1 Inductive hypothesis: T(1) = F(1) = 1 This means that: is also assumed to be true. Using the definition, we can write: F(n) = F(n-1) + F(n-2). From the hypothesis induction: In previous page we showed that: T(n) = T(n-1) + T(n-2) + 2. Thus, or 26

Example: Replace with old data 27

We showed that: and in a similar way we can show that : This means that the running time grows exponentially. This is about as bad as possible. By using an array and a for loop, the running time can be reduced substantially. This program is slow because of the amount of redundant work. We redo some of the terms. MAXIM: “DON’T COMPUTE ANYTHING MORE THAN ONCE” 28

Maximum Subsequence Sum Problem /** ? ? maximum contiguous subsequence sum algorithm. int max. Sub. Sum 1( const vector<int> & a ) { /* 1*/ int max. Sum = 0; /* 2*/ /* 3*/ /* 4*/ for( int i = 0; i < a. size( ); i++ ) for( int j = i; j < a. size( ); j++ ) { int this. Sum = 0; /* 5*/ /* 6*/ for( int k = i; k <= j; k++ ) this. Sum += a[ k ]; /* 7*/ /* 8*/ if( this. Sum > max. Sum ) max. Sum = this. Sum; */ Algorithm 1 Determine complexity: O(n 3) } /* 9*/ } return max. Sum; 29

/** ? ? maximum contiguous subsequence sum algorithm. */ int max. Sub. Sum 2( const vector<int> & a ) { /* 1*/ int max. Sum = 0; /* 2*/ for( int i = 0; i < a. size( ); i++ ) { /* 3*/ /* 4*/ int this. Sum = 0; for( int j = i; j < a. size( ); j++ ) { /* 5*/ this. Sum += a[ j ]; /* 6*/ /* 7*/ if( this. Sum > max. Sum ) max. Sum = this. Sum; Algorithm 2 Determine complexity: O(n 3) } } /* 8*/ } return max. Sum; 30

/** Recursive maximum contiguous subsequence sum algorithm. * Finds maximum sum in subarray spanning a[left. . right]. * Does not attempt to maintain actual best sequence. */ int max. Sum. Rec( const vector<int> & a, int left, int right ) { /* 1*/ if( left == right ) // Base case /* 2*/ if( a[ left ] > 0 ) /* 3*/ return a[ left ]; else /* 4*/ return 0; /* 5*/ /* 6*/ /* 7*/ /* 8*/ /* 9*/ int center = ( left + right ) / 2; int max. Left. Sum = max. Sum. Rec( a, left, center ); int max. Right. Sum = max. Sum. Rec( a, center + 1, right ); int max. Left. Border. Sum = 0, left. Border. Sum = 0; for( int i = center; i >= left; i-- ) { /*10*/ /*11*/ /*12*/ left. Border. Sum += a[ i ]; if( left. Border. Sum > max. Left. Border. Sum ) max. Left. Border. Sum = left. Border. Sum; } 31

/*13*/ /*14*/ int max. Right. Border. Sum = 0, right. Border. Sum = 0; for( int j = center + 1; j <= right; j++ ) { /*15*/ /*16*/ /*17*/ right. Border. Sum += a[ j ]; if( right. Border. Sum > max. Right. Border. Sum ) max. Right. Border. Sum = right. Border. Sum; } /*18*/ /*19*/ } return max 3( max. Left. Sum, max. Right. Sum, max. Left. Border. Sum + max. Right. Border. Sum ); /** Driver for divide-and-conquer maximum contiguous subsequence sum algorithm. */ int max. Sub. Sum 3( const vector<int> & a ) { return max. Sum. Rec( a, 0, a. size( ) - 1 ); Algorithm 3 } Determine complexity: O(nlogn) 32

/** * Linear-time maximum contiguous subsequence sum algorithm. */ int max. Sub. Sum 4( const vector<int> & a ) { /* 1*/ int max. Sum = 0, this. Sum = 0; /* 2*/ /* 3*/ /* 4*/ /* 5*/ /* 6*/ /* 7*/ /* 8*/ } { } for( int j = 0; j < a. size( ); j++ ) this. Sum += a[ j ]; if( this. Sum > max. Sum ) max. Sum = this. Sum; else if( this. Sum < 0 ) this. Sum = 0; return max. Sum; Algorithm 4 Determine complexity: 33

Binary Search Given an integer X and integers A 0, A 1, …, AN-1, which are presorted and already in memory, find i such that Ai = X, or return i = -1 if X is not in the input. template <class Comparable> int binary. Search( const vector<Comparable> & a, const Comparable & x ) { /* 1*/ int low = 0, high = a. size( ) - 1; /* 2*/ while( low <= high ) { /* 3*/ int mid = ( low + high ) / 2; /* 4*/ if( a[ mid ] < x ) /* 5*/ low = mid + 1; /* 6*/ else if( a[ mid ] > x ) /* 7*/ high = mid - 1; else /* 8*/ return mid; // Found } /* 9*/ return NOT_FOUND; // NOT_FOUND is defined as -1 34 }

Euclid’s Algorithm This algorithm is used to find the greatest common divisor. The gcd of two integers is the largest integer that divides both. gcd(50, 15)=5. long gcd( long m, long n ) { /* 1*/ while( n != 0 ) { /* 2*/ long rem = m % n; /* 3*/ m = n; /* 4*/ n = rem; } Can you prove it is O(logn) /* 5*/ return m; } 35

Exponentiation Raising an integer to the poser of another integer. bool is. Even( int n ) { return n % 2 == 0; } long pow( long x, int n ) { /* 1*/ if( n == 0 ) /* 2*/ return 1; /* 3*/ if( n == 1 ) Can you prove it is O(logn) /* 4*/ return x; /* 5*/ if( is. Even( n ) ) /* 6*/ return pow( x * x, n / 2 ); else /* 7*/ return pow( x * x, n / 2 ) * x; } 36

Checking the Analysis One way is to write a program for the algorithm and try it for different size of input to see if produces the complexity. It is very difficult to differentiate linear programs from O(Nlog. N) programs purely based on empirical evidence. A trick that is commonly used is verify that a program is O(f(N) is to compute the values T(N)/f(N) for a range of N. 37