Complexity Analysis Part I Motivations for Complexity Analysis

Complexity Analysis (Part I) • Motivations for Complexity Analysis • Example of Basic Operations • Average, Best, and Worst Cases • Problem size as Real number • Summation formulas • Simple Complexity Analysis Examples • Why Big-O notation? 1

Complexity Analysis • A data structure is a scheme of arranging data in computer memory for efficient manipulation, together with the operations (algorithms) on the data • An algorithm is a finite, unambiguous sequence of steps for solving a problem in a finite amount of time and space. • A program is an implementation of an algorithm in a particular programming language. • The efficient implementation of algorithms is important in computing, and this depends on suitable data structures. 2

Motivations for Complexity Analysis • There are often many different algorithms to solve a particular problem. Thus, it makes sense to develop techniques that allow us to: Ø compare different algorithms with respect to their “efficiency” Ø choose the most efficient algorithm for a problem • The efficiency of any algorithmic solution to a problem is a measure of the: Ø Time efficiency: the time it takes to execute. Ø Space efficiency: the space (primary or secondary memory) it uses. • We will focus on an algorithm’s efficiency with respect to time. • Often time efficiency is more important than space complexity Ø More and more space available Ø time is still a problem 3

Machine independence • The evaluation of efficiency should be as machine independent as possible. • It is not useful to measure how fast the algorithm runs as this depends on which particular computer, OS, programming language, compiler, and kind of inputs that are used in testing. Also the algorithm has to be implemented. • Instead, Ø we count the number of basic operations the algorithm performs. Ø we calculate how this number depends on the size n of the input. • A basic operation is an operation which takes a constant amount of time to execute. • Hence, the efficiency of an algorithm is the number of basic operations it performs. This number is a function of the input size n. 4

Example of Basic Operations: • Arithmetic operations: *, /, %, +, - • Boolean operations: &&, ||, ! • Assignment statements of simple data types. • Reading of primitive types • writing of a primitive types • Simple conditional tests: • method calls (Note: the execution time of a method itself may not be constant) • a method's return statement. • Memory Access (includes array indexing) • We consider an operation such as ++ , += , and *= as consisting of two basic operations. • Note: To simplify complexity analysis we shall not consider memory access (fetch or store) operations if (x < 12). . . 5

Best, Average, and Worst case complexities • There are three cases in determining the efficiency of an algorithm: – Best-case complexity: B(n), the minimum time needed to execute an algorithm for an input of size n – Average-case complexity: A(n), the average time needed to execute an algorithm for an input of size n – Worst-case complexity: T(n), the maximum time needed to execute an algorithm for an input of size n • We are usually interested in the worst case complexity: what are the most operations that might be performed for a given problem size. – Easier to compute – Usually close to the actual running time – Crucial to real-time systems (e. g. air-traffic control) • Best case depends on the input • Average case is often difficult to compute 6

Best, Average, and Worst case complexities • Example: Linear Search Complexity – Best Case : Item found at the beginning: One comparison – Worst Case : Item found at the end or not found: n comparisons – Average Case : Item may be found at index 0, or 1, or 2, . . . or n – 1 • Average number of comparisons is: (1 + 2 +. . . + n) / n = (n+1) / 2 7

Problem size as Real number • Since the problem size n is a positive integer, the function f(n) for the running time of an algorithm is not continuous: f: Z+ • To simplify our analysis of algorithms we will consider n to be a positive real number. In that case f(n) is continuous. f: R+ • Since we will be analyzing algorithms for large input sizes; the behaviour of f(n) for small values of n is not important. 8

Useful Summation Formulas , x≠ 1 A special case of the above is: 9

Properties of Summations The sequence a 1, a 2, a 3, . . . , an is an arithmetic progression if ai+1 = ai + d, 1 <= i < (n -1) and d > 0 or d < 0 The sum of the arithmetic progression Sn is: 10

Simple Complexity Analysis: Loops • We start by considering how to count operations in loops. – We use integer division throughout. • If the number of iterations of a loop is n. – The initialization statement is executed one time – The loop condition is executed n + 1 times. – Each of the statements in the loop body is executed n times. – The loop-index update statement is executed n times. 11

Simple Complexity Analysis: Linear loops • A linear loop is one in which the loop index is updated by either addition or subtraction. • A loop is independent if its index values are independent of an outer loop index. • Let k and n be non-negative integers such that n >= k • Then in each of the following independent linear loops: for(int i = k; i < n; i++){ statement 1; statement 2; } for(int i = n; i > k; i--){ statement 1; statement 2; } • The number of iterations is: (n – k ) • The initialization statement is executed one time. • The condition is executed (n – k ) + 1 times. • The update statement is executed (n – k ) times. • Each of statement 1 and statement 2 is executed (n – k ) times. 12

Simple Complexity Analysis: Linear loops (cont’d) • Let k and n be non-negative integers such that n >= k • Then in each of the following independent linear loops: for(int i = k; i <= n; i++){ statement 1; statement 2; } for(int i = n; i >= k; i--){ statement 1; statement 2; } • The number of iterations is: (n – k ) + 1 • The initialization statement is executed one time. • The condition is executed (n – k ) + 2 times. • The update statement is executed (n – k ) + 1 times. • Each of statement 1 and statement 2 is executed (n – k ) + 1 times. 13

Simple Complexity Analysis: Loop Example • Find the exact number of basic operations in the following program fragment: double x, y; x = 2. 5 ; y = 3. 0; for(int i = 0; i < n; i++){ a[i] = x * y; x = 2. 5 * x; y = y + a[i]; } • There are 2 assignments outside the loop => 2 operations. • The for loop comprises: • An assignment i = 0 that is executed once => 1 operation • A test i < n that is executed n + 1 times => n + 1 operations • An increment i++ consisting of 2 operations that are executed n times => 2 n operations • the loop body that has three assignments, two multiplications, and an addition. Theses 6 operations are executed n times => 6 n operations Thus the total number of basic operations is 6 n + 2 n + (n + 1) + 3 = 9 n + 4 14

Simple Complexity Analysis: Loop Example • • We analyse the worst and best case complexities of get. Max Assume that the array is filled with n elements, where n >= 2 public int get. Max(int[ ] array, int n){ int current. Max = array[0]; for(int i = 1; i < n; i++){ if(array[i] > current. Max){ current. Max = array[i]; // statement 1 } } return current. Max; } • The loop iterates n - 1 times • The worst case occurs when the maximum element is the last one in the array; in that case statement 1 will be executed n - 1 times Hence we have the worst case time complexity: T(n) = 2 + n + (n – 1)(2 + 1) + 1 = 5 n – 2 • • • The best case occurs when the maximum element is the first one in the array; in that case statement 1 will not be executed. Hence we have the best case time complexity: B(n) = 2 + n + (n – 1)(2 + 1) + 1 = 4 n + 1 15

Simple Complexity Analysis: Loop Example • • We analyse the worst and best case complexities of get. Sum Assume that the array is filled with n elements, where n >= 2 public int get. Sum(int[ ] array, int n){ int sum = 0; for(int i = 0; i < n; i++){ sum = sum + array[i]; } return sum; } • The loop iterates n times. • The number of iterations does not depend on the input data • Hence we have the worst case time complexity is equal to the best-case time complexity: T(n) = B(n) = 1 + (n + 1) + n(2 + 1) + 1 = 5 n + 4 16

Simple Complexity Analysis: Logarithmic loops • A logarithmic loop is one in which the loop index is updated by either multiplication or division. • In each of the following independent logarithmic loops: for(int i = 1; i < n; i = i * 2){ statement 1; statement 2; } for(int i = n; i > 1; i = i /2){ statement 1; statement 2; } – The number of iterations is: log 2 n • In each of the following independent logarithmic loops: for(int i = 1; i <= n; i = i * 2){ statement 1; statement 2; } for(int i = 1; i <= n; i = i / 2){ statement 1; statement 2; } – The number of iterations is: log 2 + 1 17

Simple Complexity Analysis: Logarithmic loops • We prove that the number of iterations of an independent logarithmic loop is of the form log n + c where c is a constant. • Consider the following independent logarithmic loop int i = n; while(i >= 1){ statement 1; i = i /2; } • Without loss of generality, assume n is a multiple of 2, i. e. , n = 2 k k = log 2 n Iteration# Value of i at start of each iteration Number of times statement 1 is executed in each iteration 1 2 n / 21 1 3 n / 22 1 . . k+1 n / 2 k 1 k+2 n / 2(k + 1) 0 (loop not executed) Total number of times statement 1 is executed = 1 +. . . + 1 = k + 1 = log 2 n + 1 18

Simple Complexity Analysis: Independent nested loops • Example: Assume n is a multiple of 2. Find the maximum number of basic operations, T(n), for the following fragment: int k, sum; for(int i = 1; i <= n; i = i * 2){ k = n; sum = 0; while(k >= 0){ sum = sum + (i + k); k- -; } System. out. println(sum); } – The number of iterations of the outer loop is: log 2 n + 1 – For each outer index value, the number of iterations of the inner loop is: n + 1 – The number of basic operations is: 1 + (log 2 n + 2) + (log 2 n + 1) [ 2 + 3 + (n + 2) + (n + 1)(3 +2) ] i. e. , 6 nlog 2 n + 6 n + 13 log 2 n + 15 Note: For independent loops: Total. Cost = initialization and condition costs of outer loop + Number of outer loop iterations *[update. Cost of outer loop + Cost of inner loop + cost of other outer loop statements] 19

Simple Complexity Analysis: Dependent nested loops • Two nested loops are dependent if the inner index is dependent of the outer index. • Example: Find T(n) the maximum number of basic operations for the following fragment: int sum; for(int i = 1; i <= n; i++){ sum = 0; for(int k = 1; k <= i; k++){ sum = sum + (i + k); } System. out. println(sum); } – The number of iterations of the outer loop is: n – The number of times the inner loop is not executed is 0 – For all outer index values, the number of iterations of the inner loop is: 1 + 2 + 3 +. . . + n = n(n + 1)/2 – The number of times the inner loop condition is executed is: 2 + 3 + 4 +. . . + (n + 1) + 0 = n((n+1) + 2)/2 = n 2 /2 + 3 n/2 – T(n) = 1 + (n + 1) + n(2 + 1 + 1) + [n 2/2 + 3 n/2] + 5 n(n + 1)/2 = 3 n 2 + 10 n + 2 • For dependent loops: Total. Cost = Cost Of Inner Loop + Cost of other statements in outer loop + initialization, update, and condition costs of the two loops 20

Simple Complexity Analysis: Dependent nested loops (Cont’d) • Example: Find T(n) the maximum number of basic operations for the following fragment: int sum; for(int i = 1; i <= n; i++){ sum = 0; for(int k = 4; k <= i; k++){ sum = sum + (i + k); } System. out. println(sum); } – The number of iterations of the outer loop is: n – The number of times the inner loop is not executed is 3 – For outer index values j >= 4, the number of iterations of the inner loop is: 1 + 2 + 3 +. . . + (n – 3) = (n – 3)(n - 2)/2 – The number of times the inner loop condition is executed is: [2 + 3 + 4 +. . . + (n – 2)] + 3 = (n – 3)(n)/2 + 3 = n 2/2 - 3 n/2 + 3 – T(n) = 1 + (n + 1) + n(2 + 1 + 1) + [n 2/2 - 3 n/2 + 3] + 5(n – 3)(n - 1)/2 = 3 n 2 - 11 n/2 + 25/2 21

Simple Complexity Analysis: Examples • Suppose n is a multiple of 2. Determine the exact number of basic operations performed by the method my. Method 1: static int my. Method 1(int n){ int sum = 0; for(int i = 1; i < n; i = i * 2) sum = sum + i + helper(n); return sum; } • static int helper(int m){ int sum = 0; for(int i = 1; i <= m; i++) sum = sum + i; return sum; } The number of iterations of the My. Method 1 loop is: log 2 n • The loops in my. Method 1 and helper are independent • The number of iterations of the loop in helper is n • The method call is one operation Hence the number of basic operations is: 1 + (1 + log 2 n) + log 2 n [2 + 4 + 1 + (n + 1) + n(2 + 2) + 1] + 1 = 3 + log 2 n[10 + 5 n] + 1 = 5 n log 2 n + 11 log 2 n + 4 22

Simple Complexity Analysis: Complex Loops • Suppose n is a power of 2. Determine the number of basic operations performed by the method my. Method 2: static int my. Method 2(int n){ int sum = 0; for(int i = 1; i <= n; i = i * 2) sum = sum + i + helper(i); return sum; } static int helper(int m){ int sum = 0; for(int j = 1; i <= m; j++) sum = sum + j; return sum; } • The loop in my My. Method 2 iterates log 2 n + 1 times • The method call is one basic operation • The helper method initialization is executed log 2 n + 1 times • Since n is a power of 2; let it be n = 2 k • The number of times the dependent loop in helper is not executed is 0 • The number of iterations of the dependent loop in helper = 20 + 21 + 22 + 23 +. . . + 2 k = 2 k+1 – 1 = 2*2 k – 1 = 2 n – 1 0 1 2 The condition in helper is executed: (2 + 1) + (23 + 1) +. . . + (2 k + 1) times = (2 k+1 – 1) + (k + 1) = (2 n – 1) + (log 2 n + 1) = 2 n + log 2 n • The number of basic operations is: 1 + (log 2 n + 2) + (log 2 n + 1)(6 + 3) + (2 n + log 2 n) + (2 n – 1)(4) = 10 n + 10 log 2 n + 9 23

Why Big-O notation? • In this lecture we determined worst case running time T(n) by counting the exact number of basic operations. • Counting the exact number of basic operations is difficult and it is usually not necessary. • In the next lecture we introduce a method of approximating T(n) called the Big-O notation that gives us an upper bound on the running time of an algorithm for very large input sizes. • The rules for computing and manipulating Big-O expressions greatly simplify the analysis of the running time of an algorithm when all we are interested in is its asymptotic behavior (i. e. , its behaviour for very large input sizes). 24