Complexity Analysis of Algorithms Input Algorithm Analysis of

(Complexity) Analysis of Algorithms Input Algorithm Analysis of Algorithms Output 1

How long does this take to open 1) know 2) don’t know. If know combination O(n) where n is number of rings. If the alphabet is size m, O(nm) n), O(2 nm), If don’t know combination O(2 Analysis of Algorithms 2

With a combination lock – the idea is q If you know the combination – you can open it fast. q If you do not know the combination – you can open it – but it will take you a very long time. q Analysis of Algorithms 3

Knapsack and Travelling Salesman problems Maximize the cost, Without exceeding the weight Minimize the route length, While visiting every city Analysis of Algorithms 4

Knapsack and Travelling Salesman problems O(2 n) Maximize the cost, Without exceeding the weight O(n!) Means n! = n. (n-1)… 2. 1 Minimize the route length, While visiting every city Analysis of Algorithms 5

With these types of problem, so called combinatorial problem. q We do not know a fast way to solve these problems q We just have to go thru all possible combinations. q Analysis of Algorithms 6

Measuring Complexity q We can measure the runtime of an algorithms with a clock (unreliable) n n q Different machines and software… Different inputs (sorted, random, reverse) We can theoretically analyse the runtime of an algorithms n n How many primitive instructions are executed. This is a reflection of actual runtime. Analysis of Algorithms 7

Which lines of code are costly 1. 2. 3. 4. Obviously, some lines in a algorithm are executed more than others. Often there is one line (or body) of code that is executed repeated. This lies at the heart of the algorithm and is responsible for most of the (time) cost. We could probe code, and put a counter at each line to see how many times executed. Analysis of Algorithms 8

For loop analysis for (int i = 0; i < n; i++) { // do some interest stuff in Java } Analysis of Algorithms 9

Single for loop with counters int n = 10; for (int i = 0; ((i < n) && (counter 1++ >= 0)); i++, counter 2++) { counter 3++; // do some interest stuff in Java } System. out. println("counter 1 = " + counter 1); System. out. println("counter 2 = " + counter 2); System. out. println("counter 3 = " + counter 3); Analysis of Algorithms 10

what is the complexity O(? ) counter 1 = 10 q counter 2 = 10 q counter 3 = 10 q Analysis of Algorithms 11

Nested for loop // nested for loop for (int i = 0; ((i < n) && (counter 1++ >= 0)); i++, counter 2++) { counter 3++; for (int j = 0; ((j < n) && (counter 4++ >= 0)); j++, counter 5++) { // do some even more interest stuff in Java counter 6++; } // do some interest stuff in Java } Analysis of Algorithms 12

what is the complexity O(? ) counter 1 q counter 2 q counter 3 q = = = 10 100 100 Analysis of Algorithms 13

Grouping Complexities 1. 2. 3. In order to compare runtimes, it is convenient to have some grouping schema. Think of runtimes as a function of the LENGTH of inputs. How do mathematicians group functions? Analysis of Algorithms 14

Functions q How might you “describe” these functions? What “group” do they belong to? n n n f(x) f(x) = = = 7 log(x + 3) 3 x + 5 4 x 2 + 15 x + 90 10 x 3 – 30 2(3 x + 3) Analysis of Algorithms 15

Functions q How might you “describe” these functions? What “group” do they belong to? n n n f(x) f(x) = = = Constant 7 log(x + 3) Logarithmic Linear 3 x + 5 4 x 2 + 15 x + 90 Quadratic 10 x 3 – 30 Cubic 2(3 x + 3) Exponential Analysis of Algorithms 16

Big-O Notation 1. 2. 3. Instead of using terms like linear, quadratic, and cubic, computer scientists use big-O notation to discuss runtimes. A function with linear runtime is said to be of order n. The shorthand looks like this: O(n) Analysis of Algorithms 17

Functions q If the following are runtimes expressed as functions of the number of inputs (x), we would label them as follows: 1. 2. 3. 4. 5. 6. f(x) f(x) = = = 7 log(x + 3) 3 x + 5 4 x 2 + 15 x + 90 10 x 3 – 30 2(3 x + 3) Analysis of Algorithms 18

Functions q If the following are runtimes expressed as functions of the number of inputs (x), we would label them as follows: 1. 2. 3. 4. 5. 6. f(x) f(x) = = = O(1) 7 log(x + 3) O(log n) O(n) 3 x + 5 4 x 2 + 15 x + 90 O(n 2) 10 x 3 – 30 O(n 3) 2(3 x + 3)O(2 n) Analysis of Algorithms 19

The common Big-O values 1. 2. 3. 4. 5. 6. 7. O(1) : constant O(log n): Logarithmic O(n) : Linear O(n log n) : n log n O(n 2): Quadratic O(n 3): Cubic O(2 n): exponential Analysis of Algorithms 20

Merge sort vs. Insertion sort. Analysis of Algorithms 21

Slide by Matt Stallmann included with permission. Comparison of Two Algorithms Insertion Sort is n 2 / 4 Merge sort is 2 n log n Sorting a million items: Insertion Sort takes roughly 70 hours Merge Sort takes roughly 40 seconds This is a slow machine, but if 100 x as fast then it’s 40 minutes versus less than 0. 5 seconds Analysis of Algorithms 22

Which number is ‘bigger? ’ 12 or 13 q It is easy to compare numbers q What about functions? ? ? q Analysis of Algorithms 23

Which function is ‘bigger? ’ Source: Wikipedia 24 Analysis of Algorithms

Constant Factors don’t Matter! q The growth rate is not affected by n n q constant factors or lower-order terms Examples n n 102 n + 105 is a linear function 105 n 2 + 108 n is a quadratic function Analysis of Algorithms 25

Big-Oh Notation q q Given functions f(n) and g(n), we say that f(n) is O(g(n)) if there are positive constants c and n 0 such that f(n) cg(n) for n n 0 Example: 2 n + 10 is O(n) n n 2 n + 10 cn (c 2) n 10/(c 2) Pick c = 3 and n 0 = 10 26 Analysis of Algorithms

Big-Oh Example q Example: the function n 2 is not O(n) n n 2 cn n c The above inequality cannot be satisfied since c must be a constant 27 Analysis of Algorithms

Pseudocode Example: find max element of an array Algorithm array. Max(A, n) Input array A of n integers Output maximum element of A current. Max A[0] for i 1 to n 1 do if A[i] current. Max then current. Max A[i] return current. Max Analysis of Algorithms 28

Primitive Operations 1. 2. 3. 4. 5. Basic computations performed by an algorithm Identifiable in pseudocode Largely independent from the programming language Exact definition not important Assumed to take a constant amount of time in the RAM model q Analysis of Algorithms Examples: n n n Evaluating an expression Assigning a value to a variable Indexing into an array Calling a method Returning from a method 29

Counting Primitive Operations q By inspecting the pseudocode, we can determine the maximum number of primitive operations executed by an algorithm, as a function of the input size Algorithm array. Max(A, n) current. Max A[0] for i 1 to n 1 do if A[i] current. Max then current. Max A[i] { increment counter i } return current. Max # operations 2 2 n 2(n 1) 1 Total Analysis of Algorithms 8 n 2 30

Computing Prefix Averages We further illustrate asymptotic analysis with two algorithms for prefix averages q The i-th prefix average of an array X is average of the first (i + 1) elements of X: A[i] = (X[0] + X[1] + … + X[i])/(i+1) q q Computing the array A of prefix averages of another array X has applications to financial analysis Analysis of Algorithms 31

Prefix Averages (Quadratic) The following algorithm computes prefix averages in quadratic time by applying the definition Algorithm prefix. Averages 1(X, n) Input array X of n integers Output array A of prefix averages of X #operations A new array of n integers n for i 0 to n 1 do n s X[0] n for j 1 to i do 1 + 2 + …+ (n 1) s s + X[j] 1 + 2 + …+ (n 1) A[i] s / (i + 1) n return A 1 Analysis of Algorithms 32

Arithmetic Progression q q The running time of prefix. Averages 1 is O(1 + 2 + …+ n) The sum of the first n integers is n(n + 1) / 2 n q There is a simple visual proof of this fact Thus, algorithm prefix. Averages 1 runs in O(n 2) time Analysis of Algorithms 33

Prefix Averages (Linear) The following algorithm computes prefix averages in linear time by keeping a running sum Algorithm prefix. Averages 2(X, n) Input array X of n integers Output array A of prefix averages of X A new array of n integers s 0 for i 0 to n 1 do s s + X[i] A[i] s / (i + 1) return A #operations n 1 n n n 1 Algorithm prefix. Averages 2 runs in O(n) time Analysis of Algorithms 34

General Guidelines The worst-case instructions determine worst-case behaviour, overall. (eg. A single n 2 statement means the whole algorithm is n 2. ) q Instant recognition: q n n Assignments/arithmetic is O(1), Loops are O(n), Two nested loops are O(n 2). How about three nested loops? Analysis of Algorithms 35