Algorithm Analysis Chris Kiekintveld CS 2401 Fall 2010

Algorithm Analysis Chris Kiekintveld CS 2401 (Fall 2010) Elementary Data Structures and Algorithms

Algorithm Analysis s There are many different algorithms to solve the same problem s Ask 5 programmers to write a non-trivial program, you will get 5 different solutions s Which is best? s Correctness s Efficiency Java Programming: Program Design Including Data Structures 2

Computational Resources s Algorithms require resources to run s s Time (processor operations) Space (computer memory) Network bandwidth Programmer time s Two types of costs s Fixed: same every time we run the algorithm s Variable: depends on the size of the input Java Programming: Program Design Including Data Structures 3

Measuring Resource Use s How can we compare the resources used by different algorithms? s Empirical s Code both algorithms s Run them an record the resources used s You did this in the Fibonacci lab! Java Programming: Program Design Including Data Structures 4

Empirical Analysis Problems s Depends on code quality/implementation s Better/worse programmers, not the algorithm itself s Depends on computer speed/architecture s Depends on language/compiler efficiency s Depends on the input s E. g. linear search is very fast for some inputs, and very slow for others Java Programming: Program Design Including Data Structures 5

Analytical Approach s Analyze the algorithm itself s Abstract away from implementation details s How many operations will be executed? s How much memory is used? s Consider different cases (depending on input) s Best s Worst s Average Java Programming: Program Design Including Data Structures 6

Counting Operations int i = 2; int j = 2; int k = i + j; System. out. println(k); How many operations are there? Assignment: Addition: Print: Total: Java Programming: Program Design Including Data Structures 7

Counting Operations int i = 2; int j = 2; int k = i + j; System. out. println(i+j+k); How many operations are there? Assignment: Addition: Print: Total: Java Programming: Program Design Including Data Structures 8

Counting Operations int i = 0; while (i < 10) { System. out. println(i); i++; } How many operations are there? Assignment: Comparison: Increment: Print: Total: Java Programming: Program Design Including Data Structures 9

Counting Operations for (int i=0; i < 10; i++) { System. out. println(i); } How many operations are there? Assignment: Comparison: Increment: Print: Total: Java Programming: Program Design Including Data Structures 10

Counting Operations for (int i=0; i < n; i++) { System. out. println(i); } How many operations are there? Assignment: Comparison: Increment: Print: Total: Java Programming: Program Design Including Data Structures 11

Counting Operations for (int i=0; i < n; i++) { for (int j=0; j < n; j++) { System. out. println(i); } } How many operations are there? Assignment: Comparison: Increment: Print: Total: Java Programming: Program Design Including Data Structures 12

Counting Operations s So far, we have counted every operation s This is quite tedious, especially for infrequent operations s Focus on the most important operation s Most frequent s May need to figure out what this is Java Programming: Program Design Including Data Structures 13

Another Look at Search Algorithms s We have discussed two ways to search a list s Linear search (unordered data) s Binary search (sorted data) s Data is sorted by “keys” s Unique for each element s Well-defined order Java Programming: Program Design Including Data Structures 14

Linear (Sequential) Search public int seq. Search(T[] list, int length, T search. Item) { int loc; boolean found = false; for (loc = 0; loc < length; loc++) { if (list[loc]. equals(search. Item)) { found = true; break; } } if (found) return loc; else return -1; } Java Programming: Program Design Including Data Structures 15

Sequential Search Analysis s The statements in the for loop are repeated several times s For each iteration of the loop, the search item is compared with an element in the list s When analyzing a search algorithm, you count the number of comparisons s Suppose that L is a list of length n s The number of key comparisons depends on where in the list the search item is located Java Programming: Program Design Including Data Structures 16

Sequential Search Analysis (continued) s Best case s The item is the first element of the list s You make only one key comparison s Worst case s The item is the last element of the list s You make n key comparisons s What is the average case Java Programming: Program Design Including Data Structures 17

Sequential Search Analysis (continued) s To determine the average case s Consider all possible cases s Find the number of comparisons for each case s Add them and divide by the number of cases s Average case s On average, a successful sequential searches half the list Java Programming: Program Design Including Data Structures 18

Binary Search public int binary. Search(T[] list, int length, T search. Item) { int first = 0; int last = length - 1; int mid = -1; boolean found = false; while (first <= last && !found) { mid = (first + last) / 2; Comparable<T> comp. Elem = (Comparable<T>) list[mid]; Java Programming: Program Design Including Data Structures 19

Binary Search (continued) if (comp. Elem. compare. To(search. Item) == 0) found = true; else if (comp. Elem. compare. To(search. Item) > 0) last = mid - 1; else first = mid + 1; } if (found) return mid; else return -1; }//end binary. Search Java Programming: Program Design Including Data Structures 20

Binary Search Example Figure 18 -1 Sorted list for a binary search Table 18 -1 Values of first, last, and middle and the Number of Comparisons for Search Item 89 Java Programming: Program Design Including Data Structures 21

Performance of Binary Search s Suppose that L is a sorted list of size n s And n is a power of 2 (n = 2 m) s After each iteration of the for loop, about half the elements are left to search s The maximum number of iteration of the for loop is about m + 1 s Also m = log 2 n s Each iteration makes two key comparisons s Maximum number of comparisons: 2(m + 1) Java Programming: Program Design Including Data Structures 22

Comparison: Linear vs Binary Worst case number of comparison List Size Linear Binary 4 4 6 8 8 8 32 32 12 512 20 1048576 42 Java Programming: Program Design Including Data Structures 23

Asymptotic Analysis: Motivation s So far, we have counted operations exactly s We don’t really care about the details s Computers execute billions of operations per second s A few here or there is negligible s Care about overall scalability s As the input size grows, does computation grow quickly or slowly? s Don’t lose the forest for the trees Java Programming: Program Design Including Data Structures 24

Asymptotic Analysis s Asymptotic means the study of the function f as n becomes larger and larger without bound s Consider functions g(n) = n 2 and f(n) = n 2 + 4 n + 20 s As n becomes larger and larger, the term 4 n + 20 in f(n) becomes insignificant s g(1000) = 1, 000 and f(1000) = 1, 004, 020 s You can predict the behavior of f(n) by looking at the behavior of g(n) Java Programming: Program Design Including Data Structures 25

Asymptotic Algorithm Analysis s Identify a function that describes the growth in runtime as the input gets large s An “upper bound” of sorts on the running time s Typically worst-case, but occasionally average case s Describe the number of operations done using a function s Focus only on most important operations s Ignore one time initializations, etc. Java Programming: Program Design Including Data Structures 26

Common Asymptotic Functions Table 18 -4 Growth Rate of Various Functions Java Programming: Program Design Including Data Structures 27

Common Functions, visual Figure 18 -9 Growth Rate of Various Functions Java Programming: Program Design Including Data Structures 28

Java Programming: Program Design Including Data Structures 29

Asymptotic Notation: Big-O Notation (continued) Table 18 -7 Some Big-O Functions That Appear in Algorithm Analysis Java Programming: Program Design Including Data Structures 30

Big-Oh Notation (Definition) A function f(n) is O(g(n)) if there exist positive constants c and n 0 such that: f(n) ≤ cg(n) for all n ≥ n 0 Java Programming: Program Design Including Data Structures 31

Big-Oh Notation s Translation: After some point, f(n) is always smaller than g(n) s “Some point” refers to increasing problem size s The constant c says that we don’t care about multipliers s So, 2 n and n have the same essential growth rate s 2 n is O(n) Java Programming: Program Design Including Data Structures 32

Asymptotic Notation: Big-O Notation (continued) Table 18 -8 Number of Comparisons for a List of Length n Java Programming: Program Design Including Data Structures 33