ITCS 6114 Algorithms and Data Structures Introduction Proof

ITCS 6114: Algorithms and Data Structures Introduction Proof by Induction Asymptotic Notation

The Course • Purpose: a rigorous introduction to the design and analysis of algorithms § Not a lab or programming course § Not a math course, either • Suggested Textbooks: Introduction to The Design & Analysis of Algorithms, Anany Levitin Introduction to Algorithms, Cormen, Leiserson, Rivest, Stein

The Course • Instructor: Zbigniew W. Ras § ras@uncc. edu § Office: Woodward Hall 430 C § Office hours: Tuesday: 12: 00 -1: 30 pm • TA: Ayman Hajja § ahajja@uncc. edu Office: Woodward Hall 404 (Future Computing Lab) § Office hours: Tuesday, Thursday 11: 30 -1: 30 pm §

The Course • • Grading policy: § Midterm: 30 points § Project: 30 points § Final: 30 points § Participation: 10 points Grade A from 86 to 100 points, Grade B from 71 to 85 points, Grade C from 56 to 70.

Review: Induction • Suppose § S(k) is true for fixed constant k o Often k = 0 § S(n) S(n+1) for all n >= k • Then S(n) is true for all n >= k David Luebke 5 6/6/2021

Proof By Induction • • Claim: S(n) is true for all n >= k Basis: § Show formula is true when n = k • Inductive hypothesis: § Assume formula is true for an arbitrary n • Step: § Show that formula is then true for n+1 David Luebke 6 6/6/2021

Induction Example: Gaussian Closed Form • Prove 1 + 2 + 3 + … + n = n(n+1) / 2 § Basis: o If n = 0, then 0 = 0(0+1) / 2 § Inductive hypothesis: o Assume 1 + 2 + 3 + … + n = n(n+1) / 2 § Step (show true for n+1): 1 + 2 + … + n+1 = (1 + 2 + …+ n) + (n+1) = n(n+1)/2 + n+1 = [n(n+1) + 2(n+1)]/2 = (n+1)(n+2)/2 = (n+1)(n+1 + 1) / 2 David Luebke 7 6/6/2021

Induction Example: Geometric Closed Form • Prove a 0 + a 1 + … + an = (an+1 - 1)/(a - 1) for all a 1 § Basis: show that a 0 = (a 0+1 - 1)/(a - 1) a 0 = 1 = (a 1 - 1)/(a - 1) § Inductive hypothesis: o Assume a 0 + a 1 + … + an = (an+1 - 1)/(a - 1) § Step (show true for n+1): a 0 + a 1 + … + an+1 = (an+1 - 1)/(a - 1) + an+1 = (an+1+1 - 1)/(a - 1) David Luebke 8 6/6/2021

Induction • • We’ve been using weak induction Strong induction also holds § Basis: show S(0) § Hypothesis: assume S(k) holds for arbitrary k <= n § Step: Show S(n+1) follows • Another variation: § Basis: show S(0), S(1) § Hypothesis: assume S(n) and S(n+1) are true § Step: show S(n+2) follows David Luebke 9 6/6/2021

Asymptotic Performance • In this course, we care most about asymptotic performance § How does the algorithm behave as the problem size gets very large? o Running time o Memory/storage requirements David Luebke 10 6/6/2021

Asymptotic Notation • By now you should have an intuitive feel for asymptotic (big-O) notation: § What does O(n) running time mean? O(n 2)? O(n lg n)? § How does asymptotic running time relate to asymptotic memory usage? • Our first task is to define this notation more formally and completely David Luebke 11 6/6/2021

Input Size • Time and space complexity § This is generally a function of the input size o E. g. , sorting, multiplication § How we characterize input size depends: o Sorting: number of input items o Multiplication: total number of bits o Graph algorithms: number of nodes & edges o Etc David Luebke 12 6/6/2021

Analysis • Worst case § Provides an upper bound on running time § An absolute guarantee • Average case § Provides the expected running time § Very useful, but treat with care: what is “average”? o Random (equally likely) inputs o Real-life inputs David Luebke 13 6/6/2021

An Example: Insertion Sort Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 14 6/6/2021

An Example: Insertion Sort 30 10 40 20 1 2 3 4 i = j = key = A[j] = A[j+1] = Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 15 6/6/2021

An Example: Insertion Sort 30 10 40 20 1 2 3 4 i=2 j=1 A[j] = 30 key = 10 A[j+1] = 10 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 16 6/6/2021

An Example: Insertion Sort 30 30 40 20 1 2 3 4 i=2 j=1 A[j] = 30 key = 10 A[j+1] = 30 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 17 6/6/2021

An Example: Insertion Sort 30 30 40 20 1 2 3 4 i=2 j=1 A[j] = 30 key = 10 A[j+1] = 30 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 18 6/6/2021

An Example: Insertion Sort 30 30 40 20 1 2 3 4 i=2 j=0 A[j] = key = 10 A[j+1] = 30 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 19 6/6/2021

An Example: Insertion Sort 30 30 40 20 1 2 3 4 i=2 j=0 A[j] = key = 10 A[j+1] = 30 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 20 6/6/2021

An Example: Insertion Sort 10 30 40 20 1 2 3 4 i=2 j=0 A[j] = key = 10 A[j+1] = 10 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 21 6/6/2021

An Example: Insertion Sort 10 30 40 20 1 2 3 4 i=3 j=0 A[j] = key = 10 A[j+1] = 10 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 22 6/6/2021

An Example: Insertion Sort 10 30 40 20 1 2 3 4 i=3 j=0 A[j] = key = 40 A[j+1] = 10 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 23 6/6/2021

An Example: Insertion Sort 10 30 40 20 1 2 3 4 i=3 j=0 A[j] = key = 40 A[j+1] = 10 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 24 6/6/2021

An Example: Insertion Sort 10 30 40 20 1 2 3 4 i=3 j=2 A[j] = 30 key = 40 A[j+1] = 40 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 25 6/6/2021

An Example: Insertion Sort 10 30 40 20 1 2 3 4 i=3 j=2 A[j] = 30 key = 40 A[j+1] = 40 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 26 6/6/2021

An Example: Insertion Sort 10 30 40 20 1 2 3 4 i=3 j=2 A[j] = 30 key = 40 A[j+1] = 40 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 27 6/6/2021

An Example: Insertion Sort 10 30 40 20 1 2 3 4 i=4 j=2 A[j] = 30 key = 40 A[j+1] = 40 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 28 6/6/2021

An Example: Insertion Sort 10 30 40 20 1 2 3 4 i=4 j=2 A[j] = 30 key = 20 A[j+1] = 40 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 29 6/6/2021

An Example: Insertion Sort 10 30 40 20 1 2 3 4 i=4 j=2 A[j] = 30 key = 20 A[j+1] = 40 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 30 6/6/2021

An Example: Insertion Sort 10 30 40 20 1 2 3 4 i=4 j=3 A[j] = 40 key = 20 A[j+1] = 20 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 31 6/6/2021

An Example: Insertion Sort 10 30 40 20 1 2 3 4 i=4 j=3 A[j] = 40 key = 20 A[j+1] = 20 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 32 6/6/2021

An Example: Insertion Sort 10 30 40 40 1 2 3 4 i=4 j=3 A[j] = 40 key = 20 A[j+1] = 40 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 33 6/6/2021

An Example: Insertion Sort 10 30 40 40 1 2 3 4 i=4 j=3 A[j] = 40 key = 20 A[j+1] = 40 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 34 6/6/2021

An Example: Insertion Sort 10 30 40 40 1 2 3 4 i=4 j=2 A[j] = 30 key = 20 A[j+1] = 40 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 35 6/6/2021

An Example: Insertion Sort 10 30 40 40 1 2 3 4 i=4 j=2 A[j] = 30 key = 20 A[j+1] = 40 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 36 6/6/2021

An Example: Insertion Sort 10 30 30 40 1 2 3 4 i=4 j=2 A[j] = 30 key = 20 A[j+1] = 30 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 37 6/6/2021

An Example: Insertion Sort 10 30 30 40 1 2 3 4 i=4 j=2 A[j] = 30 key = 20 A[j+1] = 30 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 38 6/6/2021

An Example: Insertion Sort 10 30 30 40 1 2 3 4 i=4 j=1 A[j] = 10 key = 20 A[j+1] = 30 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 39 6/6/2021

An Example: Insertion Sort 10 30 30 40 1 2 3 4 i=4 j=1 A[j] = 10 key = 20 A[j+1] = 30 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 40 6/6/2021

An Example: Insertion Sort 10 20 30 40 1 2 3 4 i=4 j=1 A[j] = 10 key = 20 A[j+1] = 20 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } David Luebke 41 6/6/2021

An Example: Insertion Sort 10 20 30 40 1 2 3 4 i=4 j=1 A[j] = 10 key = 20 A[j+1] = 20 Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } } Done! David Luebke 42 6/6/2021

Animating Insertion Sort • Check out the Animator, a java applet at: http: //www. sorting-algorithms. com/insertion-sort • Try it out with random, ascending, and descending inputs David Luebke 43 6/6/2021

Insertion Sort Insertion. Sort(A, n) { for i = 2 to n { key = A[i] j = i - 1; while (j > 0) and (A[j] > key) { A[j+1] = A[j] j = j - 1 } A[j+1] = key } How many times will } this loop execute? David Luebke 44 6/6/2021

Analysis • Simplifications § Ignore actual and abstract statement costs § Order of growth is the interesting measure: o Highest-order term is what counts Remember, we are doing asymptotic analysis ] As the input size grows larger it is the high order term that dominates ] David Luebke 45 6/6/2021

Upper Bound Notation • We say Insertion. Sort’s run time is O(n 2) § Properly we should say run time is in O(n 2) § Read O as “Big-O” (you’ll also hear it as “order”) • • In general a function § f(n) is O(g(n)) if there exist positive constants c and n 0 such that f(n) c g(n) for all n n 0 Formally § O(g(n)) = { f(n): positive constants c and n 0 such that f(n) c g(n) n n 0 David Luebke 46 6/6/2021

Insertion Sort Is O(n 2) • Proof § Suppose runtime is an 2 + bn + c • o If any of a, b, and c are less than 0 replace the constant with its absolute value § an 2 + bn + c (a + b + c)n 2 + (a + b + c)n + (a + b + c) § 3(a + b + c)n 2 for n 1 § Let c’ = 3(a + b + c) and let n 0 = 1 Question § Is Insertion. Sort O(n 3)? § Is Insertion. Sort O(n)? David Luebke 47 6/6/2021

Lower Bound Notation • • • We say Insertion. Sort’s run time is (n) In general a function § f(n) is (g(n)) if positive constants c and n 0 such that 0 c g(n) f(n) n n 0 Proof: § Suppose run time is an + b o Assume a and b are positive (what if b is negative? ) § an + b David Luebke 48 6/6/2021

Asymptotic Tight Bound • A function f(n) is (g(n)) if positive constants c 1, c 2, and n 0 such that c 1 g(n) f(n) c 2 g(n) n n 0 • Theorem § f(n) is (g(n)) iff f(n) is both O(g(n)) and (g(n)) David Luebke 49 6/6/2021

Practical Complexity David Luebke 50 6/6/2021

Practical Complexity David Luebke 51 6/6/2021

Practical Complexity David Luebke 52 6/6/2021

Practical Complexity David Luebke 53 6/6/2021

Practical Complexity David Luebke 54 6/6/2021

Other Asymptotic Notations • • • A function f(n) is o(g(n)) if positive constants c and n 0 such that f(n) < c g(n) n n 0 A function f(n) is (g(n)) if positive constants c and n 0 such that c g(n) < f(n) n n 0 Intuitively, § o() is like < § O() is like David Luebke § () is like > § () is like 55 § () is like = 6/6/2021