CS 3343 Analysis of Algorithms Lecture 4 Sum

  • Slides: 63
Download presentation
CS 3343: Analysis of Algorithms Lecture 4: Sum of series, Analyzing recursive algorithms 11/25/2020

CS 3343: Analysis of Algorithms Lecture 4: Sum of series, Analyzing recursive algorithms 11/25/2020 1

Outline • Review of last lecture • Sum of series • Analyzing recursive algorithms

Outline • Review of last lecture • Sum of series • Analyzing recursive algorithms 11/25/2020 2

L’ Hopital’s rule lim f(n) / g(n) = lim f(n)’ / g(n)’ n→∞ 11/25/2020

L’ Hopital’s rule lim f(n) / g(n) = lim f(n)’ / g(n)’ n→∞ 11/25/2020 n→∞ Condition: If both lim f(n) and lim g(n) = or 0 3

Stirling’s formula or (constant) 11/25/2020 4

Stirling’s formula or (constant) 11/25/2020 4

Properties of asymptotic notations • Textbook page 51 • Transitivity f(n) = (g(n)) and

Properties of asymptotic notations • Textbook page 51 • Transitivity f(n) = (g(n)) and g(n) = (h(n)) => f(n) = (h(n)) (holds true for o, O, , and as well). • Symmetry f(n) = (g(n)) if and only if g(n) = (f(n)) • Transpose symmetry f(n) = O(g(n)) if and only if g(n) = (f(n)) f(n) = o(g(n)) if and only if g(n) = (f(n)) 11/25/2020 5

logarithms • • lg n = log 2 n ln n = loge n,

logarithms • • lg n = log 2 n ln n = loge n, e ≈ 2. 718 lgkn = (lg n)k lg lg n = lg (lg n) = lg(2)n lg(k) n = lg lg lg … lg n lg 24 = ? lg(2)4 = ? Compare lgkn vs lg(k)n? 11/25/2020 6

Useful rules for logarithms • For all a > 0, b > 0, c

Useful rules for logarithms • For all a > 0, b > 0, c > 0, the following rules hold • logba = logca / logcb = lg a / lg b • logban = n logba = a b • • log (ab) = log a + log b – lg (2 n) = ? • log (a/b) = log (a) – log(b) – lg (n/2) = ? – lg (1/n) = ? • logba = 1 / logab 11/25/2020 7

Useful rules for exponentials • For all a > 0, b > 0, c

Useful rules for exponentials • For all a > 0, b > 0, c > 0, the following rules hold • a 0 = 1 (00 = ? ) • a 1 = a • a-1 = 1/a • (am)n = amn • (am)n = (an)m • aman = am+n 11/25/2020 8

More advanced dominance ranking 11/25/2020 9

More advanced dominance ranking 11/25/2020 9

General plan for analyzing time efficiency of a non-recursive algorithm • Decide parameter (input

General plan for analyzing time efficiency of a non-recursive algorithm • Decide parameter (input size) • Identify most executed line (basic operation) • worst-case = average-case? • T(n) = i ti • T(n) = Θ (f(n)) 11/25/2020 10

Find the order of growth for sums • • • T(n) = i=1. .

Find the order of growth for sums • • • T(n) = i=1. . n i = Θ (n 2) T(n) = i=1. . n log (i) = ? T(n) = i=1. . n n / 2 i = ? T(n) = i=1. . n 2 i = ? … • How to find out the actual order of growth? – Math… – Textbook Appendix A. 1 (page 1058 -60) 11/25/2020 11

Arithmetic series • An arithmetic series is a sequence of numbers such that the

Arithmetic series • An arithmetic series is a sequence of numbers such that the difference of any two successive members of the sequence is a constant. e. g. : 1, 2, 3, 4, 5 or 10, 12, 14, 16, 18, 20 • In general: Recursive definition Or: 11/25/2020 Closed form, or explicit formula 12

Sum of arithmetic series If a 1, a 2, …, an is an arithmetic

Sum of arithmetic series If a 1, a 2, …, an is an arithmetic series, then e. g. 1 + 3 + 5 + 7 + … + 99 = ? (series definition: ai = 2 i-1) This is ∑ i = 1 to 50 (ai) 11/25/2020 13

Geometric series • A geometric series is a sequence of numbers such that the

Geometric series • A geometric series is a sequence of numbers such that the ratio between any two successive members of the sequence is a constant. e. g. : 1, 2, 4, 8, 16, 32 or 10, 20, 40, 80, 160 or 1, ½, ¼, 1/8, 1/16 • In general: Recursive definition Or: 11/25/2020 Closed form, or explicit formula 14

Sum of geometric series if r < 1 if r > 1 if r

Sum of geometric series if r < 1 if r > 1 if r = 1 11/25/2020 15

Sum of geometric series if r < 1 if r > 1 if r

Sum of geometric series if r < 1 if r > 1 if r = 1 11/25/2020 16

Important formulas 11/25/2020 17

Important formulas 11/25/2020 17

Sum manipulation rules Example: 11/25/2020 18

Sum manipulation rules Example: 11/25/2020 18

Sum manipulation rules Example: 11/25/2020 19

Sum manipulation rules Example: 11/25/2020 19

Examples • i=1. . n n / 2 i = n * i=1. .

Examples • i=1. . n n / 2 i = n * i=1. . n (½)i = ? • using the formula for geometric series: i=0. . n (½)i = 1 + ½ + ¼ + … (½)n = 2 • Application: algorithm for allocating dynamic memories 11/25/2020 20

Examples • i=1. . n log (i) = log 1 + log 2 +

Examples • i=1. . n log (i) = log 1 + log 2 + … + log n = log 1 x 2 x 3 x … x n = log n! = (n log n) • Application: algorithm for selection sort using priority queue 11/25/2020 21

Problem of the day How do you find a coffee shop if you don’t

Problem of the day How do you find a coffee shop if you don’t know on which direction it might be? 11/25/2020 22

Recursive definition of sum of series • T (n) = i=0. . n i

Recursive definition of sum of series • T (n) = i=0. . n i is equivalent to: Recurrence relation T(n) = T(n-1) + n Boundary condition T(0) = 0 • T(n) = i=0. . n ai is equivalent to: T(n) = T(n-1) + an T(0) = 1 Recursive definition is often intuitive and easy to obtain. It is very useful in analyzing recursive algorithms, and some non-recursive algorithms too. 11/25/2020 23

Analyzing recursive algorithms 11/25/2020 24

Analyzing recursive algorithms 11/25/2020 24

Recursive algorithms • General idea: – Divide a large problem into smaller ones •

Recursive algorithms • General idea: – Divide a large problem into smaller ones • By a constant ratio • By a constant or some variable – Solve each smaller one recursively or explicitly – Combine the solutions of smaller ones to form a solution for the original problem Divide and Conquer 11/25/2020 25

Merge sort MERGE-SORT A[1. . n] 1. If n = 1, done. 2. Recursively

Merge sort MERGE-SORT A[1. . n] 1. If n = 1, done. 2. Recursively sort A[ 1. . n/2 ] and A[ n/2 +1. . n ]. 3. “Merge” the 2 sorted lists. Key subroutine: MERGE 11/25/2020 26

Merging two sorted arrays Subarray 1 Subarray 2 20 12 13 11 11/25/2020 7

Merging two sorted arrays Subarray 1 Subarray 2 20 12 13 11 11/25/2020 7 9 2 1 27

Merging two sorted arrays Subarray 1 Subarray 2 20 12 13 11 11/25/2020 7

Merging two sorted arrays Subarray 1 Subarray 2 20 12 13 11 11/25/2020 7 9 2 1 28

Merging two sorted arrays 20 12 13 11 7 9 2 1 11/25/2020 29

Merging two sorted arrays 20 12 13 11 7 9 2 1 11/25/2020 29

Merging two sorted arrays 20 12 13 11 7 9 2 1 11/25/2020 30

Merging two sorted arrays 20 12 13 11 7 9 2 1 11/25/2020 30

Merging two sorted arrays 20 12 13 11 7 9 2 1 1 11/25/2020

Merging two sorted arrays 20 12 13 11 7 9 2 1 1 11/25/2020 31

Merging two sorted arrays 20 12 13 11 7 9 7 2 1 2

Merging two sorted arrays 20 12 13 11 7 9 7 2 1 2 9 1 11/25/2020 32

Merging two sorted arrays 20 12 13 11 7 9 7 2 1 11/25/2020

Merging two sorted arrays 20 12 13 11 7 9 7 2 1 11/25/2020 9 2 33

Merging two sorted arrays 20 12 13 11 7 9 7 7 2 1

Merging two sorted arrays 20 12 13 11 7 9 7 7 2 1 11/25/2020 9 9 2 34

Merging two sorted arrays 20 12 13 11 7 9 7 7 2 1

Merging two sorted arrays 20 12 13 11 7 9 7 7 2 1 11/25/2020 9 2 9 7 35

Merging two sorted arrays 20 12 13 11 7 9 7 7 2 1

Merging two sorted arrays 20 12 13 11 7 9 7 7 2 1 11/25/2020 9 2 9 9 7 36

Merging two sorted arrays 20 12 13 11 7 9 7 7 2 1

Merging two sorted arrays 20 12 13 11 7 9 7 7 2 1 11/25/2020 9 2 9 7 9 9 37

Merging two sorted arrays 20 12 20 12 13 11 13 11 7 9

Merging two sorted arrays 20 12 20 12 13 11 13 11 7 9 7 7 2 1 11/25/2020 9 2 9 7 9 9 38

Merging two sorted arrays 20 12 20 12 13 11 13 11 7 9

Merging two sorted arrays 20 12 20 12 13 11 13 11 7 9 7 7 2 1 11/25/2020 9 2 9 7 9 9 11 39

Merging two sorted arrays 20 12 20 12 13 11 13 11 13 7

Merging two sorted arrays 20 12 20 12 13 11 13 11 13 7 9 7 7 2 1 11/25/2020 9 2 9 7 9 9 11 40

Merging two sorted arrays 20 12 20 12 13 11 13 11 13 7

Merging two sorted arrays 20 12 20 12 13 11 13 11 13 7 9 7 7 2 1 11/25/2020 9 2 9 7 9 9 11 12 41

How to show the correctness of a recursive algorithm? • By induction: – Base

How to show the correctness of a recursive algorithm? • By induction: – Base case: prove it works for small examples – Inductive hypothesis: assume the solution is correct for all sub-problems – Step: show that, if the inductive hypothesis is correct, then the algorithm is correct for the original problem. 11/25/2020 42

Correctness of merge sort MERGE-SORT A[1. . n] 1. If n = 1, done.

Correctness of merge sort MERGE-SORT A[1. . n] 1. If n = 1, done. 2. Recursively sort A[ 1. . n/2 ] and A[ n/2 +1. . n ]. 3. “Merge” the 2 sorted lists. Proof: 1. 2. 3. 11/25/2020 Base case: if n = 1, the algorithm will return the correct answer because A[1. . 1] is already sorted. Inductive hypothesis: assume that the algorithm correctly sorts A[1. . n/2 ] and A[ n/2 +1. . n]. Step: if A[1. . n/2 ] and A[ n/2 +1. . n] are both correctly sorted, the whole array A[1. . n/2 ] and A[ n/2 +1. . n] is sorted after merging. 43

How to analyze the time-efficiency of a recursive algorithm? • Express the running time

How to analyze the time-efficiency of a recursive algorithm? • Express the running time on input of size n as a function of the running time on smaller problems 11/25/2020 44

Analyzing merge sort T(n) MERGE-SORT A[1. . n] Θ(1) 1. If n = 1,

Analyzing merge sort T(n) MERGE-SORT A[1. . n] Θ(1) 1. If n = 1, done. 2 T(n/2) 2. Recursively sort A[ 1. . n/2 ] and A[ n/2 +1. . n ]. f(n) 3. “Merge” the 2 sorted lists Sloppiness: Should be T( n/2 ) + T( n/2 ) , but it turns out not to matter asymptotically. 11/25/2020 45

Analyzing merge sort 1. Divide: Trivial. 2. Conquer: Recursively sort 2 subarrays. 3. Combine:

Analyzing merge sort 1. Divide: Trivial. 2. Conquer: Recursively sort 2 subarrays. 3. Combine: Merge two sorted subarrays T(n) = 2 T(n/2) + f(n) +Θ(1) # subproblems subproblem size 11/25/2020 1. What is the time for the base case? 2. What is f(n)? 3. What is the growth order of T(n)? Dividing and Combining Constant 46

Merging two sorted arrays 20 12 20 12 13 11 13 11 13 7

Merging two sorted arrays 20 12 20 12 13 11 13 11 13 7 9 7 7 2 1 9 2 9 7 9 9 11 12 Θ(n) time to merge a total of n elements (linear time). 11/25/2020 47

Recurrence for merge sort T(n) = Θ(1) if n = 1; 2 T(n/2) +

Recurrence for merge sort T(n) = Θ(1) if n = 1; 2 T(n/2) + Θ(n) if n > 1. • Later we shall often omit stating the base case when T(n) = (1) for sufficiently small n, but only when it has no effect on the asymptotic solution to the recurrence. • But what does T(n) solve to? I. e. , is it O(n) or O(n 2) or O(n 3) or …? 11/25/2020 48

Binary Search To find an element in a sorted array, we 1. Check the

Binary Search To find an element in a sorted array, we 1. Check the middle element 2. If ==, we’ve found it 3. else if less than wanted, search right half 4. else search left half Example: Find 9 3 11/25/2020 5 7 8 9 12 15 49

Binary Search To find an element in a sorted array, we 1. Check the

Binary Search To find an element in a sorted array, we 1. Check the middle element 2. If ==, we’ve found it 3. else if less than wanted, search right half 4. else search left half Example: Find 9 3 11/25/2020 5 7 8 9 12 15 50

Binary Search To find an element in a sorted array, we 1. Check the

Binary Search To find an element in a sorted array, we 1. Check the middle element 2. If ==, we’ve found it 3. else if less than wanted, search right half 4. else search left half Example: Find 9 3 11/25/2020 5 7 8 9 12 15 51

Binary Search To find an element in a sorted array, we 1. Check the

Binary Search To find an element in a sorted array, we 1. Check the middle element 2. If ==, we’ve found it 3. else if less than wanted, search right half 4. else search left half Example: Find 9 3 11/25/2020 5 7 8 9 12 15 52

Binary Search To find an element in a sorted array, we 1. Check the

Binary Search To find an element in a sorted array, we 1. Check the middle element 2. If ==, we’ve found it 3. else if less than wanted, search right half 4. else search left half Example: Find 9 3 11/25/2020 5 7 8 9 12 15 53

Binary Search To find an element in a sorted array, we 1. Check the

Binary Search To find an element in a sorted array, we 1. Check the middle element 2. If ==, we’ve found it 3. else if less than wanted, search right half 4. else search left half Example: Find 9 3 11/25/2020 5 7 8 9 12 15 54

Binary Search Binary. Search (A[1. . N], value) { if (N == 0) return

Binary Search Binary. Search (A[1. . N], value) { if (N == 0) return -1; // not found mid = (1+N)/2; if (A[mid] == value) return mid; // found else if (A[mid] < value) return Binary. Search (A[mid+1, N], value) else return Binary. Search (A[1. . mid-1], value); } What’s the recurrence relation for its running time? 11/25/2020 55

Recurrence for binary search 11/25/2020 56

Recurrence for binary search 11/25/2020 56

Recursive Insertion Sort Recursive. Insertion. Sort(A[1. . n]) 1. if (n == 1) do

Recursive Insertion Sort Recursive. Insertion. Sort(A[1. . n]) 1. if (n == 1) do nothing; 2. Recursive. Insertion. Sort(A[1. . n-1]); 3. Find index i in A such that A[i] <= A[n] < A[i+1]; 4. Insert A[n] after A[i]; 11/25/2020 57

Recurrence for insertion sort 11/25/2020 58

Recurrence for insertion sort 11/25/2020 58

Compute factorial Factorial (n) if (n == 1) return 1; return n * Factorial

Compute factorial Factorial (n) if (n == 1) return 1; return n * Factorial (n-1); • Note: here we use n as the size of the input. However, usually for such algorithms we would use log(n), i. e. , the bits needed to represent n, as the input size. 11/25/2020 59

Recurrence for computing factorial • Note: here we use n as the size of

Recurrence for computing factorial • Note: here we use n as the size of the input. However, usually for such algorithms we would use log(n), i. e. , the bits needed to represent n, as the input size. 11/25/2020 60

What do these mean? Challenge: how to solve the recurrence to get a closed

What do these mean? Challenge: how to solve the recurrence to get a closed form, e. g. T(n) = Θ (n 2) or T(n) = Θ(nlgn), or at least some bound such as T(n) = O(n 2)? 11/25/2020 61

Solving recurrence • Running time of many algorithms can be expressed in one of

Solving recurrence • Running time of many algorithms can be expressed in one of the following two recursive forms or Both can be very hard to solve. We focus on relatively easy ones, which you will encounter frequently in many real algorithms (and exams…) 11/25/2020 62

Solving recurrence 1. Recursion tree / iteration method 2. Substitution method 3. Master method

Solving recurrence 1. Recursion tree / iteration method 2. Substitution method 3. Master method 11/25/2020 63