Lecture 2 Analysis of Algorithms How to estimate

Lecture 2 Analysis of Algorithms • How to estimate time complexity? • Analysis of algorithms • Techniques based on Recursions ACKNOWLEDGEMENTS: Some contents in this lecture source from slides of Yuxi Fu

ROADMAP • How to estimate time complexity? • Analysis of algorithms 9/14/2021 ‣ worst case, best case? ‣ Big-O notation Xiaojuan Cai 2

Observation System dependent effects. • Hardware: CPU, memory, cache, … • Software: compiler, interpreter, garbage collector, … • System: operating system, network, other applications, … Easy, but difficult to get precise measurements. 9/14/2021 Xiaojuan Cai 4

Mathematical model Total running time = sum of cost × frequency for all operations. – Cost depends on machine, compiler. – Frequency depends on algorithm, input data. % 1 -sum int count = 0; for (int i = 0; i < N; i++) if (a[i] == 0) count++; 9/14/2021 Xiaojuan Cai 5

Mathematical model ‣ Relative rather than Absolute. ‣ Machine independent. ‣ About large input instance • Time complexity of an algorithm is a function of the size of the input n. 9/14/2021 Xiaojuan Cai 6

Input size -- convention • Sorting and searching: the size of the array • Graph: the number of the vertices and edges • Computational geometry: the number points or line segments • Matrix operation: the dimensions of the input matrices • Number theory and Cryptography: the number of bits of the input. 9/14/2021 Xiaojuan Cai 8

Where are we? • How to estimate time complexity? • Analysis of algorithms 9/14/2021 ‣ worst case, best case? ‣ Big-O notation Xiaojuan Cai 9

Searching Problem: Searching Input: An integer array A[1. . . n], and an integer x Output: Does x exist in A? Does 45 exist in: 9, 8, 9, 6, 2, 56, 12, 5, 4, 30, 67, 93, 25, 44, 96 2, 4, 5, 6, 8, 9, 9, 12, 25, 30, 44, 56, 67, 93, 96 Linear searching v. s. Binary searching 9/14/2021 Xiaojuan Cai 10

Binary searching 2, 4, 5, 6, 8, 9, 9, 12, 25, 30, 44, 56, 67, 93, 96 mid low 9/14/2021 high Xiaojuan Cai 11

Quiz Linear Searching Binary Searching Best case 1 1 Worst case n log n+1 A. 1 9/14/2021 B. logn. C. n D. none of above Xiaojuan Cai 12

Computational Complexity Linear Searching Binary Searching n log n Worst case 10 -6 s per instruction Linear Searching Binary Searching n=1024 0. 001 s 0. 00001 s n=240 12. 7 day 0. 00004 s 9/14/2021 Xiaojuan Cai 13

Where are we? • How to estimate time complexity? • Analysis of algorithms 9/14/2021 ‣ worst case, best case? ‣ Big-O notation Xiaojuan Cai 14

Big-O Notation Definition (O-notation) Let f(n) and g(n) : f(n) is said to be O(g(n)), written f (n) = O(g(n)), if∃c>0. ∃n 0. ∀n ≥ n 0. f (n) ≤ cg(n) • The O-notation provides an upper bound of the running time; • f grows no faster than some constant times g. 9/14/2021 Xiaojuan Cai 15

Examples • • • 2 10 n + 20 n = logn 2 =O(logn). log(n!) = O(nlogn). Hanoi: T(n) = O(2 n), n is the input size. Searching: T(n) = O(n) 9/14/2021 2 O(n ). Xiaojuan Cai 16

Big-Ω Notation Definition (Ω-notation) Let f(n) and g(n) : f(n) is said to be Ω(g(n)), written f (n) = Ω(g(n)), if∃c>0. ∃n 0. ∀n ≥ n 0. f (n) ≥ cg(n) • The Ω-notation provides an lower bound of the running time; • f(n) = O(g(n)) iff g(n) = Ω(f(n)). 9/14/2021 Xiaojuan Cai 17

Examples • • log nk = Ω(logn). log n! = Ω(nlogn). n! = Ω(2 n). Searching: T(n) = Ω(1) Does there exist f, g, such that f = O(g) and f = Ω(g)? 9/14/2021 Xiaojuan Cai 18

Big-Θ notation Definition (Θ-notation) f(n)=Θ(g(n)), if both f(n)=O(g(n)) and f(n) = Ω (g(n)). • log n! = Θ(nlogn). • 10 n 2 + 20 n = Θ(n 2) 9/14/2021 Xiaojuan Cai 19

Small o-notation Definition (o-notation) Let f(n) and g(n) : f(n) is said to be o(g(n)), written f (n) = o(g(n)), if∀c. ∃n 0. ∀n ≥ n 0. f (n) < cg(n) • log n! = o(nlogn)? 2 2 • 10 n + 20 n = o(n )? • nlog n = o(n 2)? 9/14/2021 Xiaojuan Cai 20

Small ω-notation Definition (ω-notation) Let f(n) and g(n) : f(n) is said to be ω(g(n)), written f (n) = ω(g(n)), if∀c. ∃n 0. ∀n ≥ n 0. f (n) > cg(n) 9/14/2021 Xiaojuan Cai 21

Quiz 9/14/2021 A. O f(n) g(n) n 1. 01 nlog 2 n nlogn log(n!) 1+1/2+1/3+. . . +1/n logn n 2 n 3 n B. Ω. C. Θ D. none of above Xiaojuan Cai 22

Approximation by Integration 9/14/2021 Xiaojuan Cai 26

In terms of limits Suppose 9/14/2021 exists Xiaojuan Cai 27

Complexity class Definition (Complexity class) An equivalence relation R on the set of complexity functions is defined as follows: f Rg if and only if f (n) = Θ(g (n)). 9/14/2021 Xiaojuan Cai 28

Order of growth order name 1 constant log N logarithmic example add two numbers binary search find the N = 1000 N = 2000 instant instant N linear N log N linearithmic mergesort instant N 2 quadratic 2 -sum ~1 s ~2 s 2 N exponential Hanoi forever maximum Assume the computer consumes 10 -6 s per instruction 9/14/2021 Xiaojuan Cai 29

Order of growth 9/14/2021 Xiaojuan Cai 30

Quiz A. Θ(n 2) 9/14/2021 B. Θ(n). C. Θ(nlogn) Xiaojuan Cai D. none of above 31

Quiz A. Θ(n 2) 9/14/2021 B. Θ(n). C. Θ(nlogn) Xiaojuan Cai D. none of above 32

Quiz A. Θ(n 2) 9/14/2021 B. Θ(n). C. Θ(nlogn) Xiaojuan Cai D. none of above 33

Quiz A. Θ(n 2) 9/14/2021 B. Θ(n). C. Θ(nlogn) Xiaojuan Cai D. none of above 34

Quiz A. Θ(n 2) 9/14/2021 B. Θ(n). C. Θ(nlogn) Xiaojuan Cai D. none of above 35

Quiz θ(nloglogn) A. Θ(n 2) 9/14/2021 B. Θ(n). C. Θ(nlogn) Xiaojuan Cai D. none of above 36

Amortized analysis The total number of append is n. The total number of delete is between 0 and n-1. So the time complexity is O(n). 9/14/2021 Xiaojuan Cai 37

Memory • Time: Time complexity ‣ faster, better. • Memory: Space complexity ‣ less, better. • Space complexity <= Time complexity 9/14/2021 Xiaojuan Cai 38

Conclusion • How to estimate time complexity? • Analysis of algorithms 9/14/2021 ‣ worst case, best case? ‣ Big-O notation Xiaojuan Cai 39