Introduction to Algorithms 2 nd edition by Cormen

Introduction to Algorithms (2 nd edition) by Cormen, Leiserson, Rivest & Stein Chapter 3: Growth of Functions (slides enhanced by N. Adlai A. De. Pano)

Overview o Order of growth of functions provides a simple characterization of efficiency o Allows for comparison of relative performance between alternative algorithms o Concerned with asymptotic efficiency of algorithms o Best asymptotic efficiency usually is best choice except for smaller inputs o Several standard methods to simplify asymptotic analysis of algorithms

Asymptotic Notation o Applies to functions whose domains are the set of natural numbers: N = {0, 1, 2, …} o If time resource T(n) is being analyzed, the function’s range is usually the set of nonnegative real numbers: T(n) R+ o If space resource S(n) is being analyzed, the function’s range is usually also the set of natural numbers: S(n) N

Asymptotic Notation o Depending on the textbook, asymptotic categories may be expressed in terms of -a. set membership (our textbook): functions belong to a family of functions that exhibit some property; or b. function property (other textbooks): functions exhibit the property o Caveat: we will formally use (a) and informally use (b)

The Θ-Notation Θ(g(n)) = { f(n) : ∃c 1, c 2 > 0, n 0 > 0 s. t. ∀n ≥ n 0: c 1 · g(n) ≤ f(n) ≤ c 2 ⋅ g(n) } c 2 ⋅ g f c 1 ⋅ g n 0

The O-Notation O(g(n)) = { f(n) : ∃c > 0, n 0 > 0 s. t. ∀n ≥ n 0: f(n) ≤ c ⋅ g(n) } c⋅g f n 0

The Ω-Notation Ω(g(n)) = { f(n) : ∃c > 0, n 0 > 0 s. t. ∀n ≥ n 0: f(n) ≥ c ⋅ g(n) } f c⋅g n 0

The o-Notation o(g(n)) = { f(n) : ∀c > 0 ∃n 0 > 0 s. t. ∀n ≥ n 0: f(n) ≤ c ⋅ g(n) } c 3 ⋅ g c 2 ⋅ g c 1 ⋅ g f n 1 n 2 n 3

The ω-Notation ω(g(n)) = { f(n) : ∀c > 0 ∃n 0 > 0 s. t. ∀n ≥ n 0: f(n) ≥ c ⋅ g(n) } f c 3 ⋅ g c 2 ⋅ g c 1 ⋅ g n 1 n 2 n 3

Comparison of Functions o f (n) = O(g(n)) and g(n) = O(h(n)) ⇒ f (n) = O(h(n)) o f (n) = Ω(g(n)) and g(n) = Ω(h(n)) ⇒ f (n) = Ω(h(n)) o f (n) = Θ(g(n)) and g(n) = Θ(h(n)) ⇒ f (n) = Θ(h(n)) Transitivity o f (n) = O(f (n)) f (n) = Ω(f (n)) f (n) = Θ(f (n)) Reflexivity

Comparison of Functions o f (n) = Θ(g(n)) g(n) = Θ(f (n)) Symmetry o f (n) = O(g(n)) g(n) = Ω(f (n)) o f (n) = o(g(n)) g(n) = ω(f (n)) Transpose Symmetry o f (n) = O(g(n)) and f (n) = Ω(g(n)) f (n) = Θ(g(n)) Theorem 3. 1

Asymptotic Analysis and Limits

Comparison of Functions o f 1(n) = O(g 1(n)) and f 2(n) = O(g 2(n)) ⇒ f 1(n) + f 2(n) = O(g 1(n) + g 2(n)) o f (n) = O(g(n)) ⇒ f (n) + g(n) = O(g(n))

Standard Notation and Common Functions o Monotonicity A function f(n) is monotonically increasing if m n implies f(m) f(n). A function f(n) is monotonically decreasing if m n implies f(m) f(n). A function f(n) is strictly increasing if m < n implies f(m) < f(n). A function f(n) is strictly decreasing if m < n implies f(m) > f(n).

Standard Notation and Common Functions o Floors and ceilings For any real number x, the greatest integer less than or equal to x is denoted by x. For any real number x, the least integer greater than or equal to x is denoted by x. For all real numbers x, x 1 < x x x < x+1. Both functions are monotonically increasing.

Standard Notation and Common Functions o Exponentials For all n and a 1, the function an is the exponential function with base a and is monotonically increasing. o Logarithms Textbook adopts the following convention lg n = log 2 n (binary logarithm), ln n = logen (natural logarithm), lgk n = (lg n)k (exponentiation), lg lg n = lg(lg n) (composition), lg n + k = (lg n)+k (precedence of lg).

Standard Notation and Common Functions o Important relationships For all real constants a and b such that a>1, nb = o(an) that is, any exponential function with a base strictly greater than unity grows faster than any polynomial function. For all real constants a and b such that a>0, lgbn = o(na) that is, any positive polynomial function grows faster than any polylogarithmic function.

Standard Notation and Common Functions o Factorials For all n the function n! or “n factorial” is given by n! = n (n 1) (n 2) (n 3) … 2 1 It can be established that n! = o(nn) n! = (2 n) lg(n!) = (nlgn)

Standard Notation and Common Functions o Functional iteration The notation f (i)(n) represents the function f(n) iteratively applied i times to an initial value of n, or, recursively f (i)(n) = n if i=0 f (i)(n) = f(f (i 1)(n)) if i>0 Example: If f (n) = 2 n then f (2)(n) = f (2 n) = 22 n then f (3)(n) = f (f (2)(n)) = 2(22 n) = 23 n then f (i)(n) = 2 in

Standard Notation and Common Functions o Iterated logarithmic function The notation lg* n which reads “log star of n” is defined as lg* n = min {i 0 : lg(i) n 1 Example: lg* 2 = 1 lg* 4 = 2 lg* 16 = 3 lg* 65536 = 4 lg* 265536 = 5

Asymptotic Running Time of Algorithms o We consider algorithm A better than algorithm B if TA(n) = o(TB(n)) o Why is it acceptable to ignore the behavior of algorithms for small inputs? o Why is it acceptable to ignore the constants? o What do we gain by using asymptotic notation?

Things to Remember o Asymptotic analysis studies how the values of functions compare as their arguments grow without bounds. o Ignores constants and the behavior of the function for small arguments. o Acceptable because all algorithms are fast for small inputs and growth of running time is more important than constant factors.

Things to Remember o Ignoring the usually unimportant details, we obtain a representation that succinctly describes the growth of a function as its argument grows and thus allows us to make comparisons between algorithms in terms of their efficiency.