Outline Priority Queues Binary Heaps Randomized Mergeable Heaps

Outline • Priority Queues • Binary Heaps • Randomized Mergeable Heaps

Priority Queues A priority queue stores a collection of Comparable elements • • Operations: • add(x) : Add x to the collection • find. Min() : Return the smallest element in the collection • delete. Min() : Remove the smallest element in the collection • In the JCF Queue interface, these are: • add(x)/offer(x) : add(x) • peek() : find. Mind() • remove()/poll() : delete. Min()

Priority Queues: Applications We can sort a set of elements using a priority queue. How? • We just need to insert them all into a priority queue and then removing them • public static <T> void sort (T[] a) { Queue <T> pq = new Priority. Queue <T >(); for (T x : a) pq. add(x); for (int i = 0; i < a. length ; i++) a[i] = pq. remove (); }

Priority Queues: Applications In simulations, priority queues store events ordered by time of occurrence •

Priority Queues: Applications In simulations, priority queues store events ordered by time of occurrence •

Priority Queues: Applications In simulations, priority queues store events ordered by time of occurrence •

Priority Queues: Applications In simulations, priority queues store events ordered by time of occurrence •

Priority Queues: Applications In simulations, priority queues store events ordered by time of occurrence •

Priority Queues Which structure should we use to implement a priority queue? • Priority queues can be implemented as (binary) heaps • A binary heap is a binary tree where each node u stores a value u. prio • • Heap property: • u. prio < u. left. prio and u. prio < u. right. prio

Priority Queues A binary heap is a binary tree where each node u stores a value u. prio Heap property: • • • u. prio < u. left. prio and u. prio < u. right. prio 0. 1 0. 3 0. 5 0. 9 2. 1 0. 8 1. 1 1. 3 9. 2 5. 1 1. 9 0. 4

Priority Queues • A complete heap uses a complete binary tree: d+1 - 1 nodes A complete binary tree of height d has up to 2 • d+1 - 1 ≥ n To store n nodes, we require 2 • 0. 1 0. 3 0. 5 0. 9 2. 1 0. 8 1. 1 1. 3 9. 2 5. 1 d+1 ≥ n + 1 2 • • d+1 ≥ log 2(n + 1) • d ≥ log 2(n + 1) – 1 = O(log n) • 1. 9 A complete heap of size n has height O(log n) 0. 4

Priority Queues How can we maps the nodes of a complete binary tree to the positions of an array? • • Eytzinger method: a[0] is the root • • the left child of a[i] is a[2*i+1] • the right child of a[i] is a[2*i+2] • the parent of a[i] is a[(i-1)/2]

Eytzinger method • Eytzinger method: 0 2 1 3 7 8 0 5 4 1 9 2 3 11 10 4 5 6 7 6 12 8 9 13 14 10 11 12 13 14

Eytzinger method • Eytzinger method: 0 2 1 3 7 8 root 9 11 10 Left child 0 i=0 5 4 1 2 3 4 6 12 13 14 Right child 5 a[2*0+1] = a[1] 6 7 8 9 10 11 12 13 14 a[2*0+2]= a[2]

Eytzinger method • Eytzinger method: 0 2 1 3 5 4 7 8 9 6 11 10 12 13 Left child a b c d e i=5 f g h i j 14 Right child k l m n o a[2*5+1] = a[11] a[2*0+2]= a[12]

Implicit Binary Heap An implicit binary heap represents a complete binary heap in an array a using the Eytzinger method • 0. 1 0. 3 0. 5 0. 9 2. 1 0. 1 • 0. 5 0. 8 1. 1 1. 3 9. 2 5. 1 0. 4 1. 9 0. 3 0. 9 1. 1 0. 8 0. 4 2. 1 1. 3 9. 2 5. 1 1. 9 No extra pointers, all the data lives in a

Implicit Binary Heap public class Binary. Heap <T extends Comparable <T>> extends Abstract. Queue <T> { protected T[] a; protected int n; protected int left (int i) { return 2*i + 1; } protected int right (int i) { return 2*i + 2; } protected int parent (int i) { return (i -1)/2; }. . . }

Finding the minimum • • How to find the minimum value in a heap? Finding the minimum value in a heap is easy • It's stored at the root • This is a[0] public T peek () { return a [0]; } • Runs in O(1) time

Inserting elements • How to insert an element into a heap? • To insert x into a (implicit binary) heap, we 1. 2. • add x as a leaf while x is smaller than parent(x) • swap x with its parent Runs in O(log n) time

Inserting elements 1. add x as a leaf 0. 1 0. 3 0. 5 0. 9 2. 1 0. 5 0. 8 1. 1 1. 3 9. 2 5. 1 1. 9 0. 4 0. 2 0. 3 0. 9 1. 1 0. 8 0. 4 2. 1 1. 3 9. 2 5. 1 1. 9 0. 2

Inserting elements 1. 2. add x as a leaf while x is smaller than parent(x) • swap x with its parent 0. 1 0. 3 0. 5 0. 9 2. 1 0. 5 0. 2 1. 1 1. 3 9. 2 5. 1 1. 9 0. 4 0. 8 0. 2 0. 3 0. 9 1. 1 0. 2 0. 4 2. 1 1. 3 9. 2 5. 1 1. 9 0. 8

Inserting elements 1. 2. add x as a leaf while x is smaller than parent(x) • swap x with its parent 0. 1 0. 2 0. 5 0. 9 2. 1 0. 5 0. 3 1. 1 1. 3 9. 2 5. 1 1. 9 0. 4 0. 8 0. 2 0. 9 1. 1 0. 3 0. 4 2. 1 1. 3 9. 2 5. 1 1. 9 0. 8

Inserting elements public boolean offer (T x) { if (n + 1 > a. length ) grow (); a[n++] = x; bubble. Up (n -1); return true ; } protected void bubble. Up (int i) { int p = parent (i); while (i > 0 && a[i]. compare. To (a[p]) < 0) { swap (i, p); i = p; p = parent (i); } }

Removing the minimum • How can we remove the minimum value from a heap? • To delete the minimum from a (implicit binary) heap, we 1. 2. • copy x=a[n-1] into a[0] while x is larger than either of its children • swap x with the smaller of its children Runs in O(log n) time

Removing the minimum 1. copy x=a[n-1] into a[0] 0. 1 0. 2 0. 5 0. 9 2. 1 0. 5 0. 3 1. 1 1. 3 9. 2 5. 1 1. 9 0. 4 0. 8 0. 2 0. 9 1. 1 0. 3 0. 4 2. 1 1. 3 9. 2 5. 1 1. 9 0. 8

Removing the minimum 1. 2. copy x=a[n-1] into a[0] while x is larger than either of its children • swap x with the smaller of its children 0. 2 0. 8 0. 2 0. 5 0. 9 2. 1 0. 2 0. 8 0. 5 0. 3 1. 1 1. 3 9. 2 5. 1 0. 4 1. 9 0. 8 0. 9 1. 1 0. 3 0. 4 2. 1 1. 3 9. 2 5. 1 1. 9 0. 2 0. 8

Removing the minimum 1. 2. copy x=a[n-1] into a[0] while x is larger than either of its children • swap x with the smaller of its children 0. 2 0. 3 0. 8 0. 5 0. 9 2. 1 0. 2 0. 5 0. 8 0. 3 1. 1 1. 3 9. 2 5. 1 0. 4 1. 9 0. 3 0. 4 2. 1 1. 3 9. 2 5. 1 1. 9 0. 8 0. 9 1. 1 0. 8 0. 3 0. 8

Removing the minimum public T poll () { T x = a [0]; swap (0, --n); trickle. Down (0); return x; }

Removing the minimum protected void trickle. Down (int i) { do { int j = -1; int r = right (i); if (r < n && a[r]. compare. To (a[i]) < 0) { int l = left (i); if (a[l]. compare. To (a[r]) < 0) { j = l; } else { j = r; } } else { int l = left (i); if (l < n && a[l]. compare. To (a[i]) < 0) { j = l; } } if (j >= 0) swap (i, j); i = j; } while (i >= 0); }

Summary • Theorem: A (implicit) binary heap supports the operations add(x) and delete. Min() in O(log n) (amortized) time and supports the find. Min() operation in O(1) time.

Building a heap • How can we build a heap from an unsorted array a? • How quickly can we make a into an implicit binary heap? • We can insert elements a[0], . . . , a[n-1] one at a time. public Binary. Heap (T[] ia) { a = ia; for (int n = 1; n < a. length ; n++) { add(a[n]); } } This takes O(1 + log(1)) + O(1 + log(2)) + O(1 + log n) Runs in O(nlog n) time • • Can we do better?

Building a heap • We can do it in O(n) time. How? • By working bottom up. First build n/2 heaps of size 1 Next build n/4 heaps of size 3 • Next build n/8 heaps of size 7 • • … • build 1 heap of size n public Binary. Heap (T[] ia) { a = ia; n = a. length ; for (int i = n/2; i >= 0; i --) { trickle. Down (i); } }

Building a heap in linear time (analysis) j times where j is the We call trickle. Down(i) , n/2 • root of a heap of size 2 j - 1)) = O(j) time trickle. Down(i) then takes O(log(2 • Total running time is • (n/4) * O(1) • • • (n/8) * O(2) (n/16) * O(3) … • 1 * O(log n) • = O(n)

Heapsort • The heapsort algorithm for sorting an array a of length n: • Make a into a heap in (O(n) time) • Repeat n times • Delete the minimum public T delete. Min () { swap (0, --n); trickle. Down (0); } • Each deletion takes O(log n) time • Runs in O(n log n) time • Doesn't require any extra space ( does all work inside of input array a)

Merging two heaps • We know how to add one element to a heap. • What can we do if we want to add another heap? • In other words, how can we merge 2 heaps? public Node<T> merge(Node<T> h 1, Node<T> h 2){ while(h 2. size()>0){ T u = h 2. peek(); h 2. delete. Min(); h 1. add(u); } } • The cost for delete. Min() and add() is O(log n) • We perform those operations n times • Total cost O(n log n)

Merging two heaps • Can we do better? • Actually we can do it in O(log n). How? • merge(h 1, h 2) can be defined recursively 1. If either of h 1 or h 2 is nil, we just return h 2 or h 1. 2. We take the one with the biggest minimum value and merge it either with the right or left child of the other. 3. We continue the process recursively.

Merging two heaps • What is the complexity of this operation? • in an implicit binary heap it would be O(n) • we need to copy all the elements in a new array • How can we reduce it? • Using a heap ordered binary tree structure we can reduce the cost to O(log n). Why?

Merging two heaps Node<T> merge(Node<T> h 1, Node<T> h 2) { if (h 1 == nil) return h 2; if (h 2 == nil) return h 1; if (compare(h 2. x, h 1. x) < 0) return merge(h 2, h 1); // now we know h 1. x <= h 2. x if (rand. next. Boolean()) { h 1. left = merge(h 1. left, h 2); h 1. left. parent = h 1; } else { h 1. right = merge(h 1. right, h 2); h 1. right. parent = h 1; } return h 1; }

Merging two heaps • Why we need the random function to merge the heaps? • How can we define the add() and remove() operations in a heap ordered binary tree ? public boolean add(T x) { Node<T> u = new. Node(); u. x = x; r = merge(u, r); r. parent = nil; n++; return true; } public T remove() { T x = r. x; r = merge(r. left, r. right); if (r != nil) r. parent = nil; n--; return x; }

Analysis of merge(h 1, h 2) • Lemma: The expected length of a random walk in a binary tree with n nodes is at most log(n+1) • proof by induction: For n=0 is true log(0+1)=0 • Suppose it is true for every n 0 < n • n 1 the size of the left sub-tree and n 2 = n - n 1 -1 • E[W] = 1 + log(n 1+1)/2 + log(n 2+1)/2 • E[W] ≤ 1 + log( (n-1)/2 + 1) = 1 + log( (n + 1)/2 ) = log(n + 1)

Summary • Lemma: The expected running time of merge(h 1, h 2) is at most O(log n), where n = h 1. size() + h 2. size() • Theorem: A Meldable. Heap implements the (priority) Queue interface and supports the operations add(x), remove() and merge(h 1, h 2) in O(log n), expected time per operation.

Recall Quicksort • quicksort(S): – Pick a random element p from S – Compare every element to p to get 3 sets • S< = {x ϵ S : x < p} • S= = {x ϵ S : x = p} • S> = {x ϵ S : x > p} – quicksort(S<) – quicksort(S>) 0 S< 4 1 2 1 3 3 4 S= 5 7 6 6 7 S> 9 8 8 9 p=5

Quicksort i 5 4 2 1 8 3 7 j 6 9 p=5 0 i j 0 4 2 1 8 3 7 6 9 5 > i j 0 4 2 1 8 5 3 7 6 9 8 i j 0 0 4 4 2 2 1 1 3 3 5 3 7 6 7 9 6 8 9 < 8 i=j

Quicksort 5 0 0 0 1 1 0 2 1 8 3 7 6 9 0 4 2 1 3 5 7 4 2 1 3 5 6 7 9 1 4 5 6 7 8 9 2 3 4 5 6 7 8 9 3 0 0 4 2 1 2 3 4 5 6 6 7 9 8 8 9 8

Recall Heapsort • The heapsort algorithm for sorting an array a of length n: • Make a into a heap in (O(n) time) • Repeat n times • Delete the minimum 5 4 2 1 8 3 7 6 9 0 0 5 2 1 4 3 7 6 9 8 0 1 2 3 4 5 6 7 8 9

Recall Heapsort 5 4 2 1 8 3 7 6 9 0 0 1 2 5 4 3 7 6 9 8 1 4 2 5 8 3 7 6 9 2 4 3 5 8 9 7 6 3 4 6 5 8 9 7 4 5 6 7 8 9 5 7 6 9 8 6 7 8 9 0 1 2 3 4 5 6

Recall Heapsort 3 4 6 5 8 9 7 4 5 6 7 8 9 5 7 6 9 8 6 7 8 9 7 9 8 8 9 9 0 1 2 3 4 5 6 7 8 9

Mergesort • The mergesort algorithm for sorting an array a of length n: • Divide a in 2 parts a 1 and a 2 • Recursively, sort a 1 and a 2 with mergesort • Merge a 1 and a 2 5 4 2 1 8 3 7 6 9 0 1 2 4 5 8 0 3 6 7 9 0 1 2 3 4 5 6 7 8 9

Merging process i 0++ i 0 1 2 4 5 i 1++ i 0 < i 1 8 i 1 < i 0 0 1 2 0 i 1 5 4 3 3 6 6 7 7 9 8 9

Analysis of Mergesort • The mergesort algorithm for sorting an array a of length n: 1. Divide a in 2 parts a 1 and a 2 2. Recursively, sort a 1 and a 2 with mergesort 3. Merge a 1 and a 2 • The merging process ( step 3 ) costs (O(n) time) • It is performed log n times. • Total costs (O(n log n) time) Theorem: The Mergesort algorithm can sort an array of n items in O(n log n) time

Sorting Algorithms • So far, we have seen 3 sorting algorithms: • Quicksort: O(n log n) expected time • Heapsort: O(n log n) time • Mergesort: O(n log n) time • Is there a faster sorting algorithm (maybe O(n) time)? • Answer: No and yes

Comparision based Sorting Algorithms • The algorithm Quicksort, Heapsort and Mergesort are sort by comparing the elements. • Every comparison-based sorting algorithm takes (n log n) time for some input • A comparison tree is a full binary tree: • each internal node u is labelled with a pair u. i and u. j • each leaf is labelled with a permutation of { 0, . . . , n – 1}

Comparision based Sorting Algorithms < < > a[0](>/<)a[1] > a[1](>/<)a[2] a[0](>/<)a[2] > < a[0]<a[1]<a[2] < a[0]<a[2]<a[1] a[0](>/<)a[2] > a[1]<a[0]<a[2] a[1](>/<)a[2] > a[2]<a[0]<a[1] < a[1]<a[2]<a[0] a[2]<a[1]<a[0]

Comparision based Sorting Algorithms • Lemma: Every comparison tree that sorts any input of length n has at least n! Leaves • Theorem: Every comparison tree that sorts any input of length n has height at least (n/2)log 2(n/2) • The height is log 2(n!). • n!=O(nn). • log 2(nn) = n log 2(n). • The height is O( n log n ).

Sorting in O(n) time • In-class problem: • Design an algorithm that takes an array a of n integers in the range {0, . . . , k-1} and sorts them in O(n + k) time

Counting sort int [] counting. Sort (int [] a, int k) { int c[] = new int [k]; for (int i = 0; i < a. length ; i ++) c[a[i ]]++; for (int i = 1; i < k; i ++) c[i] += c[i -1]; int b[] = new int [a. length ]; for (int i = a. length -1; i >= 0; i --) return b; } b[--c[a[i ]]] = a[i];

Sorting in O(n) time 1. First count the repetition of any possible number. 2. Compute the starting position of the number 3. Copy the numbers on the output array

Recall Heapsort 7 2 9 0 1 2 0 9 7 4 4 6 9 1 0 9 3 2 5 9 0 3 1 2 0 2 1 3 0 1 2 0 1 0 1 2 0 0 1 2 4 5 3 0 1 2 3 4 5 6 7 8 9 3 5 2 3 8 1 9 2 12 1 13 1 15 5 0 20 2 15 11 2+3 3+5 8+1 9+2 11+112+1 13+2 15+015+5 7 2 9 0 1 2 0 9 7 4 4 6 9 1 0 9 3 2 5 9 0 0 0 1 1 2 2 2 3 4 4 5 6 7 7 9 9 9

Radix sort • Radix-sort uses the counting sort algorithm to sort integers one “digit" at a time • integers have w bits • “digit" has d bits • uses w/d passes of counting-sort • Starts by sorting least-significant digits first • works up to most significant digits • Correctness depends on fact that counting sort is stable • if a[i] = a[j] and i < j then a[i] appears before a[j] in the output

Radix sort • Theorem: The radix-sort algorithm can sort an array a of n w-bit integers in O(n + 2 d) time • Theorem: The radix-sort algorithm can sort an array a of n integers in the range {0, . . . , nc – 1} in O(cn) time.

Summary • Comparison based sorting algorithms: • Quicksort: O(n log n) expected time • Heapsort: O(n log n) time • Mergesort: O(n log n) time • No comparison based sorting algorithms: • counting-sort: can sort an array a of n integers in the range {0, . . . , k-1} in O(n + k) time • radix-sort can sort an array a of n integers in the range {0, . . . , nc – 1} in O(cn) time.