CS 113 Data Structures and Algorithms Abhiram G

CS 113: Data Structures and Algorithms Abhiram G. Ranade Heapsort and Heaps/Priority Queues

A generic algorithm for Sorting Input: A sequence of numbers x 1, …, xn Output: A permutation of {xi} in non decreasing order. A high level algorithm: 1. Let S denote a set of numbers, initially empty. 2. For i=1 to n: 3. Read xi and insert into S. 4. For i=1 to n: 5. Remove the smallest number from S. output it. Details that need to be determined: How to store S. How to insert into S. How to remove smallest.

Implementation 1: S = unsorted array A high level algorithm: 1. Let S denote a set of numbers, initially empty. 2. For i=1 to n: 3. Read xi and insert into S. 4. For i=1 to n: 5. Remove the smallest number from S. output it. Time requirement: • Inserting an element into S: placing at end. • Total time to insert: proportional to n. • Removing smallest out of n: n-1 comparisons. • Removing next smallest: n-2 comparisons • … • Total number of comparisons: n(n-1)/2 : proportional to n 2.

Another way: S = sorted array A high level algorithm: 1. Let S denote a set of numbers, initially empty. 2. For i=1 to n: 3. Read xi and insert into S. 4. For i=1 to n: 5. Remove the smallest number from S. output it. Time taken: • Time to insert ith element : find where to insert + create space. • Time to insert: proportional to i. • Total insertion time: proportional to 1+2+3+…+n-1. • Printing: just go from beginning to end. • Total: proportional to n 2.

A clever compromise Store input into array S maintaining “min-heap” order, but not full sorted order. Implications: • Each insertion can be done in time proportional to log n. • Each smallest removal can also be done in time proportional to log n. • Total for n elements: proportional to n log n.

“Min Heap” property/ ”Min Heap” order For every i: A[i] ≤ A[2 i+1], A[2 i+2] • A[0] ≤ A[1], A[2]. • A[1] ≤ A[3], A[4]. • … • No order implied between other pairs or elements, e. g. A[2], A[4]. • Let us visualize this pictorially: – Draw an edge from A[i] to A[2 i+1], 2[i+2] – Place A[i] above A[2 i+1], A[2 i+2]

Example of a min-Heap • A = (10, 12, 15, 26, 13, 18) 10 12 26 3 0 15 1 13 4 18 5 2

Basic ideas • All the keys that have been read reside in the array, with heap property obeyed. • Inserting a new key: – place in the next empty slot – rearrange keys so that heap order is maintained. • Removing smallest: – Smallest is at the top, can be removed. – Fill hole somehow.

How to maintain the min heap property while adding elements • Heap property is trivially satisfied if the array has at most 1 element • Suppose array has r>0 elements and heap property is satisfied for all elements. • We add a new element in position A[r]. • Heap property may not be valid for A[r]. • We perform exchanges to ensure property at all array elements.

Add new element 9 to A=(10, 12, 15, 26, 13, 18) A = (10, 12, 15, 26, 13, 18, 9) 10 12 26 3 0 15 1 13 4 18 5 2 9 6

Add new element 9 to A=(10, 12, 15, 26, 13, 18) A = (10, 12, 15, 26, 13, 18, 9) A = (10, 12, 9, 26, 13, 18, 15) 10 12 26 3 0 9 1 13 4 18 5 2 15 6

Add new element 9 to A=(10, 12, 15, 26, 13, 18) A = (10, 12, 15, 26, 13, 18, 9) A = (10, 12, 9, 26, 13, 18, 15) 9 12 26 3 A = (9, 12, 10, 26, 13, 18, 15) 0 10 1 13 4 18 5 2 15 6

Remark • If you turn the diagram upside down, it resembles a tree. • It is indeed called a ”Tree” • Top element (“vertex”) is called “root”, which it would correspond to upside down • Because there at most 2 branches going down, the tree is also called a binary tree. • Vertex A[i] is parent of A[2 i+1], A[2 i+2]. • Vertices A[2 i+1], A[2 i+2] are left and right children • Root is in level 0. Below is level 1, then level 2… • Level i has 2 i vertices.

The heap class heap{ const int n=30; int A[n]; // will hold keys int next; // next index at which to insert public: heap(){ next = 0; } bool add(int v); int remove(); private: int l(int i){return 2*i+1; } // index of left child int r(int i){return 2*i+2; } // index of right child int p(int i){return (i-1)/2; } // index of parent bool noleftchild(int i){return l(i) >= next; } bool norightchild(int i){return r(i) >= next; } }

Adding a new element • Add it to the next empty spot in the array. • Potentially heap order violated at the inserted element. • Exchange with parent. • Heap order violated with parent. • Exchange parent – grandparent. • . . . • Will this work in general?

Code for add bool heap: : add(int v){ if(next >= n) return false; //overflow A[next] = v; bubbleup(next); next++; return true; }

Bubbling up void heap: : bubbleup(int r){ while(r != 0 && A[p(r)] > A[r]){ exchange( A[p(r)], A[r] ); r = p(r); } } void heap: : exchange(int &u, int &v){. . . }

Correctness of Bubbleup • Heap property at i: A[p(i)] ≤ A[i] • Specification: – Assuming A[0. . r-1] satisfy heap property at the beginning, A[0. . r] should satisfy heap property only by exchanging elements. • As usual we must prove – Termination – Correctness

Termination • Can there be abnormal termination? – A[p(r)] is accessed only when r>0. – If r>0, p(r)=r-1/2 is at least 0 and at most r. – So no index out of bounds. • Will there be normal termination: – r is moving up towards the root – will eventually become the root. – The code exits if r == 0, i. e. r is root. The code may exit even earlier. • Is there a ”potential” in this?

Correctness • What invariant to use? • The natural choice is: – At the beginning (and end) of every iteration the heap property will hold at all A[j] except possibly at A[r]. • This is true, but cannot be proved on its own. – Suppose v = A[p(r)] > A[r] = u. – So we exchange and get A[p(r)] = u, A[r] = v. – We now need to argue that the heap property will hold at all vertices except p(r). – Consider l(r), left child s of r. – Our old invariant gives us no information about how A[l(r)] and v compare.

Proof using extended heap property EHP(j) : A[j] ≥ A[i] for all ancestors i of j Invariant: At the beginning (and end) of every iteration EHP(j) will hold at all j ≠ r Proof of base case: On first arrival, only r is new, so EHP holds elsewhere. For other iterations: Suppose at the beginning of the iteration r takes value R. If loop condition does not hold, no next iteration, so done. Else, at the end r = p(R). So we need to prove EHP for all j except p(R). • Case j = vertex not having R or p(R) as ancestor EHP(j) holds afterwards because it held earlier • Case j = vertex having p(R) as ancestor but not R Values in ancestors of j only reduce, so EHP(j) must hold afterwards too. • Case j = R EHP(p(R)) before exchange, A[p(R)] ≤ A[R] ⇒ EHP(R) after exchange. • Case j = vertex having R, p(R) as ancestors Ancestors of j or A[j] do not change, so EHP before implies EHP later.

Loop invariant + termination implies correctness • Case 1: Termination happened because r=0: – then there is no p(r) and hence the heap property is true at A[r], and is true elsewhere because of the invariant • Case 2: Termination happened because A[p(r)] ≤ A[r] : – The ancestors of A[p(r)] ≤ A[p(r)] because of the loop invariant. – But ancestors(A[r]) = A[p(r)] and its ancestors. So because of previous 2 statements these must be ≤ A[r]. – So no violation of A[r] also.

Time analysis • Number of iterations in trickledown: – In each iteration we move one level up. – number of levels in an n node tree: log 2 n – So log 2 n. (Proved next) • Time for single call to add: c+klog 2 n • Time for n insertions: proportional to nlog 2 n

Number of levels in a n element heap • Max number of elements in level i: 2 i. • All but last level will have max number. • If n element heap has levels 0. . L: 1+2+22+23+. . . 2 L-1 < n ≤ 1+2+22+. . . +2 L 2 L – 1 < n ≤ 2 L+1 – 1 L ≤ log n So number of levels, L+1 ≤ 2 log n, i. e. is at most proportional to log n.

Removing smallest • • Smallest is available at the root. Cannot just remove it and leave root “empty”. Heap property must be maintained. How? Exchanges? Which?

Note • We often needed to say “Time is at most proportional to f” for some f. • Instead of this we will say ”Time is O(f)”. • We will abuse “=“ symbol and write “Time = O(f)”. • Just remember that f = O(g) means that there is a constant c such that f < cg – f, g will be functions of a suitable parameter n, and c must be independent of n.

How should you remove? 10 15 12 26 13 18 24

How should you remove? 15 12 26 13 18 24

How should you remove? 12 15 26 13 18 24

How should you remove? 12 15 13 26 18 24

Idea 1 • Push the “space” downwards. • Will work, except that “holes” will get created in the array. – Book keeping will become more complex. – especially if we want to process a sequence of requests in which add and remove requests can be mixed up.

Idea 2 • Remove the element at the root. • Replace it with the last element of the array. • Now perform exchanges to fix heap order violation.

How should you remove? 10 15 12 26 13 18 24

How should you remove? 15 12 26 13 18 24

How should you remove? 24 15 12 26 13 18

How should you remove? 12 15 24 26 13 18

How should you remove? 12 15 13 26 24 18

Code for removing (smallest) int heap: : remove(){ if(next == 0) std: : abort(); int v = A[0]; A[0] = A[next]; next--; trickledown(0); return v; }

Recursive trickledown void heap: : trickledown(int i){ int nexti; if(noleftchild(i)) return; if(norightchild(i)){ if(A[i] <= A[l(i)]) return; nexti = l(i); } else nexti = A[l(i)]<A[r(i)]? l(i) : r(i); if(A[i] > A[nexti]{ exchange(A[i], A[nexti]); trickledown(nexti); } else {} // heap property already holds. return; }

Correctness of trickledown (sketch) • Downward Heap Property DHP(j) : A[j] <= A[all descendants of j]. • precondition: when trickledown(i) is called, DHP must hold at all j except i. • postcondition: DHP will hold everywhere. • Base case: i = leaf. post condition trivially true for i. • “Problem size” : height of i. • Induction hypothesis: trickledown(i) works correctly for height(i) = h. • We must prove that it works for height h+1.

Time required for trickledown • trickledown does a constant amount of work and then recursively calls itself on a node at a lower level. When the recursive call returns, trickledown returns. • So at each level trickledown does a constant amount of work. • So total amount of work done by trickledown is O(number of levels in the tree). • So O(log 2 n), where n = number of nodes in the tree. • So time for n removals, at most O(nlog 2 n).

Constructing a heap from an arbitrary array input: array A[0. . n-1] of integers (unordered) output: rearrangement of contents of A so that heap property is satisfied. • Natural Idea: insert elements successively. – Time = O(n log n) • We can do this in time O(n)! – make heap out of the left side of the tree – make heap out of the right side of the tree – make adjustments so as to convert entire thing to heap. – what adjustment?

heapify void heap: : heapify(int i){ // Converts everything below i // to a heap if(i >= next) return; heapify(l(i)); heapify(r(i)); trickledown(i); }

In class Exercise • Consider an array of length 15 on which you call heapify. Write down the sequence of calls to heapify and trickledown made by heapify(0). • Solution: h 0, h 1, h 3, h 7, h 8, t 3, h 4, h 9, h 10, h 21, t 4, t 1, h 2, h 5, h 11, h 12, t 5, h 6, h 13, h 14, t 6, t 2, t 0 • Calls to trickledown: 3, 4, 1, 5, 6, 2, 0 • Calls relating to distinct parts of the tree do not interfere with each other, so they can be executed in any order, in particular in the order 6, 5, 4, 3, 2, 1, 0. • So instead of a recursive implementation, heapify could simply call trickledown(i) for i going from 6 to 0. • In general, if j is largest having at least one child, then we should call heapify for j, j-1, . . . , 1, 0 in that order.

Analysis Let T(n) = time required to heapify n elements. T(n) = 2 T((n-1)/2)+clog 2 n+1, assume n=2 k-1 Let T(n) = T(2 k-1) = t(k). T((n-1)/2) = T(2 k-1 -1) = t(k-1) t(k) = 2 t(k-1) + ck = 4 t(k-2) + 2 c(k-1) + ck = 8 t(k-3) + 4 c(k-2) + 2 c(k-1) + ck = 2 kt(0) + 2 k-1 c(1) +. . . + ck t(0) will be another constant. So we can write t(k) = c’ 2 k + c∑i=0. . k 2 k-ii for some constant we will continue to call c.

Analysis contd. t(k) - c’ 2 k = c∑i=0. . k 2 k-ii = c 2 k ∑i=0. . k i/2 i ≤ 2 c 2 k T(n) = t(k) ≤ c’’ 2 k = c’’(n+1) ≤ 2 c’’n = c’’’n • Better than nlog n needed if you do a sequence of insert operations.

Concluding remarks • The same sequence can be stored in many ways: unsorted, fully sorted, heap order sorted. • Heap order sorting makes a compromise between the time required to find the minimum and the time required to insert. • We can often design an algorithm by thinking recursively, but the code in the end might be a simple loop. – “Unroll the recursion” to see what the algorithm is doing. – Recursive heapify and iterative heapify will both take time proportional to n, though the constant of proportionality will be smaller for the iterative version. • We can have max-heap analogous to min-heap. • Heap as we have discussed is also called a priority queue – Removal of elements is as per their value, value is considered as the priority.

Exercises 1. 2. 3. The heap we discussed was binary: each node could have at most two children. Suppose we let each node have d children, and still require that the key at the child must be equal or larger than the key at the parent. Write the functions for adding a key to the heap and removing the minimum element. What is the time required for these operations as a function of n and d? Suppose in a certain problem I make some P insertions and Q removals. Will it help to have a d-ary heap with d > 2? Suppose that in some complete binary tree every vertex j has a key no smaller than the key at its parent, except for one vertex v. Write a function that turns such a tree into a heap. Note that other trees besides v may not satisfy the extended heap property.

More exercises Suppose in a matrix A of numbers, aij ≤ ai+1 j, aij+1, for all ij. Suppose the matrix can also contain ∞, which is larger than any finite number, but which really denotes the absence of a real element. • Show to decide if a number x is present in the matrix. Your algorithm must run in time O(m+n) where m, n are the number of rows and columns in A. – Give good invariants which indicate what exactly your algorithm accomplishes in each iteration.

And one more. . . Suppose you have n balls of unit mass and radius positioned on a straight line at specified positions. They are free to move along the line and their initial velocities are also given. Assume there is no friction and that collisions are elastic, i. e. when balls collide their velocities just get exchanged. As you can see, there will be some number of collisions after which the balls will simply escape to infinity on either side of the line. You are to write a program which takes the initial positions and velocities and prints the position and velocities of the balls at the time just after the last collision. • Calculate the times ti when balls i, i+1 will collide (if possible), assuming there are no other balls. Assume balls are numbered left to right. • Argue that balls i, i+1 collide even when other balls are present if ti is smallest over all i. • Because of the collision you will need to recalculate a few ti only. • So at each step you need to determine the smallest ti, use a min-heap for this. Min-heaps are useful for simulation of systems like this one.