Sorting Objectives n Become familiar with the following

Sorting

Objectives n Become familiar with the following sorting methods: n n n n Insertion Sort Shell Sort Selection Sort Bubble Sort Quick Sort Heap Sort Merge Sort … more

Introduction n n One of the most common applications in computer science is sorting, the process through which data are arranged according to their values. If data were not ordered in some way, we would spend an incredible amount of time trying to find the correct information.

Introduction n To appreciate this, imagine trying to find someone’s number in the telephone book if the names were not sorted in some way!

General Sorting Concepts n n n Sorts are generally classified as either internal or external. An internal sort is a sort in which all of the data is held in primary memory during the sorting process. An external sort uses primary memory for the data currently being sorted and secondary storage for any data that will not fit in primary memory.

General Sorting Concepts n n n For example, a file of 20, 000 records may be sorted using an array that holds only 1000 records. Therefore only 1000 records are in primary memory at any given time. The other 19, 000 records are stored in secondary storage.

Sort Order n n n Data may be sorted in either ascending or descending order. The sort order identifies the sequence of sorted data, ascending or descending. If the order of the sort is not specified, it is assumed to be ascending.

Sort Stability n n Sort stability is an attribute of a sort indicating that data elements with equal keys maintain their relative input order in the output. Consider the following example.

Sort Stability n n Note the unsorted data in (a). If we use a stable sort, items with equal keys are guaranteed to be sorted in the same order as they appear in the unsorted data.

Sort Stability n If, however, we use an unstable sort, data with equal keys can be sorted in any order (i. e. , not necessarily in the same order as they appeared in the unsorted data).

Sort Efficiency n n Sort efficiency is a measure of the relative efficiency of a sort. It is usually an estimate of the number of comparisons and moves required to order an unordered list.

Passes n n During the sorting process, the data is traversed many times. Each traversal of the data is referred to as a sort pass. Depending on the algorithm, the sort pass may traverse the whole list or just a section of the list. The sort pass may also include the placement of one or more elements into the sorted list.

Types of Sorts n We now discuss several sorting methods: n n n n Insertion Sort Shell Sort Selection Sort Bubble Sort Quick Sort Heap Sort Merge Sort

Insertion Sort n In each pass of an insertion sort, one or more pieces of data are inserted into their correct location in an ordered list (just as a card player picks up cards and places them in his hand in order).

Insertion Sort n In the insertion sort, the list is divided into 2 parts: n n Sorted Unsorted In each pass, the first element of the unsorted sublist is transferred to the sorted sublist by inserting it at the appropriate place. If we have a list of n elements, it will take, at most, n-1, passes to sort the data.

Insertion Sort n n We can visualize this type of sort with the above figure. The first part of the list is the sorted portion which is separated by a “conceptual” wall from the unsorted portion of the list.

Insertion Sort Here wefirst start with anfashion unsorted list. We leave the first data nd rd On We our continue pass, in this we look at until the 2 the element entire list of has the been list On our next pass, we look at the 3 element of the list (the nd element alone and will start with the 2 element of the list st first sorted. element of the unsorted list) and place it in our sorted 1 element of the unsorted list) and place it in the sorted list st element (the 1 of the unsorted list). list in order.

Insertion Sort Example In our pass, we compare the 2 values. Because Now we look atout the next 2 order, values. Again, they are out of they These Since we they 2 first values have are swapped are of out order, ofthose wevalues, swap sofirst we them we swap need and them to then compare and continue look at arewith out of swap them. order sothe weorder, swap them. on the previous next 2 values, 2 we data to etc. values. make sure that they are still in order.

Shell Sort n n Named after its creator, Donald Shell, the shell sort is an improved version of the insertion sort. In the shell sort, a list of N elements is divided into K segments where K is known as the increment. What this means is that instead of comparing adjacent values, we will compare values that are a distance K apart. We will shrink K as we run through our algorithm.

Shell Sort Just as in the straight insertion sort, we compare 2 values and swap them if they are out of order. However, in the shell sort we compare values that are a distance K apart. Once we have completed going through the elements in our list with K=5, we decrease K and continue the process.

Shell Sort Here we have reduced K to 2. Just as in the insertion sort, if we swap 2 values, we have to go back and compare the previous 2 values to make sure they are still in order.

Shell Sort All shell sorts will terminate by running an insertion sort (i. e. , K=1). However, using the larger values of K first has helped to sort our list so that the straight insertion sort will run faster.

Shell Sort n n There are many schools of thought on what the increment should be in the shell sort. Also note that just because an increment is optimal on one list, it might not be optimal for another list.

Insertion Sort vs. Shell Sort n Comparing the Big-O notation (for the average case) we find that: n n n Insertion: O(n 2) Shell: O(n 1. 25) //empirically determined Although this doesn’t seem like much of a gain, it makes a big difference as n gets large. Note that in the worst case, the Shell sort has an efficiency of O(n 2). However, using a special incrementing technique, this worst case can be reduced to O(n 1. 5)

Shell Sort h : = 1; repeat h : = 3*h+1 until h>n; repeat h : = h div 3; for i : = h+1 to n do begin v : = a[i]; j: = i; while j>h & a[j-h]>v do begin a[j] : = a[j-h]; j : = j - h; end; a[j] : = v; end; until h = 1; • • Shellsort is a simple extension of insertion sort, which gains speeds by allowing exchange of elements that are far apart. Idea: rearrange list into h-sorted (for any sequence of values of h that ends in 1. ) Shellsort never does more than n 1. 5 comparisons (for the h = 1, 4, 13, 40, . . . ). The analysis of this algorithm is hard. Two conjectures of the complexity are n(log n)2 and n 1. 25

Insertion Sort vs. Shell Sort (Average Case)

Selection Sort n n n Imagine some data that you can examine all at once. To sort it, you could select the smallest element and put it in its place, select the next smallest and put it in its place, etc. For a card player, this process is analogous to looking at an entire hand of cards and ordering them by selecting cards one at a time and placing them in their proper order.

Selection Sort n n n The selection sort follows this idea. Given a list of data to be sorted, we simply select the smallest item and place it in a sorted list. We then repeat these steps until the list is sorted.

Selection Sort n n n In the selection sort, the list at any moment is divided into 2 sublists, sorted and unsorted, separated by a “conceptual” wall. We select the smallest element from the unsorted sublist and exchange it with the element at the beginning of the unsorted data. After each selection and exchange, the wall between the 2 sublists moves – increasing the number of sorted elements and decreasing the number of unsorted elements.

Selection Sort We an unsorted We search this list element. for the We Again, wewith search the unsorted Thisstart process continues untillist. thelistfor is the fullysmallest sorted. smallest element. We then element exchange thewith smallest element (8) then exchange the smallest (23) the first element with first element in the unsorted list (23). in thethe unsorted list (78).

Bubble Sort n n In the bubble sort, the list at any moment is divided into 2 sublists, sorted and unsorted. The smallest element is “bubbled” from the unsorted sublist to the sorted sublist.

1 Pass of the Bubble Sort 23 45 8 56 32 8 78 23 8 78 8 45 32 56 We start with 32 and compare it with 56. Because 32 is less than 56, we swap the two and step down one element. We then compare 32 and 8. Because 32 is not less than 8, we do not swap these elements. We step down one element and compare 45 and 8. They are out of sequence, so we swap them and step down again. We step down again and compare 8 with 78. These two elements are swapped. Finally, 8 is compared with 23 and swapped. We then continue this process back with 56 …

Quick Sort n n n In the bubble sort, consecutive items are compared and possibly exchanged on each pass through the list. This means that many exchanges may be needed to move an element to its correct position. Quick sort is more efficient than bubble sort because a typical exchange involves elements that are far apart, so fewer exchanges are required to correctly position an element.

Idea of Quick Sort 1) Select: pick an element 2) Divide: rearrange elements so that x goes to its final position E 3) Recurse and Conquer: recursively sort

Quick Sort n Each iteration of the quick sort selects an element, known as the pivot, and divides the list into 3 groups: n n n Elements whose keys are less than (or equal to) the pivot’s key. The pivot element Elements whose keys are greater than (or equal to) the pivot’s key.

Quick Sort n n The sorting then continues by quick sorting the left partition followed by quick sorting the right partition. The basic algorithm is as follows:

Quick Sort 1) 2) Partitioning Step: Take an element in the unsorted array and determine its final location in the sorted array. This occurs when all values to the left of the element in the array are less than (or equal to) the element, and all values to the right of the element are greater than (or equal to) the element. We now have 1 element in its proper location and two unsorted subarrays. Recursive Step: Perform step 1 on each unsorted subarray.

Quick Sort n n n Each time step 1 is performed on a subarray, another element is placed in its final location of the sorted array, and two unsorted subarrays are created. When a subarray consists of one element, that subarray is sorted. Therefore that element is in its final location.

Quick Sort n n n There are several partitioning strategies used in practice (i. e. , several “versions” of quick sort), but the one we are about to describe is known to work well. For simplicity we will choose the last element to be the pivot element. We could also chose a different pivot element and swap it with the last element in the array.

Quick Sort n 1 Below is the array we would like to sort: 4 8 9 0 11 5 10 7 6

Quick Sort n 1 The index left starts at the first element and right starts at the next-to-last element. 4 left n 8 9 0 11 5 10 7 6 right We want to move all the elements smaller than the pivot to the left part of the array and all the elements larger than the pivot to the right part.

Quick Sort n 1 We move left to the right, skipping over elements that are smaller than the pivot. 4 8 left 9 0 11 5 10 7 right 6

Quick Sort n 1 We then move right to the left, skipping over elements that are greater than the pivot. 4 8 left n 9 0 11 5 10 7 6 right When left and right have stopped, left is on an element greater than (or equal to) the pivot and right is on an element smaller than (or equal to) the pivot.

Quick Sort n If left is to the left of right (or if left = right), those elements are swapped. 1 4 8 left 9 0 11 5 10 right 7 6 1 4 5 left 9 0 11 8 10 right 7 6

Quick Sort n n The effect is to push a large element to the right and a small element to the left. We then repeat the process until left and right cross.

Quick Sort 1 4 5 left 9 0 11 8 10 right 7 6 1 4 5 9 0 11 left right 8 10 7 6 1 4 5 0 9 11 left right 8 10 7 6

Quick Sort 1 4 5 0 9 11 left right 8 10 7 6 1 4 5 0 9 11 right left 8 10 7 6

Quick Sort 1 4 n n 1 5 0 9 11 right left 8 10 7 6 At this point, left and right have crossed so no swap is performed. The final part of the partitioning is to swap the pivot element with left. 4 5 0 6 11 right left 8 10 7 9

Quick Sort n n 1 Note that all elements to the left of the pivot are less than (or equal to) the pivot and all elements to the right of the pivot are greater than (or equal to) the pivot. Hence, the pivot element has been placed in its final sorted position. 4 5 0 6 11 right left 8 10 7 9

Quick Sort n We now repeat the process using the sub-arrays to the left and right of the pivot. 1 4 5 0 6 11 8 10 7 9

Analysis of quicksort—best case n n Suppose each partition operation divides the array almost exactly in half Then the depth of the recursion in log 2 n n n Because that’s how many times we can halve n However, there are many recursions at each level! n n How can we figure this out? We note that n n Each partition is linear over its subarray All the partitions at one level cover the array

Partitioning at various levels

Best case II n We cut the array size in half each time So the depth of the recursion in log 2 n At each level of the recursion, all the partitions at that level do work that is linear in n O(log 2 n) * O(n) = O(n log 2 n) n What about the worst case? n n n

Worst case n In the worst case, partitioning always divides the size n array into these three parts: n n n A length one part, containing the pivot itself A length zero part, and A length n-1 part, containing everything else We don’t recur on the zero-length part Recurring on the length n-1 part requires (in the worst case) recurring to depth n-1

Worst case partitioning

Worst case for quicksort n n n In the worst case, recursion may be O(n) levels deep (for an array of size n). But the partitioning work done at each level is still O(n) * O(n) = O(n 2) So worst case for Quicksort is O(n 2) When does this happen? n When the array is sorted to begin with!

Typical case for quicksort n n n If the array is sorted to begin with, Quicksort is terrible: O(n 2) It is possible to construct other bad cases However, Quicksort is usually O(n log 2 n) The constants are so good that Quicksort is generally the fastest algorithm known Most real-world sorting is done by Quicksort

Average Case Analysis

Average Case Analysis n n n n. T(n) = (n-1)T(n-1) + 2 n n. T(n) = (n+1)T(n-1) + 2 n T(n)/(n+1) = T(n-1)/n + 2/(n+1) T(n)/(n+1) = T(n-2)/(n-1) + 2/n + 2/(n+1) T(n)/(n+1) = T(n-3)/(n-2) + 2/(n-1) + 2/n + 2/(n+1). . . T(n)/(n+1) = T(n-k)/(n-k+1) + 2/(n-k+2) +. . . + 2/(n-1) + 2/n + 2/(n+1) T(n)/(n+1) = T(n-k)/(n-k+1) + 2(1/(n-k+2) +. . . + 1/(n-1) + 1/n + 1/(n+1))

Average Case Analysis n n n Let k = n-1 T(n)/(n+1) = T(1)/2 + 2(1/3 +. . . + 1/(n-1) + 1/n + 1/(n+1)) T(1) = a (a is a constant) Note T(n)/(n+1) <= a + O(log n) = O(log n) T(n) <= O(n log n)

Tweaking Quicksort n n Almost anything you can try to “improve” Quicksort will actually slow it down One good tweak is to switch to a different sorting method when the subarrays get small (say, 10 or 12) n n Quicksort has too much overhead for small array sizes For large arrays, it might be a good idea to check beforehand if the array is already sorted n But there is a better tweak than this

Randomized Quick-Sort n n n Select the pivot as a random element of the sequence. The expected running time of randomized quick-sort on a sequence of size n is O(n log n). The time spent at a level of the quick-sort tree is O(n) We show that the expected height of the quick-sort tree is O(log n) good vs. bad pivots • The probability of a good pivot is 1/2, thus we expect k/2 good pivots out of k pivots • After a good pivot the size of each child sequence is at most 3/4 the size of the parent sequence • After h pivots, we expect (3/4)h/2 n elements • The expected height h of the quick-sort tree is at most: 2 log 4/3 n

Median of three n n Obviously, it doesn’t make sense to sort the array in order to find the median to use as a pivot Instead, compare just three elements of our (sub)array—the first, the last, and the middle n n n Take the median (middle value) of these three pivot It’s possible (but not easy) to construct cases which will make this technique O(n 2) Suppose we rearrange (sort) these three numbers so that the smallest is in the first position, the largest in the last position, and the other in the middle n This lets us simplify and speed up the partition as

Final comments n n n Weiss’s code shows some additional optimizations on pp. 246 -247. Weiss chooses to stop both searches on equality to pivot. This design decision is debatable. Quicksort is the fastest known general sorting algorithm, on average. For optimum speed, the pivot must be chosen carefully. “Median of three” is a good technique for choosing the pivot. There will be some cases where Quicksort runs in O(n 2) time.

Quick Sort n n n A couple of notes about quick sort: There are more optimal ways to choose the pivot value (such as the median-ofthree method). Also, when the subarrays get small, it becomes more efficient to use the insertion sort as opposed to continued use of quick sort.

Bubble Sort vs. Quick Sort n If we calculate the Big-O notation we find that (in the average case): n n Bubble Sort: O(n 2) Quick Sort: O(nlog 2 n) (quick-sort runs in time O(n 2) in the worst case)

Merge Sort n Idea: n n Take the array you would like to sort and divide it in half to create 2 unsorted subarrays. Next, sort each of the 2 subarrays. Finally, merge the 2 sorted subarrays into 1 sorted array. Efficiency: O(nlog 2 n)

Merge Sort

Merge Sort n n Although the merge step produces a sorted array, we have overlooked a very important step. How did we sort the 2 halves before performing the merge step? We used merge sort!

Merge Sort n n By continually calling the merge sort algorithm, we eventually get a subarray of size 1. Since an array with only 1 element is clearly sorted, we can back out and merge 2 arrays of size 1.

Merge Sort

Merge Sort n The basic merging algorithm consists of: n n n 2 input arrays (array. A and array. B) An ouput array (array. C) 3 position holders (index. A, index. B, index. C), which are initially set to the beginning of their respective arrays.

Merge Sort n n The smaller of array. A[index. A] and array. B[index. B] is copied into array. C[index. C] and the appropriate position holders are advanced. When either input list is exhausted, the remainder of the other list is copied into array. C.

Merge Sort array. A 1 13 24 26 15 27 38 index. A array. B 2 index. B array. C index. C We compare array. A[index. A] with array. B[index. B]. Whichever value is smaller is placed into array. C[index. C]. 1 < 2 so we insert array. A[index. A] into array. C[index. C]

Merge Sort array. A 1 13 24 26 index. A 2 array. B 15 index. B array. C 1 index. C 27 38 2 < 13 so we insert array. B[index. B] into array. C[index. C]

Merge Sort array. A 1 13 24 26 index. A array. B 2 15 27 38 index. B array. C 1 2 index. C 13 < 15 so we insert array. A[index. A] into array. C[index. C]

Merge Sort array. A 1 13 24 26 index. A array. B 2 15 27 38 index. B array. C 1 2 13 index. C 15 < 24 so we insert array. B[index. B] into array. C[index. C]

Merge Sort array. A 1 13 24 26 index. A array. B 2 15 27 38 index. B array. C 1 2 13 15 index. C 24 < 27 so we insert array. A[index. A] into array. C[index. C]

Merge Sort array. A 1 13 24 26 index. A array. B 2 15 27 38 26 < 27 so we insert array. A[index. A] into array. C[index. C] index. B array. C 1 2 13 15 24 index. C

Merge Sort array. A 1 13 24 26 array. B 2 15 27 Since we have exhausted one of the arrays, array. A, we simply copy the remaining items from the other array, array. B, into array. C 38 index. B array. C 1 2 13 15 24 26 index. C

Merge Sort array. A 1 13 24 26 array. B 2 15 27 38 array. C 1 2 13 15 24 26 27 38

Heapsort n Given Build. Heap(), an in-place sorting algorithm is easily constructed: n n Maximum element is at A[1] Discard by swapping with element at A[n] n n Decrement heap_size[A] A[n] now contains correct value Restore heap property at A[1] by calling Heapify() Repeat, always swapping A[1] for A[heap_size(A)]

$Heapsort(A) { Build. Heap(A); for (i = length(A) downto 2) { Swap(A[1], A[i]); heap_size(A)$

Heapsort(A) { Build. Heap(A); for (i = length(A) downto 2) { Swap(A[1], A[i]); heap_size(A) -= 1; Heapify(A, 1); } }

Example

Analyzing Heapsort n n n The call to Build. Heap() takes O(n) time Each of the n - 1 calls to Heapify() takes O(lg n) time Thus the total time taken by Heap. Sort() = O(n) + (n - 1) O(lg n) = O(n) + O(n lg n) = O(n lg n)

Efficiency Summary Sort Insertion Shell Selection Bubble Quick Heap Merge Worst Case O(n 2) O(n 1. 5) O(n 2) O(nlog 2 n) Average Case O(n 2) O(n 1. 25) O(n 2) O(nlog 2 n)

More Sorting n n radix sort bucket sort in-place sorting how fast can we sort?

Radix Sort n n Unlike other sorting methods, radix sort considers the structure of the keys Assume keys are represented in a base M number system (M = radix), i. e. , if M = 2, the keys are represented in binary Sorting is done by comparing bits in the same position Extension to keys that are alphanumeric strings

Radix Exchange Sort Examine bits from left to right: 1. Sort array with respect to leftmost bit 2. Partition array 3. Recursion recursively sort top subarray, ignoring leftmost bit recursively sort bottom subarray, ignoring leftmost bit Time to sort n b-bit numbers: O(b n)

Radix Exchange Sort How do we do the sort from the previous page? Same idea as partition in Quicksort. repeat scan top-down to find key starting with 1; scan bottom-up to find key starting with 0; exchange keys; until scan indices cross;

Radix Exchange Sort

Radix Exchange Sort vs. Quicksort Similarities both partition array both recursively sort sub-arrays Differences Method of partitioning radix exchange divides array based on greater than or less than 2 b-1 quicksort partitions based on greater than or less than some element of the array Time complexity Radix exchange O (bn) Quicksort average case O (n log n)

Straight Radix Sort Examines bits from right to left for k : = 0 to b-1 sort the array in a stable way, looking only at bit k Note order of these bits after sort.

Recall “sort in a stable way”!!! In a stable sort, the initial relative order of equal keys is unchanged. For example, observe the first step of the sort from the previous page: Note that the relative order of those keys ending with 0 is unchanged, and the same is true for elements ending in 1

The Algorithm is Correct (right? ) We show that any two keys are in the correct relative order at the end of the algorithm Given two keys, let k be the leftmost bit-position where they differ At step k the two keys are put in the correct relative order Because of stability, the successive steps do not change the relative order of the two keys

For Instance, Consider a sort on an array with these two keys:

Radix sorting can be applied to decimal numbers

Straight Radix Sort Time Complexity for k = 0 to b - 1 sort the array in a stable way, looking only at bit k Suppose we can perform the stable sort above in O(n) time. The total time complexity would be O(bn) As you might have guessed, we can perform a stable sort based on the keys’ kth digit in O(n) time. The method, you ask? Why it’s Bucket Sort, of course.

Bucket Sort BASICS: n numbers Each number {1, 2, 3, . . . m} Stable Time: O (n + m) For example, m = 3 and our array is: (note that there are two “ 2”s and two “ 1”s) First, we create M “buckets”

Bucket Sort Each element of the array is put in one of the m “buckets”

Bucket Sort Now, pull the elements from the buckets into the array At last, the sorted array (sorted in a stable way):

Comparison Sorting n n Elements are rearranged by comparing values of keys Comparison of keys is interleaved with rearrangements of the order of various pairs of keys as the algorithm progressively moves keys into final sorted order. Comparisons performed at given stage will depend on outcomes of previous comparisons and previous rearrangements of keys. Sorting an array of 3 elements, a, b and c produces 6 or 3! distinct orderings

Decision Tree Consider a decision tree that sorts n items. The next slide shows a decision tree for 3 items: a, b, c. n n Ovals represent comparisons. If < we traverse down a left branch, otherwise a right branch.

Decision Tree on 3 Elements true a< b false b<c true false a, b, c a<c true a, c, b true b<c a<c false c, a, b true b, a, c false c, b, a false b, c, a

Levels in Decision Trees n n n Number the levels starting at zero Level i can represent 2 i arrangements Our decision tree can represent 8 = 23 arrangements If we have 4 items, then 4! = 24 elements, but 4 levels gives at most 24 = 16, so we need 5 levels since 25 = 32 24 If we have k arrangements to represent then we need a decision tree with at least lg k levels.

Size of Decision Tree When dealing with n elements we have n! possible arrangements and need a decision tree with at least lg n! levels.

Lower Bound Result n! = n(n-1)(n-2)… 2·1 from this we get lg(n!) = lg(n) + lg(n-1) + lg(n-2) +…+ lg 2 + lg 1 this means lg(n!) < nlg(n) and the two results together show O(lg(n!)) = O(nlg(n)) No general sorting algorithm can run faster than O(n lg n).