Sorting Data Considerations Average best worst case complexity

Sorting Data • Considerations – Average, best, worst case complexity • For swaps and compares – Is extra memory required? – Difficulty to program? – Stability of equal keys • What is the fastest possible sort using comparisons?

Elementary Sorting Methods Complexity O(n 2), All are Stable • Bubble Sort (N(N-1)/2)comparisons, about N(N-1)/4 swaps) • Selection Sort (N(N-1)/2 comparisons, N-1 swaps) – Minimizes the number of swaps – Worst case equals average case • Insertion Sort (N(N-1)/4 comparisons and copies) – Good for lists that are nearly sorted (O(N) best case)

Bubble Sort Pairwise compares of adjacent elements; swap where necessary pass = 0; swaps = true; while (pass < n && swaps == true) { swaps = false; for (index=0; index<n-pass; index++) { if (sort. Array[index] > sort. Array[index+1]) { swap(sort. Array, index+1); swaps = true; } } pass++; }

Selection Sort Find Minimum n-1 times for (i=0; i<n-1; i++) { minimum = i; for (j=i+1; j<n; j++) { if ( sort. Array[j] < sort. Array[minimum]) minimum = j; } swap(sort. Array, i, minimum); }

Insertion Sort Insert next entry into a growing sorted table for (i=1; i<n; i++) { j = i; save = sort. Array[i]; while (j>0 && save < sort. Array[j-1]) { sort. Array[j] = sort. Array[j-- - 1); } sort. Array[j] = save; }

Proof by Induction • • Select Base Case (n = 1) State the Hypothesis (assume for n=k) State what is to be proved (prove for n=k+1) Example: Base case: For n = 1, 1 = 1 * 2 / 2 = 1 Hypothesis: Assume for n=k, 1 + 2 + … + k = k * (k+1)/2 To Prove: 1 + 2 + … + k+1 = (k+1) * (k+2) /2 1 + 2 + … + k+1 = 1 + 2 + … + k + (k+1) By the hypothesis, this equals k * (k+1)/2 + (k+1) = (k+1)(k/2 + 2/2) = (k+1)(k+2)/2 Therefore by induction, the relationship holds for all positive k >= 1

Recursion Useful for advanced sorts and for divide & conquer algorithms • Relationship to mathematical induction • Key design principals – Relationship between algorithm(n) and algorithm(m) where m<n. – Base Case (How does it stop? ) • When is it useful? What is the overhead? – Relationship between n and m – Tail recursion with a single recursive call – Replace by manually creating stacks • Examples – simple loop, factorial, gcd, binary search, tower of hanoii

Recursion Examples • Factorial: 5! = 5 * 4! • Greatest Common Denominator: gcd(x, y) = gcd(y%x, x) if x<y • Binary Search int bin. Search( array, low, high, value) { if (high – low <= 1) return -1; // Base case middle = (low + high) / 2; if value < array[middle]) bin. Search(array, low, middle-1, value) else if value > array[middle]) bin. Search(array, middle+1, high, value) else return middle; }

Breaking the O(N 2) Barrier Based on either bubble or insertion sort Complexity from O(N 7/6) to O(N 3/2) based on gap selection • Shell Sort while (gap > 0) { for (index=gap; index<n; index++) { temp = sort. Array[index]; compare. Index = index; while(compare. Index>=gap && sort. Array[compare. Index-gap]>=temp) { sort. Array[compare. Index]=sort. Array[compare. Index-gap]; compare. Index -= gap; } sort. Array[compare. Index] = temp; } adjust. Gap( gap ); // different patterns (/=2, (gap-1)/3, (gap+1)/2 }

Shell sort (based on bubble) int index; while (gap > 0) { swaps = true; while (swaps) { swaps = false; for (index = 0; index < gap; index++) { if (sort[index] > sort[index + gap]) { swap(sort, index + gap); swaps = true; } } } adjust. Gap( gap ); }

Merge Sort Always O(N lg. N) but need more memory • Merge Sort void merge. Sort(double[] sort. Array, int low, int high) { int mid = (low+high)/2; if (low == high) return; merge. Sort(sort. Array, low, mid); merge. Sort(sort. Array, mid+1, high); merge(sort. Array, low, mid, high); } • Merge method must: – Allocate an array for copying – Merge two sorted arrays together – Copy back to original array

Merge method void merge(double[] sort, int low, int middle, int high) { int n = high – low + 1, int low. Ptr = low, high. Ptr = middle+1, spot = 0; double work = new double[high – low + 1]; while(low <= middle && high <=top) { if (sort[low. Ptr]<sort[high. Ptr]) else work[spot++] = sort[low. Ptr++]; work[spot++] = sort[high. Ptr++]; while (low. Ptr<=middle) work[spot++] = sort[low. Ptr++]; while (high. Ptr <= top) work[spot++] = sort[high. Ptr++]; low. Ptr = low; for (spot=0; spot<high-low+1; spot++) sort. Array[low. Ptr++] = work. Array[spot]; } }

Analysis of Merge Sort 16 8 8 4 2 2 2 Work at each level totals to 16, lg 16 = 4 levels, complexity = 16 lg 16

Quick Sort O(Nlg. N) average case, in place void quick. Sort(double[] sort. Array, int left, int right) { if (right <= left) return; double pivot = sort. Array[right]; int middle = partition(sort. Array, left, right, pivot); quick. Sort(sort. Array, left, middle-1); quick. Sort(sort. Array, middle+1, right); } • Refinements to avoid O(N 2) worst case and speed up. – Choice of pivit – Combining with insertion sort • Other uses (find the kth biggest number).

Quick Sort Partitioning int partition(double[] sort. Array, int left, int right, int pivot) { int orig. Right = right; left -= 1; for (; ; ) { while(sort. Array[++left] < pivot); while(sort. Array[--right] > pivot); if (left >= right) break; swap(sort. Array, left, right); } swap(sort. Array, left, orig. Right); return left; }

Radix Sort (First Version) 1. 2. 3. 4. Choose the number of buckets (b) Drop next significant part of data into buckets Gather data from buckets back into original array Repeat the above two steps, finishing at the most significant piece of data 5. Notes a. Maximum memory needed for each bucket b. Complexity: O(p * 2 n) where i. ii. p = (Max + b – 1)/b 2 n because dropping and gathering touches each element twice

Radix Sort example Original Data 82 41 63 92 86 Buckets Pass 1 30 41, 21 82, 92 63, 43 94 0 1 2 3 Gather 30 Buckets Pass 2 0 Gather 21 41 1 82 92 21 30 41, 43 3 41 4 43 6 63 5 63 30 76 43 94 86, 76 5 21 2 30 4 21 7 9 43 94 86 76 63 76 82, 86 92, 94 8 9 6 76 8 7 82 86 92 94 Notes: 1. Each pass is a digit of the data (x / 10 (pass – 1)) % 10 2. Two passes because largest number < number of buckets squared 3. Complexity is: O( 2 pn) = O(pn) where p is the number of passes 4. In this case, only two elements are in each bucket, but we couldn’t depend on that in the general case

Refined Radix Sort 1. 2. 3. 4. 5. 6. Create and initialize an array (Counts) of size buckets + 1 Initialize the array to zeroes Store actual bucket sizes into Counts array (starting index = 1) Perform a prefix sum on Counts array to compute starting offsets Use Counts array to drop elements into a second array of numbers Advantages: a. b. 7. 8. Use alternating arrays to avoid the gather operations Only two times the memory is needed Complexity: O(p(2 n + 2 b)) = O(p(n+b)) Notes: a. b. Increased buckets can reduce the number of passes, but prefix sum overhead will limit performance benefits. Radix sort does no comparisons , so O(n lg n) limitation doesn’t apply.

Refined Radix Example Original Data 82 41 63 92 86 21 30 76 43 94 25 Counts 0 1 2 2 2 1 1 2 0 0 0 Count Array Indices 0 1 2 3 4 5 6 7 8 9 10 Prefix Sum 0 1 3 5 7 8 9 11 11 Bucket and Count Array Indices 0 1 2 3 4 5 6 7 8 9 Ending Content 1 3 5 7 8 9 11 11 Bucket Indices 0 1 2 3 4 5 6 7 8 9 Pass 1 Data 30 41 21 82 92 63 43 94 25 86 76 Data offsets 0 1 2 3 4 5 6 7 8 9 10 11 • Dump from original array to alternate array • No gather operation needed • Index to store into count array is one bigger than the bucket. count array index 5 in the above example has a count of 1 because it corresponds to bucket 4.

Optimal Comparison Sort • There are n! possible comparison sorts • All sorts can be modeled with a decision tree • Optimal sort will be completely balanced • Depth of the balanced decision tree is O(lg(n!) Decision Tree compare <= > compare <= compare > <= >

Prove optimal sort <= O(n lg n) • Optimal comparison sort <= O(n lg n) lg (n!) = lg(n) + lg (n-1) + lg (n -2) + … + lg(1) < lg n + … + lg n = n lg n = O(n lg n) • Optimal comparison sort >= O(n lg n) lg (n!) = lg(n) + … + lg(n/2+1) + lg(n/2) + … + lg(n/4+1) + lg(n/4) + … + lg(n/8+1) + … > n/2 lg(n/2) + n/4 lg(n/4) + n/8 lg(n/8) + … = n/2 lg(n) – n/2 lg 2 + n/4 lg(n) – n/4 lg 4 + n/8 lg n – n/8 lg 8 ≈ n lg (n) – ½ n – 2/4 n – 3/8 n – 4/16 n - … = n lg (n) – n (1/2 + 2/4 + 3/8 + 4/16 + … = n lg n – 2 n But O(n lg n – 2 n) = O(n lg n) • Therefore optimal sort = O(n lg n) • The series is well known: ½ + 2/4 + 3/8 + … = ∑ n/2 n ≈ 2 • Proof: S(2 n) – S(n) = 1 + 6/8 + 8/16 + 10/32 – ½ - 2/4 – 3/8 -…= 1 + ½ + ¼ + 1/8 +…= 2