Sorting Algorithms What is the Sorting Algorithm n

Sorting Algorithms

What is the Sorting Algorithm? n Definition n Given a set of records, the output is to the non-decreasing order (numerical order or lexicographical order) of records. n The output is a permutation of the input. n https: //www. toptal. com/developers/sorting-algorithms 2

Why is Sorting Important? n Sorting has been commonly used as the pre-processed method for searching and matching. n n n Sorting is also used as the basic solution for many other complex problem. In most organization, more than 25% of computing time is spent on sorting. No best algorithm for any situation: initial ordering and size of list. n n We need to know several techniques. The analysis of lower bound is good for understanding basic skill for algorithm analysis. 3

Categories of Sorting Algorithms n Comparison sort vs. non-comparison sort n n Comparative sorting algorithm determines the order of two element through a comparison operator. Comparison sorting algorithms: n n Non-comparison sorting algorithms: n n Selection sort, Bubble sort, Insertion sort, Quick sort, … Radix sort, Bucket sort, Counting sort Note: non-comparison sort is also called a linear sorting method. 4

Categories of Sorting Algorithms n Internal sort vs. external sort n Internal sorting technique is for the case where the list is small enough to sort entirely in main memory n n Minimizing the number of comparisons External sorting technique is used when the list is too big to fit into main memory, (e. g. , disk and SSD). n Minimizing the number of I/O accesses 5

Stability of Sorting Algorithms n Definition n Stable sorting algorithms maintain the relative order of records with equal keys. n The initial order of records with equal keys does not changed. Stable algorithm unstable algorithm 6

What is Selection Sort? n 0 th 10 21 5 8 1 12 1 st 1 21 5 8 10 12 2 nd 1 5 21 8 10 12 3 rd 1 5 8 21 10 12 4 th 1 5 8 10 21 12 7

Implementation of Selection Sort n Implementation void Selection. Sort(Data* list, int n) { int min, temp; for (int i = 0; i < n - 1; i++) { min = i; for (int j = i + 1; j < n; j++) { // Find an index with the minimum element. if (list[j] < list[min]) min = j; } // Exchange the minimum element and the i-th element. SWAP(list[i], list[min], temp); } } 8

Exercise: Selection Sort n Animation: sorting 3, 4, 5, 4, 9, 1, 2, 6, 7 n n Draw the step-by-step procedure of selection sort. https: //visualgo. net/en/sorting Initial list n After sorting the first element Is it stable? 9

Analysis of Selection Sort n 10

What is Bubble Sort? n 10 21 5 8 1 12 10 Check if swap 10 and 21. 10 5 21 8 1 5 8 1 21 5 8 1 12 Check if swap 21 and 5. 12 10 Check if swap 21 and 8. 10 21 5 8 21 1 12 Check if swap 21 and 1. 12 10 Check if swap 21 and 12. 5 8 1 12 21 The 0 -th iteration is ended. 11

Implementation of Bubble Sort n Implementation void Bubble. Sort(Data* list, int n) { int temp; for (int i = n - 1; i > 0; i--) { for (int j = 0; j < i; j++) { // Compare each pair of adjacent items. if (list[j] > list[j + 1]) { // Swap if they are in the wrong order. SWAP(list[j], list[j + 1], temp); } } 12

Exercise: Bubble Sort n Animation: sorting 3, 4, 5, 4, 9, 1, 2, 6, 7 n n Draw the step-by-step procedure of bubble sort. https: //visualgo. net/en/sorting Initial list n After sorting the last element Is it stable? 13

Analysis of Bubble Sort n 14

What is Insertion Sort? n 1 th 10 21 5 8 1 12 2 st 10 21 5 8 1 12 10 5 21 8 1 12 3 nd 5 10 21 8 1 12 5 8 10 21 1 12 4 rd 5 8 10 21 1 12 1 5 8 10 21 12 5 th 1 5 8 10 21 12 1 5 8 10 12 21 15

Implementation of Insertion Sort n Implementation void Insertion. Sort(Data* list, int n) { int j, key; for (int i = 1; i < n; i++) { key = list[i]; // Choose the i-th element. for (j = i - 1; j >= 0; j--) { // If the j-th element is greater than key, // move to the next position. if (key < list[j]) list[j + 1] = list[j]; else break; } // Otherwise, move the key to the (j+1)-th element. list[j + 1] = key; } } 16

Exercise: Insertion Sort n Animation: sorting 3, 4, 5, 4, 9, 1, 2, 6, 7 n n Draw the step-by-step procedure of insertion sort. https: //visualgo. net/en/sorting Initial list n After sorting four elements Is it stable? 17

Analysis of Insertion Sort n 18

What is Shell Sort? n Description n Invented by Donald Shell in 1959 Start by sorting pairs of elements far apart from each other, then reduce the gap between elements to be compared. Generalization of insertion sort n Sort every k-th elements by considering k interleaved lists. n Reduce the size of k by 1. 9 7 5 20 4 2 22 1 3 15 18 After sorting every 5 element 2 7 1 3 4 9 22 5 20 15 18 After sorting every 3 element 2 4 1 3 5 9 15 7 20 22 18 After sorting every element 1 2 3 4 5 7 19 9 15 18 20 22

Example of Shell Sort n When k = 5, sorting every k-th element 9 7 5 20 4 9 2 22 1 3 2 22 5 1 20 3 4 2 sorting 15 9 18 7 22 1 5 3 20 4 7 18 18 7 2 15 1 3 4 15 9 22 20 5 20 15 18

Example of Shell Sort n When k = 5 and k = 1, sorting every k-th element 2 7 1 2 3 4 9 22 3 7 1 5 18 18 20 3 15 5 1 15 15 9 4 20 22 4 2 5 sorting 22 7 9 18 20 2 4 1 3 5 9 15 7 20 22 18 1 2 3 4 5 7 9 15 18 20 22 21 sorting

Implementation of Shell Sort n This is similar to insertion sort, but considers the gap between elements. n When the gap is one, it is same as insertion sort. void Inc. Insertion. Sort(int* list, int first, int last, int k) { int i, j, key; // Sort every k-th element for (i = first + k; i < last; i = i + k) { key = list[i]; // Choose the i-th element. // If the j-th element is greater than key, // move to the next position with k interval. for (j = i - k; j >= first && key < list[j]; j = j - k) list[j + k] = list[j]; list[j + k] = key; } } 22

Implementation of Shell Sort n Implementation n It is non-trivial to answer which gap sequence is better. void Shell. Sort(Data* list, int n) { // Sort every k-th element by reducing from n/2 to 1. for (int k = n / 2; k > 0; k /= 2) { // Choose the k-th element as the odd. if ((k % 2) == 0) k++; for (int i = 0; i < k; i++) Inc. Insertion. Sort(list, i, n, k); } } 23

Exercise: Shell Sort n Animation: sorting 3, 4, 5, 4, 9, 1, 2, 6, 7 n Draw the step-by-step procedure of shell sort. n n Every k = 5, 3, and 1 https: //www. cs. usfca. edu/~galles/visualization/Comparison. Sort. html Initial list n After sorting every 5 -th elements Is it stable? 24

Analysis of Shell Sort n 25

Divide & Conquer (D&C) Paradigm n Definition n n Breaking down a problem into two or more subproblems of the same or related type. The solutions to the subproblems are combined to be a solution to the original problem. 7 Problem split subproblem solve subsolution 12 merge 3 1 6 split 7 solve subsolution 8 12 5 split 8 3 1 6 solve 3 7 merge 1 8 26 3 5 4 solve 12 1 merge Solution 3 4 4 5 6 merge 5 6 7 8 12

What is Quick Sort? n Description n Invented by Tony Hoare in 1959 Based on the divide and conquer (D&C) paradigm. Overall procedure n n n Pivot selection: Pick an element, called a pivot, from the list. Partitioning: reorder the list with the pivot. n The elements less than the pivot come before the pivot. n The element greater than the pivot come after the pivot. Recursively apply the above steps to the sublists. 7 4 8 1 Pivot 1 12 11 3 9 8 9 Partitioning 4 3 7 12 27 11

Example of Quick Sort n Overall procedure n n For each list, select a pivot as the left-most element. Partition the list into two sublists. 7 4 8 1 12 11 3 9 1 4 3 7 12 11 8 9 1 4 3 7 9 11 8 12 1 3 4 7 8 9 11 12 28

Quick Sort: Partitioning n Select a pivot from the list. n n Use two variables: low and high n n In general, the left-most element is chosen as the pivot. low: if L[low] is less than a pivot, move to the right element. high: if L[high] is greater than a pivot, move to the left element. Swap two elements L[low] and L[high]. If low and high are crossed, stop partitioning. n Swap two elements L[left] and L[high]. 29

Quick Sort: Partitioning n Assume that the left-most element is the pivot. n n left: an starting index for a sublist that is less than a pivot right: an ending index for a sublist that is greater than a pivot Select a pivot Move low until pivot < L[low]. 7 4 left low 7 4 left Move high until pivot >= L[high]. Swap L[low] and L[high]. 7 left 1 12 11 3 4 8 1 12 11 3 8 3 low 30 9 right, high 1 12 11 low 4 9 right, high low left 7 8 1 12 11 3 9 high right 8 9 high right

Quick Sort: Partitioning n Assume that the left-most element is the pivot. n n left: an starting index for a sublist that is less than a pivot right: an ending index for a sublist that is greater than a pivot Move low until pivot < L[low]. 7 4 3 1 7 4 3 left Swap L[left] and L[high]. 1 11 low left Move high until pivot >= L[high]. 12 4 3 left All elements in the left sublist are less than the pivot. 31 1 12 high low 7 12 high low 8 9 high right 11 8 9 right All elements in the right sublist are greater than the pivot.

Quick Sort: Partitioning n Partitioning one list into two sublists n n All elements in the left sublist are less than the pivot. All elements in the right sublist are greater than the pivot. int Partition(Data* list, int left, int right) { int pivot = list[left], temp; int low = left + 1, high = right; while (1) { while (low < right && low++; // Move while (high > left && high--; // Move list[low] < pivot) low until pivot < L[low] list[high] >= pivot) high until pivot >= L[low] if (low < high) // Swap list[low] and list[high]. SWAP(list[low], list[high], temp); else break; } SWAP(list[left], list[high], temp); return high; // return the pivot position. } 32

Implementation of Quick Sort n Overall procedure n n n Pivot selection: Pick an element, called a pivot, from the list. Partitioning: reorder the list with the pivot. Recursively apply the above steps to the sublists. void Quick. Sort(Data* list, int left, int right) { if (left < right) { // The mid refers to the pivot position. int mid = Partition(list, left, right); // All elements are less than the pivot. Quick. Sort(list, left, mid - 1); // All elements are greater than the pivot. Quick. Sort(list, mid + 1, right); } } 33

Exercise: Quick Sort n Animation: sorting 3, 4, 5, 4, 9, 1, 2, 6, 7 n n Draw the step-by-step procedure of insertion sort. https: //visualgo. net/en/sorting Initial list n After the first partitioning Q: Is it stable? No… 34

Analysis of Quick Sort n 35

Analysis of Quick Sort n The worse case occurs if the pivot is selected as an extremely skewed case. n n The time complexity of quick sort mainly depends on pivot selection. How to choose a good pivot in quick sort? 36

What is Merge Sort? n 37

What is Merge Sort? n Partitioning and merging in a recursive manner 7 7 12 12 8 8 3 1 6 3 1 5 6 3 5 4 Partitioning 7 12 8 3 1 6 5 4 7 12 3 8 1 6 4 5 Merging 3 7 1 8 3 12 4 1 5 6 4 7 38 5 8 6 12

Merge Sort: Merging n How to merge two sublists into one list? n Compare two elements in L 1 and L 2 in sequence. n If the element in L 1 is less than or equal to that in L 2, move to the next position in L 1. n If the element in L 1 is greater than that in L 2, move to the next position in L 2. First sublist 3 7 8 12 Second sublist 1 4 5 6 Because 3 > 1, move the position of the second sublist. Copy 1 in L 2 to the sorted list. Sorted list 1 39

Merge Sort: Merging n How to merge two sublists into one list? n Compare two elements in L 1 and L 2 in sequence. n If the element in L 1 is less than or equal to that in L 2, move to the next position in L 1. n If the element in L 1 is greater than that in L 2, move to the next position in L 2. First sublist 3 7 8 12 Second sublist 1 4 5 6 Because 3 < 4, move the position of the first sublist. Copy 3 in L 1 to the sorted list. Sorted list 1 3 40

Merge Sort: Merging n How to merge two sublists into one list? n Compare two elements in L 1 and L 2 in sequence. n If the element in L 1 is less than or equal to that in L 2, move to the next position in L 1. n If the element in L 1 is greater than that in L 2, move to the next position in L 2. First sublist 3 7 8 12 Second sublist 1 4 5 6 Because 7 > 4, move the position of the second sublist. Copy 4 in L 2 to the sorted list. Sorted list 1 3 4 41

Merge Sort: Merging n How to merge two sublists into one list? n Compare two elements in L 1 and L 2 in sequence. n If the element in L 1 is less than or equal to that in L 2, move to the next position in L 1. n If the element in L 1 is greater than that in L 2, move to the next position in L 2. First sublist 3 7 8 12 Second sublist 1 4 5 6 Because 7 > 5 and 7 > 6, move the position of the second sublist. Copy 5 and 6 in L 2 to the sorted list. Sorted list 1 3 4 5 6 42

Merge Sort: Merging n How to merge two sublists into one list? n Compare two elements in L 1 and L 2 in sequence. n If the element in L 1 is less than or equal to that in L 2, move to the next position in L 1. n If the element in L 1 is greater than that in L 2, move to the next position in L 2. First sublist 3 7 8 12 Second sublist 1 4 5 6 Copy remaining items in L 1 to the sorted list. Sorted list 1 3 4 5 6 7 43 8 12

Implementation of Merging void Merge(Data* list, int left, int mid, int right) { int sorted[MAX_SIZE]; int first = left, second = mid + 1, i = left; // Merge two lists by comparing elements in sequence. while (first <= mid && second <= right) { if (list[first] <= list[second]) sorted[i++] = list[first++]; else sorted[i++] = list[second++]; } // For remaining items, add them in sequence. if (first > mid) for (int j = second; j <= right; j++) sorted[i++] = list[j]; else for (int j = first; j <= mid; j++) sorted[i++] = list[j]; // Copy the sorted list to the list. for (int j = left; j <= right; j++) list[j] = sorted[j]; } 44

Implementation of Merge Sort n Overall procedure n n n Partitioning: split the list into two halves. Merge: Merge two sorted sublits into one list. Recursively apply the above steps to the sublists. void Merge. Sort(Data* list, int left, int { if (left < right) { int mid = (left + right) / 2; Merge. Sort(list, left, mid); Merge. Sort(list, mid + 1, right); Merge(list, left, mid, right); } } 45 right) // // Equal partitioning Sorting sublists Merging two sublists

Exercise: Merge Sort n Animation: sorting 3, 4, 5, 4, 9, 1, 2, 6, 7 n n Draw the step-by-step procedure of insertion sort. https: //visualgo. net/en/sorting Initial list n After sorting two sublists Q: Is it stable? 46

Iterative Merge Sort n Iterative merge sort algorithm n Increase the length of merging two lists in sequence. 7 12 8 3 1 6 5 3 Length 1 7 12 8 3 1 6 5 4 Length 2 7 12 3 8 1 6 4 5 Length 4 3 7 8 12 1 4 5 6 Length 8 1 3 4 5 6 7 8 12 47

Iterative Merge Sort void Iter. Merge. Sort(Data* list, int n) { // Merge subarrays in bottom up manner. First merge subarrays of // size 1 to create sorted subarrays of size 2, then merge subarrays // of size 2 to create sorted subarrays of size 4, and so on. for (int size = 1; size <= n - 1; size = 2 * size) { // Pick starting point of different subarrays of current size for (int left_start = 0; left_start < n - 1; left_start += 2 * size) { // Find ending point of left subarray. // mid+1 is starting point of right int mid = left_start + size - 1; int right_end = MIN(left_start + 2 * size - 1, n - 1); // Merge Subarrays arr[left_start. . . mid] & arr[mid+1. . . right_end] Merge(list, left_start, mid, right_end); } } } 48

Analysis of Merge Sort n 49

Comparison of Sorting Algorithms Algorithm Best Average Selection sort Bubble sort Insertion sort Shell sort Quick sort Merge sort Heap sort Radix sort 50 Worst

Comparison of Sorting Algorithms n Comparing the running time (N = 100 K) Algorithm Running time (sec) Selection sort 10. 842 Bubble sort 22. 894 Insertion sort 7. 438 Shell sort 0. 056 Quick sort 0. 014 Merge sort 0. 026 Heap sort 0. 034 51

Summary of Sorting Algorithms n 52

Summary of Sorting Algorithms n Combining sorting algorithms n If # of of elements is small, insertion sort is the fastest. void Quick. Sort(Data* list, int left, int right) { if (left < right) { // The mid refers to the pivot position. int mid = Partition(list, left, right); // All elements are less than the pivot. if (mid - left < 20) Insertion. Sort(list, left, mid - 1); else Quick. Sort(list, left, mid - 1); // All elements are greater than the pivot. if (right - mid < 20) Insertion. Sort(list, mid+1, right); else Quick. Sort(list, mid+1, right); } } 53

Optimal Sorting Time n 54

Optimal Sorting Time n We now compare two elements Y and Z. n n If Y < Z, then how may candidates will be removed? n X Y Z n X Z Y n Y X Z n Y Z X n Z X Y n Z Y X OK, if you compare once more ? n n Ideally, if you compare once, a half of candidate list will be removed. How many comparisons do we need? 55

Optimal Sorting Time n We have a list with three elements X, Y, and Z. Yes [X, Y, Z] [X, Z, Y] [Z, X, Y] No No [X, Y, Z] [X, Z, Y] [Y, X, Z] [Y, Z, X] [Z, X, Y] [Z, Y, X] Yes Stop [X, Y, Z] [Y, X, Z] [Y, Z, X] [Z, X, Y] No Yes [Y, Z, X] [Z, X, Y] Stop Yes [X, Z, Y] [Z, X, Y] [Y, X, Z] No No Stop [X, Z, Y] [Z, X, Y] [Y, Z, X] [Z, X, Y] 56

Optimal Sorting Time n 57

Sorting Algorithms Non-comparison sorting algorithms

What is Counting Sort? n Description n Non-comparison sorting algorithm Count the number of elements with distinct key values. Output each element from the input sequence followed by decreasing its count by one. index count input 1 6 1 2 1 0 0 1 3 2 1 3 0 4 0 5 0 6 1 59 output 1 1 1 2 6 59

Example of Counting Sort n Description n Count the number of elements with distinct key values. n n Determine the positions of key values in the output. Output each element from the input sequence followed by decreasing its count by one. index count input 1 6 1 2 1 index count 0 0 1 3 2 1 2 4 3 0 3 4 4 0 4 4 5 0 5 4 6 1 6 5 60 60

Example of Counting Sort n Description n Count the number of elements with distinct key values. n n Determine the positions of key values in the output. Output each element from the input sequence followed by decreasing its count by one. index count input 1 6 1 2 1 Because the index of 1 is 3, store 1 to output[3]. 0 0 1 3 2 4 3 4 4 4 5 4 6 5 61 output 1 61

Example of Counting Sort n Description n Count the number of elements with distinct key values. n n Determine the positions of key values in the output. Output each element from the input sequence followed by decreasing its count by one. input 1 6 1 2 1 Because the index of 6 is 5, store 5 to output[5]. 0 0 1 2 2 4 3 4 4 4 5 4 6 5 62 output 1 6 62

Example of Counting Sort n Description n Count the number of elements with distinct key values. n n Determine the positions of key values in the output. Output each element from the input sequence followed by decreasing its count by one. input 1 6 1 2 1 Because the index of 1 is 2, store 1 to output[2]. 0 0 1 2 2 4 3 4 4 4 5 4 63 output 1 1 6 63

Example of Counting Sort n Description n Count the number of elements with distinct key values. n n Determine the positions of key values in the output. Output each element from the input sequence followed by decreasing its count by one. input 1 6 1 2 1 Because the index of 2 is 4, store 2 to output[4]. 0 0 1 1 2 4 3 4 4 4 5 4 64 output 1 1 2 6 64

Example of Counting Sort n Description n Count the number of elements with distinct key values. n n Determine the positions of key values in the output. Output each element from the input sequence followed by decreasing its count by one. input 1 6 1 2 1 Because the index of 1 is 1, store 1 to output[1]. 0 0 1 1 2 3 3 4 4 4 5 4 65 output 1 1 1 2 6 65

Implementation of Counting Sort void Counting. Sort(Data* list, int n) { Data count[MAX_SIZE] = { 0 }; Data output[MAX_SIZE]; // Counting the redundant elemnts for (int i = 0; i < n; i++) count[list[i]]++; // Cumulate the number of elements. for (int i = 1; i < MAX_SIZE; i++) count[i] += count[i - 1]; // Read the elements in the list and copy them to the output list. for (int i = 0; i < n; i++) { output[count[list[i]] - 1] = list[i]; count[list[i]]--; } // Copy the output list to the original list. for (int i = 0; i < n; i++) list[i] = output[i]; } 66 66

Exercise: Counting Sort n Animation: sorting 3, 4, 5, 4, 9, 1, 2, 6, 7 n n Draw the step-by-step procedure of counting sort. https: //visualgo. net/en/sorting Initial list n Q: Is it stable? 67 67

Analysis of Counting Sort n Characteristics n The time complexity is linear in the number of items and the difference between the maximum and minimum key values. n n It is only suitable for the case where the variation in keys is not significantly greater than the number of items. It is often used as a subroutine in another sorting algorithm, radix sort, that can handle larger keys more efficiently. 68 68

What is Radix Sort? n Description n n Non-comparison sorting algorithm Grouping keys by the individual digits which share the same significant position and value. index count input 12 16 22 34 46 X 2 X 4 X 6 0 0 1 0 2 2 3 0 4 1 5 0 6 2 69 output 12 22 34 16 46 69

What is Radix Sort? n Description n n Non-comparison sorting algorithm Grouping keys by the individual digits which share the same significant position and value. index count input 12 22 34 16 46 1 X 2 X 3 X 1 X 4 X 0 0 1 0 2 2 3 0 4 1 5 0 6 2 70 output 12 22 34 16 46 70

Implementation of Radix Sort void Counting(int list[], int n, int exp) { int count[10] = { 0 }; int output[MAX_SIZE]; // Store count of occurrences in count list. for (int i = 0; i < n; i++) count[(list[i] / exp) % 10]++; // Change count[i] so that count[i] contains actual position of this digit in output list. for (int i = 1; i < 10; i++) count[i] += count[i - 1]; // Build the output list. for (int i = n - 1; i >= 0; i--) { output[count[(list[i] / exp) % 10] - 1] = list[i]; count[(list[i] / exp) % 10]--; } // Copy the output list to list[], so that list[] now // contains sorted numbers according to current digit for (int i = 0; i < n; i++) list[i] = output[i]; } 71 71

Implementation of Radix Sort n Implementation void Radix. Sort(Data* list, int n) { // Find the maximum number to know the number of digits. int max = list[0]; for (int i = 1; i < n; i++) { if (list[i] > max) max = list[i]; } // Do counting sort for every digit. Note that instead // of passing digit number, exp is passed. exp is 10^i // where i is current digit number for (int exp = 1; max / exp > 0; exp *= 10) Counting(list, n, exp); } 72 72

Exercise: Radix Sort n Animation: sorting 3, 4, 5, 4, 9, 1, 2, 6, 7 n n Draw the step-by-step procedure of radix sort. https: //visualgo. net/en/sorting Initial list n Q: Is it stable? 73 73

Analysis of Radix Sort n 74 74