Algorithm Design Methods Spring 2007 CSE POSTECH Algorithm

  • Slides: 44
Download presentation
Algorithm Design Methods Spring 2007 CSE, POSTECH

Algorithm Design Methods Spring 2007 CSE, POSTECH

Algorithm Design Methods l l l Greedy method Divide and conquer Dynamic programming Backtracking

Algorithm Design Methods l l l Greedy method Divide and conquer Dynamic programming Backtracking Branch and bound

Some Methods Not Covered l l l Linear Programming Integer programming Simulated annealing Neural

Some Methods Not Covered l l l Linear Programming Integer programming Simulated annealing Neural networks Genetic algorithms Tabu search

Optimization Problem l A problem in which some function (called the optimization/objective function) is

Optimization Problem l A problem in which some function (called the optimization/objective function) is to be optimized (usually minimized or maximized) l It is subject to some constraints.

Machine Scheduling l Find a schedule that minimizes the finish time. – – optimization

Machine Scheduling l Find a schedule that minimizes the finish time. – – optimization function … finish time constraints l Each job is scheduled continuously on a single machine for an amount of time equal to its processing requirement. l No machine processes more than one job at a time.

Bin Packing l Pack items into bins using fewest number of bins. – –

Bin Packing l Pack items into bins using fewest number of bins. – – optimization function … number of bins constraints l Each item is packed into a single bin. l The capacity of no bin is exceeded.

Min Cost Spanning Tree l Find a spanning tree that has minimum cost. –

Min Cost Spanning Tree l Find a spanning tree that has minimum cost. – – optimization function … sum of edge costs constraints l Must select n-1 edges of the given n vertex graph. l The selected edges must form a tree.

Feasible and Optimal Solutions l A feasible solution is a solution that satisfies the

Feasible and Optimal Solutions l A feasible solution is a solution that satisfies the constraints. l An optimal solution is a feasible solution that optimizes the objective/optimization function.

Greedy Method l l Solve problem by making a sequence of decisions. Decisions are

Greedy Method l l Solve problem by making a sequence of decisions. Decisions are made one by one in some order. Each decision is made using a greedy criterion. A decision, once made, is (usually) not changed later.

Machine Scheduling LPT Scheduling. l Schedule jobs one by one in decreasing order of

Machine Scheduling LPT Scheduling. l Schedule jobs one by one in decreasing order of processing time (i. e. , longest processing time first). l Each job is scheduled on the machine on which it finishes earliest. l Scheduling decisions are made serially using a greedy criterion (minimize finish time of this job). l LPT scheduling is an application of the greedy method.

LPT Schedule l LPT rule does not guarantee minimum finish time schedules. (LPT Finish

LPT Schedule l LPT rule does not guarantee minimum finish time schedules. (LPT Finish Time)/(Minimum Finish Time) <= 4/3 – 1/(3 m) where m is number of machines. l l l Minimum finish time scheduling is NP-hard. In this case, the greedy method does not work. Greedy method does, however, give us a good heuristic for machine scheduling.

Container Loading Problem l l l Ship has capacity c. m containers are available

Container Loading Problem l l l Ship has capacity c. m containers are available for loading. Weight of container i is wi. Each weight is a positive number. Sum of container weight < c. Load as many containers as possible without sinking the ship.

Greedy Solution l l l Load containers in increasing order of weight until we

Greedy Solution l l l Load containers in increasing order of weight until we get to a container that does not fit. Does this greedy algorithm always load the maximum number of containers? Yes. May be proved using a proof by induction. (see Theorem 13. 1, p. 624 of text. )

Container Loading with 2 Ships l Can all containers be loaded into 2 ships

Container Loading with 2 Ships l Can all containers be loaded into 2 ships whose capacity is c each? – – – Same as bin packing with 2 bins (Are 2 bins sufficient for all items? ) Same as machine scheduling with 2 machines (Can all jobs be completed by 2 machines in c time units? ) NP-hard

0/1 Knapsack Problem l l l Hiker wishes to take n items on a

0/1 Knapsack Problem l l l Hiker wishes to take n items on a trip. The weight of item i is wi. The knapsack has a weight capacity c. If sum of items weights <= c, all n items can be carried in the knapsack. If sum of item weights > c, some items must be left behind. Which items should be taken out?

0/1 Knapsack Problem l l l Hiker assigns a profit/value pi to item i.

0/1 Knapsack Problem l l l Hiker assigns a profit/value pi to item i. All weights and profits are positive numbers. Hiker wants to select a subset of n items to take. – – The weight of the subset should not exceed the capacity of the knapsack. (constraint) Cannot select a fraction of an item. (constraint) The profit/value of the subset is the sum of the profits of the selected items. (optimization function) The profit/value of the selected subset should be maximum. (optimization criterion)

0/1 Knapsack Problem l Let xi=1 when item i is selected and let xi=0

0/1 Knapsack Problem l Let xi=1 when item i is selected and let xi=0 when item i is not selected. maximize Sigma(i=1…n) pixi subject to Sigma(i=1…n) wixi <= c l See the formula and constraints on page 625

Greedy Attempt 1 l l l Be greedy on capacity utilization (select items in

Greedy Attempt 1 l l l Be greedy on capacity utilization (select items in increasing order of weights). n = 2, c = 7 w = [3, 6] p = [2, 10] Only 1 item is selected, x = [1, 0]. Profit/value of selection is 2. It is not the best selection.

Greedy Attempt 2 l l l Be greedy on profit earned (select items in

Greedy Attempt 2 l l l Be greedy on profit earned (select items in decreasing order of profit). n = 3, c = 7 w = [7, 3, 2] p = [10, 8, 6] Only 1 item is selected , x = [1, 0, 0]. Profit/value of selection is 10. It is not the best selection.

Greedy Attempt 3 l l l Be greedy on profit density (p/w) (select items

Greedy Attempt 3 l l l Be greedy on profit density (p/w) (select items in decreasing order of profit density). n = 2, c = 7 w = [1, 7] p = [10, 20] Only 1 item is selected, x = [1, 0]. Profit/value of selection is 10. It is not the best selection.

Greedy Attempt 4 l l Select a subset with <= k items. If the

Greedy Attempt 4 l l Select a subset with <= k items. If the weight of this subset is > c, discard the subset. If the subset weight is <= c, fill as much of the remaining capacity as possible by being greedy on profit density. Try all subsets with <= k items and select the one that yields maximum profit.

Chapter 14: Divide and Conquer l A large problem is solved as follows: –

Chapter 14: Divide and Conquer l A large problem is solved as follows: – – – l Divide the large problem into smaller problems. Solve the smaller problems somehow. Combine the results of the smaller problems to obtain the result for the original large problem. A small problem is solved in some other way.

Small and Large Problem l Small problem – – l Sort a list that

Small and Large Problem l Small problem – – l Sort a list that has n <= 10 elements. Find the minimum of n <= 2 elements. Large problem – – Sort a list that has n > 10 elements. Find the minimum of n > 2 elements.

Solving a Small Problem l A small problem is solved using some direct/simple strategy.

Solving a Small Problem l A small problem is solved using some direct/simple strategy. – – Sort a list that has n <= 10 elements. Use insertion, bubble, or selection sort. Find the minimum of n <= 2 elements. When n = 0, there is no minimum element. When n = 1, the single element is the minimum. When n = 2, compare the two elements and determine which is smaller.

Sort a Large List l Sort a list that has n > 10 elements.

Sort a Large List l Sort a list that has n > 10 elements. – – – Sort 15 elements by dividing them into 2 smaller lists. One list has 7 elements and the other has 8 elements. Sort these two lists using the method for small lists. Merge the two sorted lists into a single sorted list.

Find the Min of a Large List l Find the minimum of 20 elements.

Find the Min of a Large List l Find the minimum of 20 elements. – – – Divide into two groups of 10 elements each. Find the minimum element in each group somehow. Compare the minimums of each group to determine the overall minimum.

Recursion In Divide and Conquer l Often the smaller problems that result from the

Recursion In Divide and Conquer l Often the smaller problems that result from the divide step are instances of the original problem (true for our sort and min problems). In this case, – – l If the new problem is a smaller problem, it is solved using the method for small problems. If the new problem is a large instance, it is solved using the divide-and-conquer method recursively. Generally, performance is best when the smaller problems that result from the divide step are of approximately the same size.

Recursive Find Min l Find the minimum of 20 elements. – – – Divide

Recursive Find Min l Find the minimum of 20 elements. – – – Divide into two groups of 10 elements each. Find the minimum element in each group recursively. The recursion terminates when the number of elements is <= 2. At this time the minimum is found using the method for small problems. Compare the minimums of each group to determine the overall minimum.

Merge Sort – another Divide & Conquer Example l Sort the first half of

Merge Sort – another Divide & Conquer Example l Sort the first half of the array using merge sort. l Sort the second half of the array using merge sort. l Merge the first half of the array with the second half.

Merge Sort Algorithm l l l Merge is an operation that combines two sorted

Merge Sort Algorithm l l l Merge is an operation that combines two sorted arrays. Assume the result is to be placed in a separate array called result (already allocated). The two given arrays are called front and back are in increasing order. For the complexity analysis, the size of the input, n, is the sum nfront + nback.

Merge Sort Algorithm l l l For each array keep track of the current

Merge Sort Algorithm l l l For each array keep track of the current position. REPEAT until all the elements of one of the given arrays have been copied into result: – Compare the current elements of front and back. – Copy the smaller into the current position of result (break the ties however you like). – Increment the current position of result and the array that was copied from. Copy all the remaining elements of the other given array into result.

Merge Sort Algorithm - Complexity l Every element in front and back is copied

Merge Sort Algorithm - Complexity l Every element in front and back is copied exactly once. Each copy is two accesses, so the total number of accessing due to copying is 2 n. l The number of comparisons could be as small as min(nfront, nback) or as large as n-1. Each comparison is two accesses.

Merge Sort Algorithm - Complexity l In the worst case the total number of

Merge Sort Algorithm - Complexity l In the worst case the total number of accesses is 2 n +2(n-1) = O(n). l In the best case the total number of accesses is 2 n + 2 min(nfront, nback) = O(n). l The average case is between the worst and best case and is therefore also O(n).

Merge Sort Algorithm l Split an. Array into two non-empty parts anyway you like.

Merge Sort Algorithm l Split an. Array into two non-empty parts anyway you like. For example, front = the first n/2 elements in an. Array back = the remaining elements in an. Array l Sort front and back by recursively calling Merge. Sort. l Now you have two sorted arrays containing all the elements from the original array. Use merge to combine them, put the result in an. Array.

Merge. Sort Call Graph (n=7) 0~6 0~2 0~0 1~2 1~1 l l 3~6 3~4

Merge. Sort Call Graph (n=7) 0~6 0~2 0~0 1~2 1~1 l l 3~6 3~4 2~2 3~3 5~6 4~4 Each box represents one invocation of Merge. Sort. How many levels are there in general if the array is divided in half each time? 5~5 6~6

Merge. Sort Call Graph (general) l l l Suppose n = 2 k. How

Merge. Sort Call Graph (general) l l l Suppose n = 2 k. How many levels? How many boxes on level j? What values is in each box at level j? n n/2 n/4 1 1

Quick Sort l Quicksort can be seen as a variation of mergesort in which

Quick Sort l Quicksort can be seen as a variation of mergesort in which front and back are defined in a different way.

Quicksort Algorithm l Partition an. Array into two non-empty parts. – – l l

Quicksort Algorithm l Partition an. Array into two non-empty parts. – – l l Pick any value in the array, pivot. small = the elements in an. Array < pivot large = the elements in an. Array > pivot Place pivot in either part, so as to make sure neither part is empty. Sort small and large by recursively calling Quick. Sort. How would you merge the two arrays? – You could use merge to combine them, but because the elements in small are smaller than elements in large, simply concatenate small and large, and put the result into an. Array.

Quicksort: Complexity Analysis l Like mergesort, a single invocation of quicksort on an array

Quicksort: Complexity Analysis l Like mergesort, a single invocation of quicksort on an array of size p has complexity O(p): – – l p comparisons = 2*p accesses 2*p moves (copying) = 4*p accesses Best case: every pivot chosen by quicksort partitions the array into equal-sized parts. In this case quicksort is the same big-O complexity as mergesort – O(n log n)

Quicksort: Complexity Analysis l l What would be the worst case scenario? Worst case:

Quicksort: Complexity Analysis l l What would be the worst case scenario? Worst case: the pivot chosen is the largest or smallest value in the array. Partition creates one part of size 1 (containing only the pivot), the other of size p-1. n 1 n-1 1 n-2 1 1

Quicksort: Complexity Analysis Worst case: l There are n-1 invocations of quicksort (not counting

Quicksort: Complexity Analysis Worst case: l There are n-1 invocations of quicksort (not counting base cases) with arrays of size: p = n, n-1, n-2, …, 2 l Since each of these does O(p), the total number of accesses is O(n) + O(n-1) + … + O(1) = O(n 2) l Ironically the worst case occurs when the list is sorted (or near sorted)!

Quicksort: Complexity Analysis l l The average case must be between the best case

Quicksort: Complexity Analysis l l The average case must be between the best case O(n log n) and the worst case is O(n 2). Analysis yields a complex recurrence relation. The average case number of comparisons turns out to be approximately 1. 386*n*log n – 2. 846*n. Therefore the average case time complexity is O(n log n).

Quicksort: Complexity Analysis l l Best case O(n log n) Worst case O(n 2)

Quicksort: Complexity Analysis l l Best case O(n log n) Worst case O(n 2) Average case O(n log n) Note that the quick sort is inferior to insertion sort and merge sort if the list is sorted, nearly sorted, or reverse sorted.

READING l READ Chapter 13 & 14 l Chapters 15 (Dynamic Programming), 16 (Backtracking)

READING l READ Chapter 13 & 14 l Chapters 15 (Dynamic Programming), 16 (Backtracking) and 17 (Branch and Bound) are useful algorithm design methods and you should read and try to understand them l The final exam will mostly cover (about 90%) from the second half materials