Greedy Algorithms Slide 1 Optimization Problems Optimization Problem

Greedy Algorithms Slide 1

Optimization Problems • Optimization Problem – Problem with an objective function to either: • Maximize some profit • Minimize some cost • Optimization problems appear in so many applications – Maximize the number of jobs using a resource [Activity-Selection Problem] – Encode the data in a file to minimize its size [Huffman Encoding Problem] – Collect the maximum value of goods that fit in a given bucket [knapsack Problem] – Select the smallest-weight of edges to connect all nodes in a graph [Minimum Spanning Tree] Slide 2

Solving Optimization Problems • Two techniques for solving optimization problems: – Greedy Algorithms (“Greedy Strategy”) – Dynamic Programming Space of optimization problems Greedy algorithms can solve some problems optimally Dynamic programming can solve more problems optimally (superset) We still care about Greedy Algorithms because for some problems: • Dynamic programming is overkill (slow) • Greedy algorithm is simpler and more efficient Slide 3

Greedy Algorithms • Main Concept – Divide the problem into multiple steps (sub-problems) – For each step take the best choice at the current moment (Local optimal) (Greedy choice) – A greedy algorithm always makes the choice that looks best at the moment – The hope: A locally optimal choice will lead to a globally optimal solution • For some problems, it works. For others, it does not Slide 4

Activity-Selection Problem • Given a set of activities A 1, A 2, …An (E. g. , talks or lectures) • Each activity has a start time and end time – Each Ai has (Si, Ei) • Activities should use a common resource (E. g. , Lecture hall) • Objective: Maximize the number of “compatible” activities that use the resource – Cannot have two overlapping activities A 3 A 1 A 7 A 5 A 2 A 4 A 6 A 8 Time dimension Slide 5

Another Version Of The Same Problem • A set of events A 1, A 2, …An • Each event has a start time and end time • You want to attend as many events as possible Slide 6

Example of Compatible Activities • Set of activities A = {A 1, A 2, …. , An} Examples of compatible activities: {A 1, A 3, A 8} {A 1, A 4, A 5, A 7} {A 2, A 5, A 8} …. • Each Ai = (Si, Ei) A 3 A 1 A 7 A 5 A 2 A 4 A 6 A 8 Time dimension Slide 7

Greedy Algorithm • Select the activity that ends first (smallest end time) – Intuition: it leaves the largest possible empty space for more activities • Once selected an activity – Delete all non-compatible activities – They cannot be selected • Repeat the algorithm for the remaining activities – Either using iterations or recursion Slide 8

Greedy Algorithm • Select the activity that ends first (smallest end time) – Intuition: it leaves the largest possible empty space for more activities • Once selected an activity – Delete all non-compatible activities – They cannot be selected • Repeat the algorithm for the remaining activities – Either using iterations or recursion Greedy Choice: Select the next best activity (Local Optimal) Sub-problem: We created one sub-problem to solve (Find the optimal schedule after the selected activity) Hopefully when we merge the local optimal + the subproblem optimal solution we get a global optimal Slide 9

Example The greedy algorithm will select: {A 1, A 4, A 6, A 7} Is that an optimal answer? ? Can we find a larger set? Slide 10

Optimal Solution • The Greedy Algorithm leads to optimal solution • How to prove it – We can convert any other optimal solution (S’) to the greedy algorithm solution (S) • Idea: – Compare the activities in S’ and S from left-to-right – If they match in the selected activity skip – If they do not match • We can replace the activity in S’ by that in S because the one in S finishes first Slide 11

Example • • Greedy solution S = {A 1, A 4, A 6, A 7} Another solution S’ = {A 1, A 4, A 5, A 8} A 5 in S’ can be replaced by A 6 from S (finishes earlier) A 8 in S’ can be replaced by A 7 from S (finishes earlier) By these replacements, we saved time. So, Greedy must be at least as good as any other optimal solution Slide 12

Recursive Solution Two arrays containing the start and end times (Assumption: they are sorted based on end times) The activity chosen in the last call The problem size Recursive-Activity-Selection(S, E, k, n) m = k +1 Find the next activity starting While (m <= n) && ( S[m] < E[k]) after the end of k m++; If (m <= n) return {Am} U Recursive-Activity-Selection(S, E, m, n) Else return Φ Time Complexity: O(n) (Assuming arrays are already sorted, otherwise we add O(n Log n) Slide 13

Iterative Solution Two arrays containing the start and end times (Assumption: they are sorted based on end times) Iterative-Activity-Selection(S, E) n = S. Length List = {A 1} last. Selection = 1 For (i = 2 to n) if (S[i] >= E[last. Selection]) List = List U {Ai} last. Selection = i End If End Loop Return List Slide 14

Elements Of Greedy Algorithms • Greedy-Choice Property – At each step, we do a greedy (local optimal) choice • Top-Down Solution – The greedy choice is usually done independent of the subproblems – Usually done “before” solving the sub-problem • Optimal Substructure – The global optimal solution can be composed from the local optimal of the sub-problems Slide 15

Elements Of Greedy Algorithms • Proving a greedy solution is optimal – Remember: Not all problems have optimal greedy solution – If it does, you need to prove it – Usually the proof includes mapping or converting any other optimal solution to the greedy solution We mapped S’ to S and showed that S is even better • • Greedy solution S = {A 1, A 4, A 6, A 7} Another solution S’ = {A 1, A 4, A 5, A 8} A 5 in S’ can be replaced by A 6 from S (finishes earlier) A 8 in S’ can be replaced by A 7 from S (finishes earlier) Slide 16

Activity-Selection Problem Slide 17

Knapsack Problem • Several Types – 0 -1 Knapsack Problem – Fractional Knapsack Problem • 0 -1 Knapsack does not have an optimal greedy algorithm • Fractional Knapsack has an optimal greedy solution Slide 18

Knapsack Problem: Definition n Thief has a knapsack with maximum capacity W, and a set S consisting of n items n Each item i has some weight wi and benefit value vi (all wi , vi and W are integer values) n Problem: How to pack the knapsack to achieve maximum total value of packed items? Slide 19

0 -1 Knapsack • Items cannot be divided – Either take it or leave it qfind xi such that for all xi = {0, 1}, i = 1, 2, . . , n wixi W and xivi is maximum If Xi = 1, then item i will be taken If Xi = 0, then item i will be skipped Slide 20

0 -1 Knapsack - Greedy Strategy Does Not Work • E. g. 1: 30 $120 Item 3 50 Item 2 Item 1 30 20 $100 + 20 10 $60 50 10 $100 $120 W 50 20 Greedy choice: – Compute the benefit per pound – Sort the items based on these values $100 $60 $160 $6/pound $5/pound $4/pound • + Not optimal $220

Fractional Knapsack • Items can be divided – Can take part of it as needed qfind xi such that for all 0 <= xi <= 1, i = 1, 2, . . , n wixi W and xivi is maximum If Xi = 0, then item i will be skipped If Xi > 0, then Xi fraction of item i will be taken Slide 22

Fractional Knapsack - Greedy Strategy Works • E. g. 1: 2/3 Of 30 Item 3 50 Item 2 Item 1 30 20 $100 + 10 $100 $120 + 50 20 10 $60 $80 W $60 $240 $6/pound $5/pound $4/pound • Greedy choice: Optimal – Compute the benefit per pound – Sort the items based on these values – Take as much as you can from the top items in the list

Example q q q Item: 1 2 3 4 5 6 Benefit: 5 8 3 2 7 9 Weight: 7 8 4 10 4 6 B/W: 0. 71 1 0. 75 0. 2 1. 75 1. 5 Knapsack holds a maximum of 22 pounds q I 5 (remaining 18) q I 6 (remaining 12) q I 2 (remaining 4) q I 7 (remaining 0) 7 4 4 1 Slide 24