CSCE 411 Design and Analysis of Algorithms Set

CSCE 411 Design and Analysis of Algorithms Set 5: Dynamic Programming Slides by Prof. Jennifer Welch Spring 2014 CSCE 411, Spring 2014: Set 5 1

Drawback of Divide & Conquer Sometimes can be inefficient n Fibonacci numbers: n n n F 0 = 0 F 1 = 1 Fn = Fn-1 + Fn-2 for n > 1 Sequence is 0, 1, 1, 2, 3, 5, 8, 13, … CSCE 411, Spring 2014: Set 5 2

Computing Fibonacci Numbers n Obvious recursive algorithm: n Fib(n): n n if n = 0 or 1 then return n else return (F(n-1) + Fib(n-2)) CSCE 411, Spring 2014: Set 5 3

Recursion Tree for Fib(5) Fib(4) Fib(3) Fib(2) Fib(1) Fib(3) Fib(2) Fib(0) Fib(2) Fib(1) Fib(0) CSCE 411, Spring 2014: Set 5 4

How Many Recursive Calls? If all leaves had the same depth, then there would be about 2 n recursive calls. n But this is over-counting. n However with more careful counting it can be shown that it is Ω((1. 6)n) n Still exponential! n CSCE 411, Spring 2014: Set 5 5

Save Work n Wasteful approach - repeat work unnecessarily n n Fib(2) is computed three times Instead, compute Fib(2) once, store result in a table, and access it when needed CSCE 411, Spring 2014: Set 5 6

More Efficient Recursive Alg F[0] : = 0; F[1] : = 1; F[n] : = Fib(n); n Fib(n): n n n if n = 0 or 1 then return F[n] if F[n-1] = NIL then F[n-1] : = Fib(n-1) if F[n-2] = NIL then F[n-2] : = Fib(n-2) return (F[n-1] + F[n-2]) computes each F[i] only once CSCE 411, Spring 2014: Set 5 7

Example of Memoized Fib 0 1 2 3 4 5 F 0 1 1 NIL 2 NIL 3 NIL 5 NIL Fib(5) fills in F[4] with 3, returns 3+2 = 5 Fib(4) fills in F[3] with 2, returns 2+1 = 3 Fib(3) fills in F[2] with 1, returns 1+1 = 2 Fib(2) returns 0+1 = 1 CSCE 411, Spring 2014: Set 5 8

Get Rid of the Recursion n Recursion adds overhead n n n extra time for function calls extra space to store information on the runtime stack about each currently active function call Avoid the recursion overhead by filling in the table entries bottom up, instead of top down. CSCE 411, Spring 2014: Set 5 9

Subproblem Dependencies Figure out which subproblems rely on which other subproblems n Example: n F 0 F 1 F 2 F 3 … CSCE 411, Spring 2014: Set 5 Fn-2 Fn-1 Fn 10

Order for Computing Subproblems n Then figure out an order for computing the subproblems that respects the dependencies: n n when you are solving a subproblem, you have already solved all the subproblems on which it depends Example: Just solve them in the order F 0 , F 1 , F 2 , F 3 , … CSCE 411, Spring 2014: Set 5 11

DP Solution for Fibonacci n Fib(n): n n F[0] : = 0; F[1] : = 1; for i : = 2 to n do n n n F[i] : = F[i-1] + F[i-2] return F[n] Can perform application-specific optimizations n e. g. , save space by only keeping last two numbers computed CSCE 411, Spring 2014: Set 5 12

Matrix Chain Order Problem n Multiplying non-square matrices: n n must be equal A is n x m, B is m x p AB is n x p whose (i, j) entry is ∑aik bkj Computing AB takes nmp scalar multiplications and n(m-1)p scalar additions (using basic algorithm). n Suppose we have a sequence of matrices to multiply. What is the best order? n CSCE 411, Spring 2014: Set 5 13

Why Order Matters n Suppose we have 4 matrices: n n A, 30 x 1 B, 1 x 40 C, 40 x 10 D, 10 x 25 ((AB)(CD)) : requires 41, 200 mults. n (A((BC)D)) : requires 1400 mults. n CSCE 411, Spring 2014: Set 5 14

Matrix Chain Order Problem n Given matrices A 1, A 2, …, An, where Ai is di-1 x di: [1] What is minimum number of scalar mults required to compute A 1· A 2 ·… · An? [2] What order of matrix multiplications achieves this minimum? Focus on [1]; see textbook for how to do [2] CSCE 411, Spring 2014: Set 5 15

A Possible Solution Try all possibilities and choose the best one. n Drawback is there are too many of them (exponential in the number of matrices to be multiplied) n Need to be more clever - try dynamic programming! n CSCE 411, Spring 2014: Set 5 16

Step 1: Develop a Recursive Solution Define M(i, j) to be the minimum number of mults. needed to compute Ai· Ai+1 ·… · Aj n Goal: Find M(1, n). n Basis: M(i, i) = 0. n Recursion: How to define M(i, j) recursively? n CSCE 411, Spring 2014: Set 5 17

Defining M(i, j) Recursively Consider all possible ways to split Ai through Aj into two pieces. n Compare the costs of all these splits: n n n best case cost for computing the product of the two pieces plus the cost of multiplying the two products Take the best one n M(i, j) = mink(M(i, k) + M(k+1, j) + di-1 dkdj) n CSCE 411, Spring 2014: Set 5 18

Defining M(i, j) Recursively (Ai ·…· Ak)·(Ak+1 ·… · Aj) P 1 P 2 • minimum cost to compute P 1 is M(i, k) • minimum cost to compute P 2 is M(k+1, j) • cost to compute P 1· P 2 is di-1 dkdj CSCE 411, Spring 2014: Set 5 19

Step 2: Find Dependencies Among Subproblems M: 2 1 1 0 2 n/a 0 3 4 5 GOAL! 3 n/a 0 4 n/a n/a 0 5 n/a n/a 0 CSCE 411, Spring 2014: Set 5 computing the red square requires the blue ones: to the left and below. 20

Defining the Dependencies n Computing M(i, j) uses n n everything in same row to the left: M(i, i), M(i, i+1), …, M(i, j-1) and everything in same column below: M(i, j), M(i+1, j), …, M(j, j) CSCE 411, Spring 2014: Set 5 21

Step 3: Identify Order for Solving Subproblems Recall the dependencies between subproblems just found n Solve the subproblems (i. e. , fill in the table entries) this way: n n go along the diagonal start just above the main diagonal end in the upper right corner (goal) CSCE 411, Spring 2014: Set 5 22

Order for Solving Subproblems M: 2 1 1 0 3 4 5 2 n/a 0 3 n/a 0 4 n/a n/a 0 5 n/a n/a 0 CSCE 411, Spring 2014: Set 5 23

Pseudocode for i : = 1 to n do M[i, i] : = 0 for d : = 2 to n-1 do // diagonals for i : = 1 to n-d+1 to // rows w/ an entry on d-th diagonal j : = i + d - 1 // column corresponding to row i on d-th diagonal M[i, j] : = infinity pay attention here for k : = i to j-1 to to remember actual M[i, j] : = min(M[i, j], M[i, k]+M[k+1, j]+di-1 dkdj) sequence of mults. endfor running time O(n 3) endfor CSCE 411, Spring 2014: Set 5 24

Example M: 1 2 3 4 1 0 1200 700 1400 2 n/a 0 400 650 3 n/a 0 10, 000 4 n/a n/a 0 CSCE 411, Spring 2014: Set 5 1: A is 30 x 1 2: B is 1 x 40 3: C is 40 x 10 4: D is 10 x 25 25

Keeping Track of the Order It's fine to know the cost of the cheapest order, but what is that cheapest order? n Keep another array S and update it when computing the minimum cost in the inner loop n After M and S have been filled in, then call a recursive algorithm on S to print out the actual order n CSCE 411, Spring 2014: Set 5 26

Modified Pseudocode for i : = 1 to n do M[i, i] : = 0 for d : = 1 to n-1 do // diagonals for i : = 1 to n-d to // rows w/ an entry on d-th diagonal j : = i + d // column corresponding to row i on d-th diagonal M[i, j] : = infinity for k : = 1 to j-1 to M[i, j] : = min(M[i, j], M[i, k]+M[k+1, j]+di-1 dkdj) if previous line changed value of M[i, j] then S[i, j] : = k endfor keep track of cheapest split point found so far: between Ak and Ak+1 �� CSCE 411, Spring 2014: Set 5 27

Example M: S: 1 2 3 4 1 0 1200 700 2 n/a 0 400 3 n/a 0 10, 000 4 n/a n/a 0 1 1 2 1400 650 1 3 1: A is 30 x 1 2: B is 1 x 40 3: C is 40 x 10 4: D is 10 x 25 3 CSCE 411, Spring 2014: Set 5 28

Using S to Print Best Ordering Call Print(S, 1, n) to get the entire ordering. Print(S, i, j): if i = j then output "A" + i //+ is string concat else k : = S[i, j] output "(" + Print(S, i, k) + Print(S, k+1, j) + ")" CSCE 411, Spring 2014: Set 5 29

Example S: 1 2 4 1 n/a 1 1 1 2 n/a 2 3 3 n/a n/a 3 4 n/a n/a <<draw recursion tree on board>> CSCE 411, Spring 2014: Set 5 30

Longest Common Subsequence n n Another example of dynamic programming Given two strings n n n find the longest common subsequence (LCS) of X and Y n n X = [x 1, x 2 , …, xm] and Y = [y 1, y 2, …, yn] not necessarily contiguous! Example: n n X = [A, B, C, B, D, A, B] Y = [B, D, C, A, B, A] one LCS is [B, C, B, A] another LCS is [B, D, A, B] CSCE 411, Spring 2014: Set 5 31

Brute Force Solution Take one of the strings, say X n Consider each possible subsequence of X n Check if that subsequence appears in Y n n Would take exponential time since there are 2 m possible subsequences of X n for each of the m characters in X, there are two choices (it is in the subsequence or is not) CSCE 411, Spring 2014: Set 5 32

Recursive Expression for Solution n Let C[i, j] be length of LCS of n n Xi = [x 1, …, xi] and Yj = [y 1, …, yj] Goal: C[m, n] n Basis: n n C[0, j] = 0 (Xi is empty) for all j C[i, 0] = 0 (Yj is empty) for all i i ranges from 0 to m, j ranges from 0 to n CSCE 411, Spring 2014: Set 5 33

Recursive Expression for Solution Formula for C[i, j]: n Case 1: xi = yj. n n n Recursively find LCS of Xi-1 and Yj-1 and append xi So C[i, j] = C[i-1, j-1] + 1 if i > 0, j > 0, and xi = yj Xi-1 xi equal Yj-1 yj CSCE 411, Spring 2014: Set 5 34

Recursive Expression for Solution n Case 2: xi ≠ yj. n n Recursively find LCS of Xi-1 and Yj Recursively find LCS of Xi and Yj-1 Take the longer one So C[i, j] = max{C[i, j-1], C[i-1, j]} if i > 0, j > 0, and x i ≠ yj Xi-1 xi not equal Yj-1 yj CSCE 411, Spring 2014: Set 5 35

Determine Dependencies Among Subproblems n C[i, j] depends on C[i-1, j-1], C[i-1, j], and C[i, j j-1 j n -1] 0 1 i-1 i m Goal CSCE 411, Spring 2014: Set 5 36

Determine Order for Solving Subproblems n An order for solving the subproblems (i. e. , filling in the array) that respects the dependencies is row major order: n n do the rows from top to bottom inside each row, go from left to right CSCE 411, Spring 2014: Set 5 37

Pseudocode fill in row 0 with all 0’s fill in column 0 with all 0’s for each row 1 through m do for each column 1 through n do fill in C[i, j] according to formula CSCE 411, Spring 2014: Set 5 38

Time Complexity There are O(mn) subproblems (entries in the array). n Each subproblem takes O(1) time to compute. n Total time is O(mn). n CSCE 411, Spring 2014: Set 5 39

Computing the Actual Subsequence n Keep track of which neighboring table entry gives the optimal solution to a subproblem: n n n Use another 2 -D array, b, for this information. n n above, to the left, or diagonal break ties arbitrarily store in b[i, j] if xi = yj (this character is part of a common subsequence) store in b[i, j] if LCS of Xi and Yj-1 was longer store in b[i, j] if LCS of Xi-1 and Yj was longer Trace backwards from b[m, n] until reaching an edge n every time is encountered, the corresponding character is added to the longest common subsequence being constructed CSCE 411, Spring 2014: Set 5 40