Optimization Problems Where we learn more fun things

Optimization Problems Where we learn more fun things we can do with constraints. CSC 5240 - Simplification, Optimization and Implication 1

Optimization - 1 Often given some problem which is modelled by constraints we don’t want just any solution, but a “best” solution n This is an optimization problem n We need an objective function, a mapping from solutions to reals, so that we can rank solutions n By convention, we always try to minimize n CSC 5240 - Simplification, Optimization and Implication 2

Optimization - 2 An optimization problem (C, f) consists of a constraint C and objective function f n A valuation v 1 is preferred to valuation v 2 if f(v 1) < f(v 2) n An optimal solution is a solution of C such that no other solution of C is preferred to it n Optimization problems may have more than one optimal solution n CSC 5240 - Simplification, Optimization and Implication 3

Optimization Example An optimization problem Find the closest point to the origin satisfying C. Some solutions and f values Optimal solution CSC 5240 - Simplification, Optimization and Implication 4

Optimization Peculiarities n Some optimization problems have no solution. u Constraint has no solution u For any solution, there is a more preferable one. In other words, problem has no optimum. Can you give an example? CSC 5240 - Simplification, Optimization and Implication 5

Optimization for FD CSPs Domains are finite n We can use a solver to build a straightforward optimizer n retry_int_opt(C, D, f, best) n u D 2 : = int_solv(C, D) u if D 2 is a false domain then return best u let sol be the solution corresponding to D 2 u return retry_int_opt(C / f < sol(f), D, f, sol) CSC 5240 - Finite Constraint Domains 6

Retry Optimization Example Smugglers knapsack problem (optimize profit) First solution found: Corresponding solution CSC 5240 - Finite Constraint Domains 7

Retry Optimization Example Smugglers knapsack problem (optimize profit) First solution found: Corresponding solution CSC 5240 - Finite Constraint Domains 8

Retry Optimization Example Smugglers knapsack problem (optimize profit) Next solution found: Corresponding solution CSC 5240 - Finite Constraint Domains 9

Retry Optimization Example Smugglers knapsack problem (optimize profit) Next solution found: Corresponding solution CSC 5240 - Finite Constraint Domains 10

Retry Optimization Example Smugglers knapsack problem (optimize profit) Next solution found: Corresponding solution CSC 5240 - Finite Constraint Domains 11

Retry Optimization Example Smugglers knapsack problem (optimize profit) No next solution! Corresponding solution CSC 5240 - Finite Constraint Domains 12

Retry Optimization Example Smugglers knapsack problem (optimize profit) No next solution! Return best solution CSC 5240 - Finite Constraint Domains 13

Branch and Bound (BAB) Since the solver may use backtrack search anyway combine it with the optimization n At each step in backtracking search, if best is the best solution so far add the constraint f < best(f) n CSC 5240 - Finite Constraint Domains 14

Branch and Bound (BAB) Eg Smugglers knapsack problem (whiskey available) Current domain: CSC 5240 - Finite Constraint Domains 15

Branch and Bound (BAB) Eg Smugglers knapsack problem (whiskey available) Current domain: Initial bounds consistency CSC 5240 - Finite Constraint Domains 16

Branch and Bound (BAB) Eg Smugglers knapsack problem (whiskey available) Current domain: Initial bounds consistency CSC 5240 - Finite Constraint Domains 17

Branch and Bound (BAB) Eg Smugglers knapsack problem (whiskey available) Current domain: Initial bounds consistency W=0 CSC 5240 - Finite Constraint Domains 18

Branch and Bound (BAB) Eg Smugglers knapsack problem (whiskey available) Current domain: Initial bounds consistency W=0 CSC 5240 - Finite Constraint Domains 19

Branch and Bound (BAB) Eg Smugglers knapsack problem (whiskey available) Current domain: Initial bounds consistency W=0 P=1 CSC 5240 - Finite Constraint Domains 20

Branch and Bound (BAB) Eg Smugglers knapsack problem (whiskey available) Current domain: Initial bounds consistency W=0 P=1 CSC 5240 - Finite Constraint Domains 21

Branch and Bound (BAB) Eg Smugglers knapsack problem (whiskey available) Current domain: Initial bounds consistency W=0 P=1 (0, 1, 3) Solution Found: add constraint CSC 5240 - Finite Constraint Domains 22

Branch and Bound (BAB) Eg Smugglers knapsack problem (whiskey available) Current domain: Initial bounds consistency W=0 P=1 (0, 1, 3) Solution Found: add constraint CSC 5240 - Finite Constraint Domains 23

Branch and Bound (BAB) Eg Smugglers knapsack problem (whiskey available) Initial bounds consistency W=0 P=1 (0, 1, 3) CSC 5240 - Finite Constraint Domains 24

Branch and Bound (BAB) Eg Smugglers knapsack problem (whiskey available) Initial bounds consistency W=0 P=1 P=2 (0, 1, 3) false CSC 5240 - Finite Constraint Domains 25

Branch and Bound (BAB) Eg Smugglers knapsack problem (whiskey available) Initial bounds consistency W=0 P=1 P=2 P=3 (0, 1, 3) false CSC 5240 - Finite Constraint Domains 26

Branch and Bound (BAB) Eg Smugglers knapsack problem (whiskey available) Initial bounds consistency W=0 W=1 P=2 P=3 (0, 1, 3) false (1, 1, 1) CSC 5240 - Finite Constraint Domains 27

Branch and Bound (BAB) Eg Smugglers knapsack problem (whiskey available) Initial bounds consistency W=0 W=1 P=2 P=3 (0, 1, 3) false (1, 1, 1) Modify constraint CSC 5240 - Finite Constraint Domains 28

Branch and Bound (BAB) Eg Smugglers knapsack problem (whiskey available) Initial bounds consistency W=0 W=1 P=2 P=3 (0, 1, 3) false (1, 1, 1) Modify constraint CSC 5240 - Finite Constraint Domains 29

Branch and Bound (BAB) Eg Smugglers knapsack problem (whiskey available) Initial bounds consistency W=0 P=1 P=2 P=3 (0, 1, 3) false W=1 W=2 (1, 1, 1) false Modify constraint CSC 5240 - Finite Constraint Domains 30

Branch and Bound (BAB) Eg Smugglers knapsack problem (whiskey available) Initial bounds consistency W=0 P=1 P=2 P=3 (0, 1, 3) false W=1 W=2 (1, 1, 1) false Return last sol (1, 1, 1) CSC 5240 - Finite Constraint Domains 31

The Simplex Algorithm - 1 The most widely used optimization algorithm n Optimizes a linear function wrt to linear constraints n Related to Gauss-Jordan elimination n Based on the following observations n u Linear constraints form convex polygons u Optimal solution always occur at a vertex CSC 5240 - Simplification, Optimization and Implication 32

The Simplex Form n An optimization problem (C, f) is in simplex form: u C is the conjunction of CE and CI u CE is a conjunction of linear equations u CI constrains all variables in C to be non- negative u f is a linear expression over variables in C CSC 5240 - Simplification, Optimization and Implication 33

Simplex Form Example n An optimization problem in simplex form n An arbitrary problem can be put in simplex form by replacing u unconstrained var X by new vars and u inequality by new vars and CSC 5240 - Simplification, Optimization and Implication 34

Simplex Solved Form - 1 The simplex method works by repeatedly transforming an optimization problem in “basic feasible solved form” and to another n A simplex optimization problem is in basic feasible solved (bfs) form if: n u The equations are in solved form u Each constant on the right hand side is non- negative u Only parameters occur in the objective CSC 5240 - Simplification, Optimization and Implication 35

Simplex Solved Form - 2 Every problem in bfs form has a corresponding “basic feasible solution” which corresponds to a vertex of the polygon defined by the constraint n A basic feasible solution is obtained by setting each parameter to 0 and each nonparameter to the constant in its equation n CSC 5240 - Simplification, Optimization and Implication 36

Simplex Solved Form Example n An equivalent problem to the previous example in the bfs form n We can read off a solution and its objective value CSC 5240 - Simplification, Optimization and Implication 37

The Simplex Algorithm - 2 The simplex algorithm finds the optimum by repeatedly looking for an “adjacent” basic feasible solved form whose basic feasible solution decreases the value of the objective function until no such neighbour can be found n By adjacent, we mean that the new bfs form can be reached by a single “pivoting” operation n CSC 5240 - Simplification, Optimization and Implication 38

The Simplex Algorithm - 3 starting from a problem in bfs form repeat Choose a variable y with negative coefficient in the obj. func. Find the equation x = b + cy +. . . where c<0 and -b/c is minimal Rewrite this equation with y being the subject y = -b/c + 1/c x +. . . Substitute -b/c + 1/c x +. . . for y in all other eqns and obj. func. until no such variable y exists or no such equation exists if no such y exists then optimum is found else there is no optimum solution CSC 5240 - Simplification, Optimization and Implication 39

Simplex Example Choose variable Z, the 2 nd eqn is only one with neg. coeff Choose variable Y, the first eqn is only one with neg. coeff No variable can be chosen, optimal value 2 is found CSC 5240 - Simplification, Optimization and Implication 40

Another Simplex Example - 1 An equivalent simplex form is: An optimization problem showing contours of the objective function CSC 5240 - Simplification, Optimization and Implication 41

Another Simplex Example - 2 Basic feasible solved form: circle Choose S 3, replace using 2 nd eq Optimal solution: box CSC 5240 - Simplification, Optimization and Implication 42

The Missing Part - 1 How do we get the initial bfs form and its basic feasible solution? n Solve a different simplex problem n u Add artificial variables to make equations in basic feasible solved form u Minimize the sum of the artificial variables u If the sum is zero, we can construct a bfs form for the original problem CSC 5240 - Simplification, Optimization and Implication 43

The Missing Part Example - 1 Original simplex form equations With artificial vars in bfs form: Objective function rewritten: minimize CSC 5240 - Simplification, Optimization and Implication 44

The Missing Part Example - 2 Problem after minimization of objective function Removing the artificial variables, the original problem is equivalent to CSC 5240 - Simplification, Optimization and Implication 45

The Missing Part - 2 Must remember to eliminate the basic variables of the resultant bfs form from the original objective function n Finding a bfs form of the original constraint is exactly a constraint satisfaction problem, and thus the first phase of the simplex method is an efficient constraint solver for linear inequalities n CSC 5240 - Simplification, Optimization and Implication 46

Integer Linear Programming Unlike Simplex, the previous methods don’t use the objective function to direct search n Integer Linear Programming for (C, f) n u use simplex to find a real optimal, u if solution is integer stop u otherwise choose a var x with non-integer opt value d and examine the problems u use the current best solution to constrain prob CSC 5240 - Finite Constraint Domains 47

Integer Linear Programming Eg Smugglers knapsack problem CSC 5240 - Finite Constraint Domains 48

Integer Linear Programming Eg Smugglers knapsack problem CSC 5240 - Finite Constraint Domains 49

Integer Linear Programming Eg Smugglers knapsack problem CSC 5240 - Finite Constraint Domains 50

Integer Linear Programming Eg Smugglers knapsack problem CSC 5240 - Finite Constraint Domains 51

Integer Linear Programming Eg Smugglers knapsack problem CSC 5240 - Finite Constraint Domains 52

Integer Linear Programming Eg Smugglers knapsack problem CSC 5240 - Finite Constraint Domains 53

Integer Linear Programming Eg Smugglers knapsack problem Solution (2, 0, 0) = 30 CSC 5240 - Finite Constraint Domains 54

Integer Linear Programming Eg Smugglers knapsack problem Solution (2, 0, 0) = 30 CSC 5240 - Finite Constraint Domains 55

Integer Linear Programming Eg Smugglers knapsack problem Solution (2, 0, 0) = 30 CSC 5240 - Finite Constraint Domains 56

Integer Linear Programming Eg Smugglers knapsack problem Solution (2, 0, 0) = 30 CSC 5240 - Finite Constraint Domains 57

Integer Linear Programming Eg Smugglers knapsack problem Solution (2, 0, 0) = 30 CSC 5240 - Finite Constraint Domains 58

Integer Linear Programming Eg Smugglers knapsack problem Solution (2, 0, 0) = 30 false CSC 5240 - Finite Constraint Domains 59

Integer Linear Programming Eg Smugglers knapsack problem Solution (2, 0, 0) = 30 false CSC 5240 - Finite Constraint Domains 60

Integer Linear Programming Eg Smugglers knapsack problem Solution (2, 0, 0) = 30 false CSC 5240 - Finite Constraint Domains 61

Integer Linear Programming Eg Smugglers knapsack problem Solution (2, 0, 0) = 30 false CSC 5240 - Finite Constraint Domains 62

Integer Linear Programming Eg Smugglers knapsack problem Solution (2, 0, 0) = 30 false CSC 5240 - Finite Constraint Domains 63

Integer Linear Programming Eg Smugglers knapsack problem Solution (2, 0, 0) = 30 false CSC 5240 - Finite Constraint Domains 64

Integer Linear Programming Eg Smugglers knapsack problem Solution (2, 0, 0) = 30 false CSC 5240 - Finite Constraint Domains 65

Integer Linear Programming Eg Smugglers knapsack problem Solution (2, 0, 0) = 30 false CSC 5240 - Finite Constraint Domains 66

Integer Linear Programming Eg Smugglers knapsack problem Solution (2, 0, 0) = 30 false CSC 5240 - Finite Constraint Domains 67

Integer Linear Programming Eg Smugglers knapsack problem Solution (2, 0, 0) = 30 false CSC 5240 - Finite Constraint Domains 68

Integer Linear Programming Eg Smugglers knapsack problem Solution (2, 0, 0) = 30 false CSC 5240 - Finite Constraint Domains 69

Integer Linear Programming Eg Smugglers knapsack problem Solution (2, 0, 0) = 30 Solution (1, 1, 1) = 32 false CSC 5240 - Finite Constraint Domains 70

Integer Linear Programming Eg Smugglers knapsack problem Solution (2, 0, 0) = 30 Solution (1, 1, 1) = 32 false Worse than best sol false CSC 5240 - Finite Constraint Domains 71

Integer Linear Programming Eg Smugglers knapsack problem Solution (2, 0, 0) = 30 Solution (1, 1, 1) = 32 false Worse than best sol false CSC 5240 - Finite Constraint Domains 72

Integer Linear Programming Eg Smugglers knapsack problem false Solution (2, 0, 0) = 30 Solution (1, 1, 1) = 32 false Worse than best sol false CSC 5240 - Finite Constraint Domains 73

Integer Linear Programming Eg Smugglers knapsack problem false Solution (2, 0, 0) = 30 Solution (1, 1, 1) = 32 false Worse than best sol false CSC 5240 - Finite Constraint Domains 74