CS 5321 Numerical Optimization 12 1072020 Theory of

  • Slides: 12
Download presentation
CS 5321 Numerical Optimization 12 10/7/2020 Theory of Constrained Optimization 1

CS 5321 Numerical Optimization 12 10/7/2020 Theory of Constrained Optimization 1

General form min f (x) subject t o x 2 R n l l

General form min f (x) subject t o x 2 R n l l l ½ ci (x) = 0 i 2 E ci (x) ¸ 0 i 2 I E, I are index sets for equality and inequality constraints Feasible set ={x |ci(x)=0, i E; ci(x) 0, i I} Outline l l l 10/7/2020 Equality and inequality constraints Lagrange multipliers Linear independent constraint qualification First/second order optimality conditions Duality 2

A single equality constraint l min f (x) = x 1 + x 2

A single equality constraint l min f (x) = x 1 + x 2 An example x c 1 = x 21 + x 22 = 2 subject toµ ¶ µ r f (x) = l l 1 1 r c 1 (x) = The optimal solution is at Optimal condition: l 10/7/2020 µ 2 x 1 2 x 2 x ¤ 1 x ¤ 2 ¶ ¶ µ = ¡ 1 ¶ r f (x ¤ ) = ¸ ¤ r c 1 (x ¤ ) *=− 1/2 in the example 3

A single inequality constraint l l min f (x) = x 1 + x

A single inequality constraint l l min f (x) = x 1 + x 2 Example: x c 1 : 2 ¡ x 21 ¡ x 22 ¸ 0 subject to Case 1: the solution is inside c 1. ¤ c 1 s f r f (x ) = 0 Unconstrained optimization l Case 2: the solution½is on the boundary of c. 1 ¤ = c 1 (x ) 0 r f (x ¤ ) = ¸ ¤ r c 1 (x ¤ ) l Equality constraint l l Complementarity condition: *c 1(x*)=0 l 10/7/2020 Let *=0 in case 1. 4

Lagrangian function l Define L (x, ) = f (x) − c 1(x) l

Lagrangian function l Define L (x, ) = f (x) − c 1(x) l l L (x, ) = −c 1(x) x. L (x, ) = f (x) − c 1(x) ½ conditions of inequality constraint The optimality ¤ ¤ = ¸ 1 c 1 (x ) r f (x ¤ ) = 0 ¸ ¤ 1 r c 1 (x ¤ ) L (x, ) is called the Lagrangian function; is called the Lagrangian multiplier 10/7/2020 5

Lagrangian multiplier l l At optimal solution x*, f (x*) = 1 c 1(x*)

Lagrangian multiplier l l At optimal solution x*, f (x*) = 1 c 1(x*) If c 1(. ) changes unit, f (. ) changes 1 unit. l l l 10/7/2020 1 is the change of f per unit change of c 1. 1 means the sensitivity of f to ci. 1 is called the shadow price or the dual variable. 6

Constraint qualification l l Active set A(x) = E {i I | ci(x)=0} Linearly

Constraint qualification l l Active set A(x) = E {i I | ci(x)=0} Linearly Independent Constraint Qualification l l The gradients of constraints in the active set { ci(x) | i A(x)} are linearly independent A point is called regular if it satisfies LICQ. l Other constraint qualifications may be used. c 1 (x) c 2 (x) x 10/7/2020 = = = 1 ¡ x 21 ¡ (x 2 ¡ 1) 2 ¸ 0 ¡ x 2 ¸ 0 (0; 0) 7

First order conditions l A regular point that is a minimizer of the constrained

First order conditions l A regular point that is a minimizer of the constrained problem must satisfies the KKT condition (Karush-Kuhn-Tucker) r x L (x¤; ¸ ¤) ci (x¤) ¸¤ ¸ ¤ci (x¤) l 10/7/2020 = = ¸ ¸ = 0 0 0 for all i 2 E i 2 I i The last one is called complementarity condition 8

Second order conditions l The critical 8 cone C(x*, *) is a set of

Second order conditions l The critical 8 cone C(x*, *) is a set of vectors that w 2 C(x ¤ ; ¸ ¤ ) , l < r ci (x ¤ ) T w = 0 i 2 E r ci (x ¤ ) T w = 0 i 2 I A (x ¤ ) wit h ¸ ¤i > 0 : r ci (x ¤ ) T w ¸ 0 i 2 I A (x ¤ ) wit h ¸ ¤i = 0 Suppose x* is a solution to a constrained optimization problem and * satisfies KKT conditions. For all w C (x*, *), w. T r 10/7/2020 2 L ( x ¤ ; ¸ ¤ )w xx ¸ 0 9

Projected Hessian l l l Let A(x*)=[ ci(x*)]i A(x*): A(x*) is the active set.

Projected Hessian l l l Let A(x*)=[ ci(x*)]i A(x*): A(x*) is the active set. It can be shown that C(x*, *) = Null A(x*) can via QR µ be computed ¶ µ decomposition ¶ ¡ ¢ A(x ¤ ) T l l l = Q R 0 = span{Q 2} = Null A(x*) Q 1 Q 2 H = QT 2 r R 0 = Q 1 R 2 L (x ¤ ; ¸ ¤ )Q 2 xx Define projected Hessian The second order optimality condition is that H is positive semidefinite. 10/7/2020 10

Duality: An example min f (x) = 3 x 1 + 4 x 2

Duality: An example min f (x) = 3 x 1 + 4 x 2 x¸ 0 subject to c 1 c 2 : x 1 + x 2 ¸ 3 : 2 x 1 + 0: 5 x 2 ¸ 2 Find a lower bound for f (x) via the constraints l l Ex: f (x) >3 c 1 3 x 1+4 x 2 >3 x 1+3 x 2 9 Ex: f (x) >2 c 1 +0. 5 c 2 3 x 1+4 x 2 >3 x 1+1. 25 x 2 7 What is the maximum lower bound for f (x)? y 1 + 2 y 2 · 3 max g(y) = 3 y 1 + 2 y 2 y¸ 0 10/7/2020 s. t. y 1 + 0: 5 y 2 · 4 11

Dual problem Primal problem l 1. 2. min f (x) s. t. ci (x)

Dual problem Primal problem l 1. 2. min f (x) s. t. ci (x) ¸ 0 x No equality constraints. Inequality constraints ci are concave. (−ci are convex) Let c(x)=[c 1(x), …cm(x)]T. The Lagrangian is L (x, ) = f (x) − Tc(x), Rm. The dual object function is q( )=infx. L (x, ) l l l If L (: , ) is convex, infx. L (x, ) is at x. L (x, )=0 The dual problem is l l 10/7/2020 max q(¸ ) s. t. ¸ ¸ 0 ¸ If L (: , ) is convex, additional constraint is x. L(x, )=0 12