# BASIC CONCEPTS IN OPTIMIZATION PART II Continuous Unconstrained

• Slides: 45

BASIC CONCEPTS IN OPTIMIZATION: PART II: Continuous & Unconstrained Important concepts for the optimization of systems with continuous variables and non-linear equations. Since we will limit the topic to unconstrained problems, we will concentrate on the OBJECTIVE FUNCTION. • Optimality Conditions for Single Variable • Optimality Conditions for Multivariable Variable • Revisit Convexity and Its Importance Introducción a la Optimización de procesos químicos. Curso 2005/2006

BASIC CONCEPTS IN OPTIMIZATION: PART II: Continuous & Unconstrained Wait a minute. No problem is unconstrained; so, why do we need to know this? • Unconstrained problems sometimes the solution doesn’t involve constraints • Used in methods for constrained problems Introducción a la Optimización de procesos químicos. Curso 2005/2006

BUILDING EXPERIENCE IN OPTIMIZATION CLASS EXERCISE: The reactor is isothermal and the reaction kinetics are first order. Is this system linear or non-linear? • What must we define before defining an optimum? - The goal is to maximize CB in the effluent at S-S - You can adjust only the flow rate of feed This is an isothermal CFSTR with the reaction: A B C You can only adjust F Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM? For LP, the optimum is at a corner point. For NLP the optimum is located ……. ? Optimum Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM: Single variable? We will start with a single-variable system and then generalize to multiple variable. We will not yet include constraints. The general definition of a minimum of f(x) is x* is a minimum if f(x*) f(x* + x) for small x We want to apply this concept, but we need to determine specific criteria that test for conformance to the statement in the box above. Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM: Single variable? Necessary Condition for a single-variable system: [df(x)/dx]x* = 0 Let’s look at the definition of a derivative, which is continuous If this exists and f(x*) (f(x*+ x), then Why isn’t this sufficient for a minimum? Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM: Single variable? Necessary Condition for a single-variable system: [df(x)/dx]x* = 0 (a) (b) (d) (e) (c) Where is the derivative zero? Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM: Single variable? Sufficient condition: A function with f’(x*)=0 has f’’(x*) = …= fn-1(x*) = 0 (the next n-1 derivatives = zero) has for n = even fn(x*) > 0 (the nth derivative at x* > 0 ) Approximate the function with a Taylor Series. 0 0 Remainder ( 0 h 1) Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM : Single variable? Sufficient condition: A function with f’(x*)=0 has f’’(x*) = …= fn-1(x*) = 0 (2 nd to n-1 derivatives = zero) has for n = even fn(x*) > 0 (the nth derivative at x* > 0 ) Rearrange the result. For n = even, ( x)n > 0; when nth derivative is positive, the condition for a minimum is satisfied! Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM : Single variable? Necessary & Sufficient Conditions: [df(x)/dx]x* = 0 ; d 2 f(x*)/dx 2 > 0 (a) (b) (d) (e) (c) Which satisfy the necessary & sufficient? Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM : Single variable? Let’s look at the following examples. f 1 = 3 + 2 x + 5 x 2 df 1/dx = 2 + 10 x = 0 : x = -. 20 d 2 f 1/dx 2 = 10 > 0 at x = -. 20 Therefore, the function has a local minimum at x = x* = -. 20 f 1 = 3 + 2 x - 5 x 2 df 1/dx = 2 - 10 x = 0 : x =. 20 d 2 f 1/dx 2 = -10 < 0 at x =. 20 Therefore, the function has a local maximum at x = x* =. 20 Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM: Single variable? Necessary & Sufficient Conditions: [df(x)/dx]x* = 0 ; d 2 f(x*)/dx 2 > 0 • Are these results consistent with the methods you have learned previously? • What do we conclude if n = odd? • What type of extremum occurs for f(x) = x 4? Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM: Single variable? Necessary & Sufficient Conditions: [df(x)/dx]x* = 0 ; d 2 f(x*)/dx 2 > 0 • Are these results consistent with the methods you have learned previously? Hopefully, these are the rules that you learned in first-year calculus! • What do we conclude if n = odd? The sign of the remainder depends on the sign of x. This is not a local minimum. It is termed a saddle point. Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM: Single variable? Necessary & Sufficient Conditions: [df(x)/dx]x* = 0 ; d 2 f(x*)/dx 2 > 0 • What type of extremum occurs for f(x) = x 4? Therefore, the extreme point is a minimum! Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM: Multivariable? Necessary: Let’s extend these results to multivariable systems, with x a vector of dimension n. Necessary condition: The proof is similar to the single-variable case. We call these equations the “stationarity conditions”. Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? The necessary condition for unconstrained optimization of a multivariable system is often stated as the following. The gradient equaling zero is the stationarity condition. Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? Sufficient: Let’s extend these results to multivariable systems, with x a vector of dimension n. We will restrict sufficient conditions to second derivatives. The first and second differential is defined as Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? These terms can be used in the expression for a Taylor series to determine the sufficient condition. 0 Remainder ( 0 h 1) The condition for a minimum is satisfied when the remainder is positive. Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? H = the Hessian of second derivatives It is symmetric. Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? For a minimum, the right hand side is positive for any non-zero values of the vector x. How can we tell? We need to evaluate an infinite number of values of x! Let’s try a little mathematics to improve the situation Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? We will consider a two-dimensional system. We start by defining a new vector of variables, w. x 1 w 2 x 2 Can we define the b’s to make the test for optimality easier? Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? The optimality test would be easy if the hessian were diagonal. How can we determine the b’s to give this nice, diagonal hessian matrix? Then, If, Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? The answer is determined from the eigenvalues and eigenvectors of the hessian matrix!!! When we prove that the function f(w) has a minimum at w* from 1 > 0 and 2 > 0 we also prove that the function f(x) has a minimum at x*! Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? A schematic of what we did. The coordinates are rotated to express the quadratic as the sum of variables squared times eigenvalues. w 1 Clearly, the remainder term must only increase if all i are positive. w 2 Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? Positive Definite: A matrix is positive definite if all values of its eigenvalues ( ) are positive. Eigenvalues are the solution to the following equation, with H evaluated at x*. | H - I | = 0 What is the form of this equation? How many solutions are there? Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? The following two conditions are necessary & sufficient at x* The gradient is zero The Hessian is positive definite • Some good news - We do not typically perform these calculations to test problems • But, these concepts are used in many solution methods for non-linear optimization. Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? This is a “nice” objective function, which is convex and symmetric. Local derivative information will direct us toward the minimum. All eigenvalues are positive. Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? This is an objective function with a ridge. We will find the valley quickly; then, we will search the ridge with little success. One eigenvalue is near zero. Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? This objective function has a saddle point, which has a minimum in one direction and maximum in another direction. Derivative information will not direct us well. One eigenvalue is positive, and another is negative. Introducción a la Optimización de procesos químicos. Curso 2005/2006

WHAT DEFINES THE LOCATION OF AN OPTIMUM : Multivariable? What’s going on here? 1 2 3 4 What is the hessian for these stationary points Introducción a la Optimización de procesos químicos. Curso 2005/2006

CONVEXITY: AN IMPORTANT PROPERTY IN OPTIMIZATION Convexity and the objective function. A function of x (a vector) is convex if the following is true. For points x 1 and x 2 and 0 1. Is this function convex (over the region in the figure)? f(x) x Introducción a la Optimización de procesos químicos. Curso 2005/2006

CONVEXITY: AN IMPORTANT PROPERTY IN OPTIMIZATION Convexity and the objective function. A function of x (a vector) is convex over a region if the following is true over the region. Gradient Test: Hessian Test: The function is convex if its Hessian matrix is positive definite positive Introducción a la Optimización de procesos químicos. Curso 2005/2006

CONVEXITY: AN IMPORTANT PROPERTY IN OPTIMIZATION Any local minimum of a convex function (over an unconstrained region) is a global minimum! Introducción a la Optimización de procesos químicos. Curso 2005/2006

BASIC CONCEPTS IN OPTIMIZATION: PART II: Continuous & Unconstrained Conclusions on OBJECTIVE FUNCTION properties • Opt. Conditions for Single Variable • Opt. Conditions for Multivariable Variable • Convexity and Its Importance When is local = global optimum? Basis of many optimization algorithms and tests for convergence We seek to formulate our models to yield a convex programming problem Introducción a la Optimización de procesos químicos. Curso 2005/2006

OPTIMIZATION BASICS II - WORKSHOP #1 We covered the conditions for optimality and convexity in this section. They seemed similar. • What is difference between suff. condition for optimality and convexity? • Why is convexity important? Introducción a la Optimización de procesos químicos. Curso 2005/2006

OPTIMIZATION BASICS II - WORKSHOP #2 Since convexity is important, let’s evaluate convexity for a very important function. Is the following function convex or concave? with ci constants Introducción a la Optimización de procesos químicos. Curso 2005/2006

OPTIMIZATION BASICS II - WORKSHOP #3 The statement below is very important. Prove the statement. Hint: Consider directions of improvement for convex and non-convex functions. Any local minimum of a convex function (over an unconstrained region) is a global minimum! Introducción a la Optimización de procesos químicos. Curso 2005/2006

OPTIMIZATION BASICS II - WORKSHOP #4 All convex functions have a unique minimum, i. e. , they are unimodal. Determine whether all unimodal functions are convex f(x) x Introducción a la Optimización de procesos químicos. Curso 2005/2006

OPTIMIZATION BASICS II - WORKSHOP #5 We seek a global, rather than a local, optimum. • Define a global optimum in words • Determine a mathematical test for the global optimum. • Discuss how you would find a global optimum. Introducción a la Optimización de procesos químicos. Curso 2005/2006

OPTIMIZATION BASICS II - WORKSHOP #6 The objective function is often the sum of several functions, for example, costs, revenues, taxes, and so forth. Determine if the following are a convex functions, when each term [gi(x)] is convex individually. Introducción a la Optimización de procesos químicos. Curso 2005/2006

OPTIMIZATION BASICS II - WORKSHOP #7 A function is convex if its Hessian matrix is positive definite over the range of the variable x Positive definite One way to determine if a matrix (the hessian) is positive definite is to evaluate the determinants of its principle minors. If they are positive, the matrix is positive definite. The principle minors are the sub-matrices formed by eliminating n-k columns and rows, with k = 0 to n-1. Apply this approach to the following functions. Introducción a la Optimización de procesos químicos. Curso 2005/2006

OPTIMIZATION BASICS II - WORKSHOP #7 A function is convex if its Hessian matrix is positive definite over the range of the variable x Positive definite Introducción a la Optimización de procesos químicos. Curso 2005/2006

OPTIMIZATION BASICS II - WORKSHOP #7 SOLUTION Therefore, the function is convex Introducción a la Optimización de procesos químicos. Curso 2005/2006

OPTIMIZATION BASICS II - WORKSHOP #7 SOLUTION Therefore, the function is not convex Introducción a la Optimización de procesos químicos. Curso 2005/2006

OPTIMIZATION BASICS II - WORKSHOP #7 SOLUTION Therefore, the function is convex Introducción a la Optimización de procesos químicos. Curso 2005/2006