Chapter 2 Single Variable Optimization ShiShang Jang National

Chapter 2 Single Variable Optimization Shi-Shang Jang National Tsing-Hua University Chemical Engineering

Contents n Introduction n Examples n Methods of Regional Elimination n Methods of Polynomial Approximation n Methods that requires derivatives n Conclusion

1. Introduction n A single variable problem is such that n Min f(x), x [a, b] n Given f continuous, the optimality criteria is such that: df/dx=0 at x=x* where x* is either a local minimum or a local maximum or a saddle point (a stationary point)

Property of a single variable function (i) a discrete function (ii) a discontinuous function (iii) a continuous function

2. Example n In a chemical plant, the cost of pipes, their fittings, and pumping are important investment cost. Consider a design of a pipeline L feet long that should carry fluid at the rate of Q gpm. The selection of economic pipe diameter D(in. ) is based on minimizing the annual cost of pipe, pump, and pumping. Suppose the annual cost of a pipeline with a standard carbon steel pipe and a motor-driven centrifugal pump can be expressed as: Formulate the appropriate single-variable optimization problem for designing a pipe of length 1000 ft with a fluid rate of 20 gpm. The diameter of the pipe should be between 0. 25 to 6 in.

The MATLAB Code for Process Design Example n d=linspace(0. 25, 6); n l=1000; q=20; n for i=1: 100 n hp=4. 4 e-8*(l*q^3)/d(i)^5 cost + 1. 92 e-9*(l*q^2. 68) /(d(i)^4. 68); n f(i)=0. 45*l+0. 245*l*d(i)^ 1. 5+3. 25*hp^0. 5+61. 6*h p^0. 925+102; n end diameter

2. Example- Continued n Carry out a single-variable search of f(x)=3 x 2+12/x-5 df/dx=6 x-12/x 2=0

Solutions Methods (1) Analytical Approach n Problem: n Algorithm n Step 1: Set df/dx=0 and solve all stationary points. n Step 2: Select all stationary point x 1, x 2, …, x. N in [a, b]. n Step 3: Compare function values f(x 1), f(x 2), …, f(x. N) and find global minimum.

Example: Polynomial Problem n Example: In the interval of [-2, 4]. and f(3)=37, f(-1)=5 3 is the optimum point. f(x) x x*

Example- Inventory Problem n Example : Inventory Control (Economic Order Quantity) n Inventory quantity each time =Q units n Set up cost or ordering cost =$K n Acquisition cost =$C/unit n Storing cost per unit =$h/year n Demand (constant)= units/time unit, this implies ordering period =T=Q/ n Question: What is optimum ordering amount Q?

Solution Objective function =Total cost per year f(Q)=Cost per cycle*

Problems n Cost function must be explicitly expressed n The derivative of the cost function must also be explicitly written down n The derivative equation must be explicitly solved n In many cases, the derivative equation is solved numerically, such as Newton’s method, it is more convenient to solve the optimization problem numerically.

Problems - Continued x* a b Location of global minimum

The Importance of One-dimensional Problem - The Iterative Optimization Procedure n Optimization is basically performed in a fashion of iterative optimization. We give an initial point x 0, and a direction s, and then perform the following line search: where * is the optimal point for the objective function and satisfies all the constraints. Then we start from x 1 and find the other direction s’, and perform a new line search until the optimum is reached.

The Importance of One-dimensional Problem - The Iterative Optimization Procedure Continued Consider a objective function Min f(X)= x 12+x 22, with an initial point X 0=(-4, -1) and a direction (1, 0), what is the optimum at this direction, i. e. X 1=X 0 + *(1, 0). This is a one -dimensional search for .

The Importance of One-dimensional Problem - The Iterative Optimization Procedure Continued n The problem can be converted into:

Solution Methods (2) – Numerical Approaches: (i) Numerical Solution to Optimality Condition n Example: Determine the minimum of f(x)=(10 x 3+3 x 2+x+5)2 The optimality criteria leads: 2(10 x 3+3 x 2+x+5)(30 x 2+6 x +1)=0 Problem: What is the root of the above equation?

Newton’s Method n Consider a equation to be solved: n f(x)=0; n Step 1: give an initial point x 0 n Step 2: xn+1=xn-f(xn)/f’(xn) n Step 3: is f(xn) small enough? If not go back to step 2 n Step 4: stop

Newton’s Method (example) MATLAB code x 0=10; fx=100; iter=0; ff=[]; xx=[]; while abs(fx)>1. e-5 fx=2*(10*x 0^3+3*x 0^2+x 0+5)*(30*x 0^2+6*x 0+1); ff=[ff; fx]; fxp=2*((30*x 0^2+6*x 0+1)*(30*x 0^2+6*x 0+1)+(10*x 0^3+3*x 0^2+ 5)*(60*x 0+6)); x 0=x 0 -fx/fxp; xx=[xx; x 0]; iter=iter+1; end fx = -8. 4281 e-006 iter = 43 x 0 = --0. 8599

A Numerical Differentiation Approach n Problem: Find df/dx at a point xk n Approach: n Define xk n Find f(xk) n Find f(xk+ xk ) n Approximate df/dx=[(f(xk+ xk )- f(xk))/ xk ]

A Numerical Differentiation Approach -MATLAB code x 0=10; fx=100; iter=0; ff=[]; xx=[]; dx=0. 001; while abs(fx)>1. e-5 fx=2*(10*x 0^3+3*x 0^2+x 0+5)*(30*x 0^2+6*x 0+1); ff=[ff; fx]; xp=x 0+dx; ffp=2*(10*xp^3+3*xp^2+x 0+5)*(30*xp^2+6*xp+1); fxp=(ffp-fx)/dx; x 0=x 0 -fx/fxp; xx=[xx; x 0]; iter=iter+1; end fx = -2. 4021 e-006 iter = 25 x 0 = -0. 8599

Remarks (Numerical Solution to Optimality Condition) n Difficult to formulate the optimality condition n Difficult to solve (multi-solutions, complex number solutions) n Derivative may be very difficult to solve numerically n Function calls are not saved in most cases n New Frontier: Can we simply implement objective function instead of its derivative?

Solution Methods (2) – Numerical Approaches (ii) Reginal Elimination Methods n Theorem: Suppose f is uni-model on the interval a x b, with a minimum at x* (not necessary a stationary point), let x 1, x 2 [a, b] such that a< x 1< x 2<b, then: n n If f(x 1)> f(x 2) x* [x 1, b] If f(x 1)< f(x 2) x* [a, x 2]

Two Phase Approach n Phase I. Bounding Phase: An initial course search that will bound or bracket the optimum n Phase II. Interval Refinement Phase: A finite sequence of interval reductions or refinements to reduce the initial search interval to desired accuracy.

Phase II- Interval Refinement Phase - Interval Halving n Algorithm n n n Step 1: Let , L= b-a, find f(xm) Step 2: Set x 1=a+L/4, x 2=b-L/4. Step 3: Find f(x 1), if f(x 1)<f(xm), then b xm, go to step 1. if f(x 1)>f(xm), continue Step 4: Find f(x 2) § § If f(x 2)<f(xm), then a xm, go to step 1. If f(x 2)>f(xm), then a x 1, b x 2, go to step 1.

Interval Halving f(x) f 2 fm f 1 a x 1 xm x 2 b x

Example: y=(10*x^3+3*x^2+x+5)^2; n n n n n global fun_call a=-3; b=3; l=b-a; xm=(a+b)/2; x 1=(a+xm)/2; x 2=(xm+b)/2; fm=inter_hal_obj(xm); iter=1; fun_call=0; while l>1. e-8 f 1=inter_hal_obj(x 1); if f 1<=fm b=xm; fm=f 1; else f 2=inter_hal_obj(x 2); if f 2<=fm a=xm; fm=f 2; else a=x 1; b=x 2; end xm=(a+b)/2; x 1=(a+xm)/2; x 2=(xm+b)/2; %fm=inter_hal_obj(xm); l=b-a; iter=iter+1; end n b =-0. 8599 n iter = 31 n fun_call =51

Remarks At each stage of the algorithm, exactly half of the search is deleted. n At most two function evaluations are necessary at each iteration. n After n iterations, the initial search interval will be reduced to n According to Krefer, the three point search is most efficient among all equal-interval searches. n

Phase II- Interval Refinement Phase – The Golden Search (Non-equal Interval Search) n Assume that the total length of the region of search =1, two experiments are done at and 1 -. One can compare the above two experiments and hence needs to delete either section between the end point and two trial points. Then one trial point is the new end point, the other is a new comparing point. n Problem: We want the original trial point to be a new trial point, i. e. , . It is possible to solve , =0. 61803…

The Algorithm: Golden-Search Method Step 1: Set L =b-a, =0. 61803…, x 2=a+ L, x 1=a+(1 - )L n Step 2: Find f(x 1), f(x 2), compare If f(x 1)> f(x 2) a=x 1, x 1 x 2, go to step 3. If f(x 1)< f(x 2) b=x 2, x 2 x 1, go to step 3. n Step 3: Set L =b-a, x 2 a+ L, if (i) true x 1 a+(1 - )L, if (ii) true. Go to step 2. n

MATLAB CODING – The Golden Search n function al_opt=goldsec(op 2_func, tol, x 0, d) n b=1; a=0; l=b-a; n tau=0. 61803; x 2=a+tau*l; x 1=a+(1 -tau)*l; n while l>tol n xx 1=x 0+x 1*d; xx 2=x 0+x 2*d; n y 1=feval(op 2_func, xx 1); y 2=feval(op 2_func, xx 2); n if(y 1>=y 2) a=x 1; x 1=x 2; l=b-a; x 2=a+tau*l; n else n b=x 2; x 2=x 1; l=b-a; x 1=a+(1 -tau)*l; n end n al_opt=b;

Example : The Piping Problem n x 0=0. 25; d=6 -0. 25; tol=1. e-6; n al_opt=goldsec('piping', tol, x 0, d); n D=x 0+al_opt*d; n n n function y=obj_piping(D) %D(in) L=1000; %ft Q=20; %gpm hp=4. 4 e-8*(L*Q^3)/(D^5)+1. 92 e-9*(L*Q^2. 68)/(D^4. 68); y=0. 45*L+0. 245*L*D^1. 5+3. 25*(hp)^0. 5+61. 6*(hp)^0. 925+102; n al_opt =0. 1000 n D =0. 8250 n fun_call =29

Remarks n At each stage, only one function evaluation (or one experiment) is needed. n The length of searching is narrowed by a factor of at each iteration, i. e. n Variable transformation technique may be useful for this algorithm, i. e. set the initial length equal to 1.

Remarks - Continued n Define , N = number of experiments, or function evaluations. n Let E = FR(N) Method E=0. 1 E=0. 05 E=0. 01 E=0. 001 I. H. 7 9 14 20 G. S. 6 8 11 16

Solution Methods (2) – Numerical Approaches (iii) Polynomial Approximation Methods – Powell’s Method n Powell’s method is to approximate an objective function by a quadratic function such as f(x)=ax 2+bx+c, then it can be shown the optimum is located at x*=-b/2 a. n Given the above equation we need to do three experiments (function calls) to fit a quadratic function, let the three experiments (function calls) located at: f(x 1), f(x 2), f(x 3) and let’s rewrite the quadratic equation based on the new notation:

Powell’s Method- Continued n The parameters in the previous slide can be found using three experiments: n At the optimum point, it can be derived based on the above three experiments, such that or

Algorithm (Powell’s Method) n n n n Step 1: Given x 0, x, x 1=x 0+ x, , Step 2: Evaluate f(x 0), f(x 1) If f(x 1)> f(x 0), then x 2=x 0 - x If f(x 1)< f(x 0), then x 2=x 0+2 x Step 3: find f(x 2) Step 4: Find Fmin=min (f(x 0), f(x 1), f(x 2)) Xmin= x 0, x 1, x 2, such that f(xmin)= Fmin, i=0, 1, 2. Step 5: Get a 1, a 2 Step 6: Get x*, find f(x*). Step 7: Check if (i) or (ii) Yes, stop. No, continue Step 8: set x 2 x*, x 1 xmin, x 0 one of x 0, x 1, x 2 not xmin Go to step 4.

Powell’s Method – MATLAB code n n n n n function alopt=one_dim_pw(xx, s, op 2_func) dela=0. 005; alp 0=0. 01; alpha(1)=alp 0; alpha(2)=alpha(1)+dela; al=alpha(1); x 1=xx+al*s; y(1)=feval(op 2_func, x 1); al=alpha(2); x 2=xx+s*alpha(2); y(2)=feval(op 2_func, x 2); if(y(2)>=y(1)) alpha(3)=alpha(1)-dela; else alpha(3)=alpha(1)+2*dela; end eps=100; delta=100; while eps>0. 001|delta>0. 001 x 3=xx+s*alpha(3); y(3)=feval(op 2_func, x 3); fmin=min(y);

Powell’s Method – MATLAB code -Continued n n n n n if(fmin==y(1)) almin=alpha(1); i=1; else if(fmin==y(2)) almin=alpha(2); i=2; else almin=alpha(3); i=3; end a 0=y(1); a 1=(y(2)-y(1))/(alpha(2)-alpha(1)); a 2=1/(alpha(3)-alpha(2))*((y(3)-y(1))/(alpha(3)-alpha(1))-(y(2)-y(1))/(alpha(2)-alpha(1))); alopt=(alpha(2)+alpha(1))/2 -a 1/(2*a 2); xxopt=xx+alopt*s; yopt=feval(op 2_func, xxopt); eps=abs(fmin-yopt); delta=abs(alopt-almin); for j=1: 3 if(j~=i) alpha(1)=alpha(j); end alpha(3)=alopt; alpha(2)=almin; x 1=xx+s*alpha(1); x 2=xx+s*alpha(2); y(1)=feval(op 2_func, x 1); y(2)=feval(op 2_func, x 2); end

Example: Piping Design n global fun_call n x 0=0. 25; x_end=6; l=x_end-x 0; n al_opt=one_dim_pw(x 0, l, 'obj_piping') n D=x 0+al_opt*l n fun_call n al_opt = 0. 1000 n D =0. 8250 n fun_call =61

Comparison – Interval_halving (tol=1. e-6) n l= n 6. 8545 e-007 n b= n 0. 8250 n a= n 0. 8250 n iter = n 24 n fun_call = n 63