Nonlinear optimization An overview problems and a guide

Non-linear optimization An overview, problems and a guide

Optimization ¡ Unconstraint non-linear optimization E(w) w 2 w 1

Classes of Methods ¡ ¡ ¡ Linear optimization Constraint <-> unconstraint Gradient based 1 st order, 2 nd order Genetic Algorithms, Evolutionary Strategies Stochastic methods (Simulated Annealing, Tabu Search, …)

Ellipsoid

Rosenbrock-function

Cross-Function

Canyon-function

Step-function

Performance criteria Number of function evaluations ¡ Number of gradient calculation ¡ Time ¡ Number of fails ¡ Number of method params. ¡ Sensitivity of method params. ¡ Accuracy ¡

Methods ¡ Direct methods l l ¡ Gradient based methods l l l ¡ Successive variation Hooke-Jeeves Gradient decent Back-propagation Polak-Ribiere Second order methods l l Newton-Raphson BFGS

Successive Variation

Hooke-Jeeves

Gradient descent

Gradient Decent

Gradient descent

Back-propagation Gradient decent Momentum

Back-propagation

Back-propagation Error E Cycle

Conjugated gradients Qn property

Beam search

Polak-Ribiere Beam search

Polak-Ribiere

Newton-method Q 1 property

BFGS

Comparison: Ellipsoid

Comparison: Cross-Function

Comparison: Rosenbrock-Function

Comparison: Canyon-Function n(E)=8983

Comparison: Step-Function n(E)=2487 n(E)=2448

Decision tree Complexity many MC / SA some GA / ES few Knowledge #minima one Multi-start differentiable no yes elliptic? aligned? no #parameters many NM / LBFGS few HJ / ROS no yes coordinate axis no ROS channels? yes curved along axes SV yes #parameters many PR / LBFGS few BFGS G / PR / BFGS flat QP / RPROP BP