Conjugate Gradient Methods for Largescale Unconstrained Optimization Dr
- Slides: 76
Conjugate Gradient Methods for Large-scale Unconstrained Optimization Dr. Neculai Andrei Research Institute for Informatics Bucharest and Academy of Romanian Scientists Ovidius University, Constantza - Romania, March 27, 2008
Contents § § § Problem definition Unconstrained optimization methods Conjugate gradient methods - Classical conjugate gradient algorithms - Hybrid conjugate gradient algorithms - Scaled conjugate gradient algorithms - Modified conjugate gradient algorithms - Parametric conjugate gradient algorithms § Applications
Problem definition - continuously differentiable - gradient is available - n is large - Hessian is unavailable Necessary optimality conditions: Sufficient optimality conditions:
Unconstrained optimization methods Step length Search direction 1) Line search 2) Trust-Region algorithms Quadratic approximation Influences
Step length computation: 1) Armijo rule: 2) Goldstein rule:
3) Wolfe conditions: Implementations: Shanno (1978) Moré - Thuente (1992 -1994)
Proposition Assume that is a descent direction and satisfies the Lipschitz condition for all where on the line segment connecting and is a positive constant. If the line search satisfies the Goldstein conditions then If the line search satisfies the Wolfe conditions, then
Remarks: 1) The Newton method, the quasi-Newton or the limited memory quasi-Newton methods has the ability to accept unity step lengths along the iterations. 2) In conjugate gradient methods the step lengths may differ from 1 in a very unpredictable manner. They can be larger or smaller than 1 depending on how the problem is scaled*. *N. Andrei, (2007) Acceleration of conjugate gradient algorithms for unconstrained optimization. (submitted JOTA)
Methods for Unconstrained Optimization 1) Steepest descent (Cauchy, 1847) 2) Newton 3) Quasi-Newton (Broyden, 1965; and many others)
4) Conjugate Gradient Methods (1952) is known as the conjugate gradient parameter 5) Truncated Newton method (Dembo, et al, 1982) 6) Trust Region methods
7) Conic model method (Davidon, 1980) 8) Metode tensoriale (Schnabel & Frank, 1984)
9) Methods based on systems of Differential Equations Gradient flow Method (Courant, 1942) 10) Direct searching methods Hooke-Jevees (form searching) (1961) Powell (conjugate directions) (1964) Rosenbrock (coordinate system rotation)(1960) Nelder-Mead (rolling the simplex) (1965) Powell –UOBYQA (quadratic approximation) (1994 -2000) N. Andrei, Critica Retiunii Algoritmilor de Optimizare fara Restrictii Editura Academiei Romane, 2008.
Conjugate Gradient Methods Magnus Hestenes (1906 -1991) Eduard Stiefel (1909 -1978)
The prototype of Conjugate Gradient Algorithm Step 1. Select the initial starting point: Step 2. Test a criterion for stopping the iterations. Step 3. Determine the steplength by Wolfe conditions. Step 4. Update the variables: ► Step 5. Compute: Step 6. Compute the search direction: Step 7. Restart. If: then set Step 8. Compute the initial guess: and continue with step 2. set
Convergence Analysis Theorem. Suppose that: 1) the level set 2) the function is bounded, is continuously differentiable, 3) the gradient is Lipschitz continuous, i. e. Consider any conjugate gradient method where: is a descent direction, 1) 2) is obtained by the strong Wolfe line search. If then
Classical conjugate gradient algorithms 1. Hestenes - Stiefel (HS) 2. Polak – Ribière - Polyak (PRP) 3. Liu - Storey (LS) 4. Fletcher - Reeves (FR) 5. Conjugate Descent – Fletcher (CD) 6. Dai – Yuan (DY)
Classical conjugate gradient algorithms Performance Profiles
Classical conjugate gradient algorithms 7. Dai – Liao (DL) 8. Dai – Liao plus (DL+) 9. Andrei - Sufficient Descent Condition (CGSD)* * N. Andrei, A Dai-Yuan conjugate gradient algorithm with sufficient descent and conjugacy conditions for unconstrained optimization. Applied Mathematics Letters, vol. 21, 2008, pp. 165 -171.
Classical conjugate gradient algorithms
Hybrid conjugate gradient algorithms - projections 10. Hybrid Dai - Yuan (h. DY) 11. Hybrid Dai – Yuan zero (h. DYz) 12. Gilbert – Nocedal (GN)
Hybrid conjugate gradient algorithms - projections 13. Hu – Storey (Hu. S) 14. Touati-Ahmed and Storey (Ta. S) 15. Hybrid LS – CD (LS-CD)
Hybrid conjugate gradient algorithms - projections
Hybrid conjugate gradient algorithms - convex combination 16. Convex combination of PRP and DY from conjugacy condition (CCOMB - Andrei) If then N. Andrei, New hybrid conjugate gradient algorithms for unconstrained optimization. Encyclopedia of Optimization, 2 nd Edition, Springer, August 2008, Entry 761.
Hybrid conjugate gradient algorithms - convex combination 17. Convex combination of PRP and DY from Newton direction (NDOMB - Andrei) If then N. Andrei, New hybrid conjugate gradient algorithms as a convex combination of PRP and DY for unconstrained optimization. ICI Technical Report, October 1, 2007. (submitted AML)
Hybrid conjugate gradient algorithms - convex combination
Hybrid conjugate gradient algorithms - convex combination
Hybrid conjugate gradient algorithms - convex combination 18. Convex combination of HS and DY from Newton direction (HYBRID - Andrei) (Secant condition) If then N. Andrei, A hybrid conjugate gradient algorithm for unconstrained optimization as a convex combination of Hestenes-Stiefel and Dai-Yuan. Studies in Informatics and Control, vol. 17, No. 1, March 2008, pp. 55 -70.
Hybrid conjugate gradient algorithms - convex combination 19. Convex combination of HS and DY from Newton direction with modified secant condition (HYBRIDM - Andrei) If then N. Andrei, A hybrid conjugate gradient algorithm with modified secant condition for unconstrained optimization. ICI Technical Report, February 6, 2008 (submitted to Numerical Algorithms)
Hybrid conjugate gradient algorithms - convex combination
Scaled conjugate gradient algorithms N. Andrei, Scaled memoryless BFGS preconditioned conjugate gradient algorithm for unconstrained optimization. Optimization Methods and Software, 22 (2007), pp. 561 -571.
Scaled conjugate gradient algorithms A) Secant Condition B) Modified secant Condition N. Andrei, Accelerated conjugate gradient algorithm with modified secant condition for unconstrained optimization. ICI Technical Report, March 3, 2008. (Submitted, JOTA, 2007)
Scaled conjugate gradient algorithms C) Hessian / vector approximation by finite difference N. Andrei, Accelerated conjugate gradient algorithm with finite difference Hessian / vector product approximation for unconstrained optimization. ICI Technical Report, March 4, 2008 (submitted Math. Programm. )
Scaled conjugate gradient algorithms 20. Birgin – Martínez (BM) 21. Birgin – Martínez plus (BM+) 22. Scaled Polak-Ribière-Polyak (s. PRP)
Scaled conjugate gradient algorithms 23. Scaled Fletcher – Reeves (s. FR) 24. Scaled Hestenes – Stiefel (s. HS)
Scaled conjugate gradient algorithms 25. SCALCG (secant condition) N. Andrei, Scaled conjugate gradient algorithms for unconstrained optimization. Computational Optimization and Applications, vol. 38, no. 3, (2007), pp. 401 -416.
Scaled conjugate gradient algorithms Theorem Suppose that then the direction satisfies the Wolfe conditions is a descent direction. N. Andrei, A scaled BFGS preconditioned conjugate gradient algorithm for unconstrained optimization. Applied Mathematics Letters, 20 (2007), pp. 645 -650.
Scaled conjugate gradient algorithms The Powell restarting procedure : The direction is computed using a double update scheme: for ANDREI, N. , A scaled nonlinear conjugate gradient algorithm for unconstrained optimization. Optimization. A journal of mathematical programming and operations research, DOI, accepted.
Scaled conjugate gradient algorithms N. Andrei, Scaled memoryless BFGS preconditioned conjugate gradient algorithm for unconstrained optimization. Optimization Methods and Software, 22 (2007), pp. 561571.
Scaled conjugate gradient algorithms Lemma: If Lipschitz continuous, then Theorem: For strongly convex functions, with Wolfe line search
Scaled conjugate gradient algorithms
Scaled conjugate gradient algorithms 26. ASCALCG (secant condition) In conjugate gradient methods the step lengths may differ from 1 in a very unpredictable manner. They can be larger or smaller than 1 depending on how the problem is scaled. General Theory of Acceleration N. Andrei, (2007) Acceleration of conjugate gradient algorithms for unconstrained optimization. ICI Technical Report, October 24, 2007. (submitted JOTA, 2007)
Scaled conjugate gradient algorithms
Scaled conjugate gradient algorithms
Scaled conjugate gradient algorithms If is a descent direction, then
Scaled conjugate gradient algorithms computation:
Proposition. Suppose that is a uniformly convex function on the level set and the sufficient descent condition and satisfies where Then the sequence generated by ACG converges linearly to solution to the optimization problem.
Scaled conjugate gradient algorithms N. Andrei, Accelerated scaled memoryless BFGS preconditioned conjugate gradient algorithm for unconstrained optimization. ICI Technical Report, March 10, 2008 (submitted – Numerischke Mathematik, 2008)
Scaled conjugate gradient algorithms
Scaled conjugate gradient algorithms
Modified conjugate gradient algorithms 27. Andrei – Sufficient Descent Condition from PRP (A-prp) 28. Andrei – Sufficient Descent Condition from DY (ACGA) 29. Andrei – Sufficient Descent Condition from DY zero (ACGA+)
Modified conjugate gradient algorithms 30) CG_DESCENT (Hager-Zhang, 2005, 2006)
Modified conjugate gradient algorithms
Modified conjugate gradient algorithms
Modified conjugate gradient algorithms
Modified conjugate gradient algorithms
Modified conjugate gradient algorithms (Slide 30) 31) ACGMSEC For Theorem we get exactly the Perry method. If then N. Andrei, Accelerated conjugate gradient algorithm with modified secant condition for unconstrained optimization. ICI Technical Report, March 3, 2008. (Submitted: Applied Mathematics and Optimization, 2008)
Modified conjugate gradient algorithms
Modified conjugate gradient algorithms
Modified conjugate gradient algorithms 32) ACGHES N. Andrei, Accelerated conjugate gradient algorithm with finite difference Hessian / vector product approximation for unconstrained optimization. ICI Technical Report, March 4, 2008 (submitted Math. Programm. )
Modified conjugate gradient algorithms
Modified conjugate gradient algorithms
Comparisons with other UO methods
Parametric conjugate gradient algorithms 33) Yabe-Takano 34) Yabe-Takano +
Parametric conjugate gradient algorithms 35) Parametric CG suggested by Dai-Yuan 36) Parametric CG suggested by Nazareth
Parametric conjugate gradient algorithms 37) Parametric CG suggested by Dai-Yuan
Applications A 1) Elastic-Plastic Torsion (c=5) (nx=200, ny=200) MINPACK 2 Collection
SCALCG: #iter=445, #fg=584, cpu=8. 49(s) ASCALCG: #iter=240, #fg=269, cpu=6. 93(s) n=40000 variables
A 2) Pressure Distribution in a Journal Bearing (ecc=0. 1 b=10) (nx=200, ny=200)
A 3) Optimal Design with Composite Materials
A 4) Steady State Combustion - Bratu Problem
A 5) Ginzburg-Landau (1 -dimensional) Free Gibbs energy:
n=1000 variables
Thank you !
- Method of steepest descent
- Unimodal function
- One dimensional unconstrained optimization
- Unconstrained multivariable optimization
- Fminsearch
- Optimality conditions for unconstrained optimization
- Constrained and unconstrained optimization in economics
- Balaraman ravindran
- Unconstrained restoration
- Unconstrained demand
- Unconstrained def
- Unconstrained decay
- Unconstrained decay
- Indirect wax pattern
- Biologiska arvet
- Byggprocessen steg för steg
- Presentera för publik crossboss
- Rbk-mätning
- Densitet vatten
- Kung dog 1611
- Tack för att ni lyssnade bild
- Tack för att ni har lyssnat
- Smärtskolan kunskap för livet
- Referatmarkeringar
- Luftstrupen för medicinare
- Vad är en punkthöjd
- Lufttryck formel
- Vilka tal pekar pilarna på
- Magnetsjukhus
- Elektronik för barn
- Personalliggare bygg undantag
- Toppslätskivling dos
- Borra hål för knoppar
- Bris för vuxna
- Mat för idrottare
- Frgar
- Jiddisch
- Ledarskapsteorier
- Datorkunskap för nybörjare
- Indikation för kejsarsnitt på moderns önskan
- Ekologiskt fotavtryck
- Rita perspektiv
- Redogör för vad psykologi är
- Bästa kameran för astrofoto
- Skapa med geometriska former
- Offentlig förvaltning
- Skriven med rytm och rim
- Nyckelkompetenser för livslångt lärande
- Modell för handledningsprocess
- Fredsgudinnan
- Tidbok yrkesförare
- Orubbliga rättigheter
- Ministerstyre för och nackdelar
- Big brother rösta
- Verktyg för automatisering av utbetalningar
- Hur stor skarns är det för ett barn att få cancer
- Tillitsbaserad ledning
- Bat mitza
- Ro i rom pax
- Stig kerman
- Varför kallas perioden 1918-1939 för mellankrigstiden
- Vad är verksamhetsanalys
- Borstål, egenskaper
- Vishnuismen
- Cks
- Urban torhamn
- Inköpsprocessen steg för steg
- Påbyggnader för flakfordon
- A gastrica
- Strategi för svensk viltförvaltning
- Typiska drag för en novell
- Relativ standardavvikelse formel
- Tack för att ni har lyssnat
- Rutin för avvikelsehantering
- Myndigheten för delaktighet
- Läkarutlåtande för livränta
- Iso 22301 utbildning