 # CSE 203 B Convex Optimization Chapter 9 Unconstrained

• Slides: 23 CSE 203 B Convex Optimization: Chapter 9: Unconstrained Minimization CK Cheng Dept. of Computer Science and Engineering University of California, San Diego 1 Chapter 9 Unconstrained Minimization • • • Introduction Taylor’s Expansion & Bounds Descent Methods Newton Method Summary 2 Introduction 3 Taylor’s Expansion & Bounds: Scalar case 4 Taylor’s Expansion & Bounds: Example 5 Taylor’s Expansion & Bounds: Bounds 4 1 3 2 1 6 Taylor’s Expansion & Bounds: Bounds 1 7 Taylor’s Expansion & Bounds: Bounds 2 8 3 Taylor’s Expansion & Bounds: Bounds 9 Taylor’s Expansion & Bounds: Bounds 10 Taylor’s Expansion & Bounds 11 II. Descent Methods 12 II. Descent Methods: Example 13 II. Descent Methods: Descent for various norms 14 II. Descent Methods: Descent for quadratic norm 15 II. Descent Methods: Descent for quadratic norm 16 II. Descent Methods: Descent for L 1 norm 17 Gradient descent method: Convergence analysis 18 Gradient descent method : Convergence analysis 19 Newton Step 20 Newton Method : Convergence analysis 21 Newton Method: Affine Invariant 22 Summary 1. Gradient Descent Method: (minimization solution) 1. Vector operations per iteration 2. Linear convergence rate 2. Newton’s Method: (equality solution) 1. Matrix operations per iteration 2. Quadratic convergence rate (near the solution) 3. Gradient Descent Method Variations: 1. Conjugate gradient method 2. Nesterov gradient descent method 3. Quasi-Newton method 23