CSE 203 B Convex Optimization Chapter 9 Unconstrained

CSE 203 B Convex Optimization: Chapter 9: Unconstrained Minimization CK Cheng Dept. of Computer Science and Engineering University of California, San Diego 1

Chapter 9 Unconstrained Minimization • • • Introduction Taylor’s Expansion & Bounds Descent Methods Newton Method Summary 2

Introduction 3

Taylor’s Expansion & Bounds: Scalar case 4

Taylor’s Expansion & Bounds: Example 5

Taylor’s Expansion & Bounds: Bounds 4 1 3 2 1 6

Taylor’s Expansion & Bounds: Bounds 1 7

Taylor’s Expansion & Bounds: Bounds 2 8

3 Taylor’s Expansion & Bounds: Bounds 9

Taylor’s Expansion & Bounds: Bounds 10

Taylor’s Expansion & Bounds 11

II. Descent Methods 12

II. Descent Methods: Example 13

II. Descent Methods: Descent for various norms 14

II. Descent Methods: Descent for quadratic norm 15

II. Descent Methods: Descent for quadratic norm 16

II. Descent Methods: Descent for L 1 norm 17

Gradient descent method: Convergence analysis 18

Gradient descent method : Convergence analysis 19

Newton Step 20

Newton Method : Convergence analysis 21

Newton Method: Affine Invariant 22

Summary 1. Gradient Descent Method: (minimization solution) 1. Vector operations per iteration 2. Linear convergence rate 2. Newton’s Method: (equality solution) 1. Matrix operations per iteration 2. Quadratic convergence rate (near the solution) 3. Gradient Descent Method Variations: 1. Conjugate gradient method 2. Nesterov gradient descent method 3. Quasi-Newton method 23