MSML 604 Intro to Optimization Lecture 2 Convex

MSML 604: Intro. to Optimization Lecture 2 – Convex Functions Richard J. La Spring 2020 Copyright © 2020 Richard J. La

Domain of a Function • Definition: Consider a function. The effective domain or, simply, the domain, of is the set over which the function is well defined: • Examples 1) 2) has

Convex Function (1) • Definition: A function is convex if (a) (b) for all , we have is a convex set and (1) • Definition: A function is strictly convex if it is convex and, for all , we have • Definition: A function is said to be concave (resp. strictly concave) if is convex (resp. strictly convex)

Convex Function (2) • Definition: A function such that is strongly convex if there exists an Is convex, i. e. , – Implies strict convexity • Continuity of convex functions: If is convex, then it is continuous over the interior of. In addition, is Lipschitz on every compact subset , i. e. , there exists a constant such that

Convex Function (3) • Useful fact: A function is convex if and only if it is convex when restricted to any line that intersects its domain – is convex if and only if, for all and , the function , is convex on its domain – Allows us to check whether a function is convex or not by restricting it to a line

Convex Function (4) • Examples of convex functions 1. Affine function: • Both convex and concave 2. Exponential function: 3. Powers of absolute value: 4. Negative entropy: 5. Norms: 6. Spectral norm:

Extended-Valued Function • Useful to extend a convex function be outside – to all of by defining its value to – Allows us to drop the qualifier “for all “ – Easy to verify still satisfies the condition in (1) for convex function – Notation: We will often drop `~’ in the extended-valued function

First-Order Conditions for Convex Functions • Suppose that every point in only if (i) – near Similarly, all is differentiable, i. e. , its gradient exists at , which is assumed open. Then, is convex if and is convex and (ii) for all is the first-order Taylor series approximation of and is a global underestimator is strictly convex if and only if (i) , we have is convex and (ii) for

Second-Order Conditions for Convex Functions • Suppose that is twice differentiable, i. e. , its Hessian exists at every point in , which is assumed open. Then, is convex if and only if (i) is convex and (ii) its Hessian is positive semidefinite at all. If the Hessian , i. e. , positive definite, for all is strictly convex. The converse, however, is not true. • Counterexample – , then

Examples 1. 2. Quadratic function – – Convex if 3. Least-squares – 4. Log-sum-exponential – , where

Sublevel Sets • Definition: The a-sublevel set of a function is defined to be – Useful fact: The a-sublevel sets of a convex function are conv for any value of a. The converse is not in general true. – Counterexample:

Graph and Epigraph • Definition: (i) The graph of a function (ii) The epigraph of a function • Useful fact: A function is defined to be is is convex if and only if is a convex set

Establishing Convexity of a Function • Practical ways to establish convexity of a function 1. Check against the definition – Can be simplified by restricting to a line 2. For twice differentiable functions, show that the Hessian matrix is positive semidefinite at all If all fail … 3. Show that is obtained from simple convex functions by operations that preserve convexity

Operations That Preserve Convexity (1) • Nonnegative weighted sum of convex functions – Suppose convex functions. Then, is convex • Composition with affine function – Suppose. Then, are is convex, with , is convex • Pointwise maximum and supremum – Suppose is convex are convex. Then,

Operations That Preserve Convexity (2) • Composition of two functions – Let 1. Scalar case (k=1) – is convex if a. is convex, b. is convex, – is concave if a. is concave, b. is concave, is non-decreasing, and is convex is non-increasing, and is concave is non-decreasing, and is concave is non-increasing, and is convex . Define

Operations That Preserve Convexity (3) • Composition of two functions – Let . Define 1. Vector case (k>1) – is convex if a. is convex, is non-decreasing in each argument, and are convex b. is convex, is non-increasing in each argument, and are concave – is concave if a. is concave, is non-decreasing in each argument , and are concave b. is concave, is non-increasing in each argument, and are convex

Operations That Preserve Convexity (4) • Minimum and infimum – Suppose is a non-empty convex set. Then, is convex in , provided is convex in for some with and

Operations That Preserve Convexity (5) • Perspective of a function – If the function given by , then the “perspective” of with. This perspective function preserves convexity, i. e. , if is is convex, so is.