MSML 604 Intro to Optimization Lecture 1 Linear

MSML 604: Intro. to Optimization Lecture 1 – Linear Algebra Richard J. La Spring 2020 Copyright © 2020 Richard J. La

Field (1) • Definition: A field F is a set on which two operations – addition and multiplication – are defined so that for each pair of elements , there are unique elements and in F, which satisfy the following conditions for all : a) Commutativity: b) Associativity: c) Identity element: 0 and 1 in F such that d) Inverse element: For each element a in F and each nonzero element b in F, elements c and d in F such that e) Distributivity of multiplication over addition:

Field (2) • Elements of a field are called scalars • Example: numbers with the usual addition and multiplication of real

Vector Space (1) • Definition: A vector space over a field F is a set V along with two operations – (vector) addition and scalar multiplication – which satisfy the following: 1. 2. 3. 4. 5. 6. 7. Closed under vector addition – Vector addition is commutative – Vector addition is associative – There is a `zero vector’ such that Each element has additive inverse , i. e. , The set is closed under scalar multiplication – Addition of scalars distributes over scalar multiplication

Vector Space (2) 8. Scalar multiplication distributes over vector addition 9. Ordinary multiplication of scalars associates with scalar multiplication 10. Multiplication by the scalar 1 is the identity operation – • Elements of a vector space are called vectors

Vector Space (3) • Examples of vector space 1. with element-wise addition and scalar multiplication • Vector: – an ordered finite array/list of numbers • Vector addition • Scalar multiplication

Vector Space (4) • We will assume column vectors in the course unless stated otherwise – Elements, entries, coefficients, or components of a vector – values in the array • Notation – - i-th element of – (Row) vector: • Transpose of a vector –

Vector Space (5) • (Standard) unit vectors: - zero-one vector whose only nonzero element is the i-th element – • Zero and one vectors

Vector Space (6) • Examples of vector space 2. Set of polynomials of degree three or less • Vector addition • Scalar multiplication

Linear Independence • Definition: A set of vectors is said to be linearly independent if no vector can be written as a linear combination of the other vectors • Example 1: Linearly independent • Example 2: Linearly dependent

Spanning Set • Definition: A set of vectors is said to be span the vector space if every element in the vector space can be written as a linear combination of the elements in the set – Called a spanning set • Example of a spanning set for

Basis of a Vector Space (1) • Definition: A basis for a vector space is a set of vectors in , which (a) is linearly independent and (b) spans the vector space, i. e. , any vector in can be written as a linear combination of the vectors in the basis Example: is a basis for

Basis of a Vector Space (2) • Theorem: A subset is a basis for a vector space if and only if every vector in the vector space can be expressed as a linear combination of the elements in the subset in a unique way • Theorem: In any (finite-dimensional) vector space, all bases have the same number of elements

Dimension • Definition: The dimension of a (finite-dimensional) vector space is the number of elements in a basis • Theorem: The number of elements in a linearly independent set in a vector space cannot exceed the dimension of the vector space

Inner Product (1) • Definition: Suppose that V is a vector space over F. An inner product on V is a function that assigns to every ordered pair of vectors v and w in V a scalar in F, denoted by , such that for all v, w and z in V and all r in F, we have 1. 2. 3. 4. • Special case: F = 3. , where the overline denotes complex conjugation

Inner Product (2) • Example: – • Also called the dot product • Definition: A vector space V over F endowed with an inner product is called an inner product space – Example: endowed with the dot product • Definition: Let V be an inner product space. For define the norm (or length) of by , we

Inner Product (3) • Theorem: Let V Be an inner product space over F. Then, for all and , the following hold: a) b) c) if and only if Cauchy-Schwarz inequality: d) Triangle inequality: • Definition: Two vectors and in V are said to be orthogonal (or perpendicular) if

Subspaces • Definition: A subset W of a vector space V (over F) is called a subspace of V if W is a vector space (over F) under the operations of vector addition and scalar multiplication defined on V • Example 1: • Example 2: • Example 3:

Projection (1) • Suppose that is a subspace of V and the vector can be uniquely written as a sum where (called the projection of onto ) . Then, , • If is an orthogonal basis for , then the projection of onto is the sum of the projections of onto the individual basis vectors

Projection (2) • Example:

Matrices • Matrix – a rectangular array of numbers – We will mostly deal with real-numbered matrices – - element of A in the i-th row and j-th column • Notation – – - i-th row of A - j-th column of A

Invertible Matrices and Matrix Inverse • Definition: An matrix is said to be invertible (or nonsingular) if there is another matrix such that Moreover, the matrix is denoted by is called the inverse of matrix and

Matrix Similarity (1) • Two square matrices there is an invertible matrix are said to be similar if such that – Called a similarity transformation (or conjugation) • Example: – Let ,

Matrix Similarity (2) • Both matrices A and B can be viewed as a linear transformations (e. g. , rotation) with different bases – In some cases, it is easier to see what is going on using a new basis using a linear transformation given by A (using the new coordinate) – In the original coordinate (using the old basis), this is equivalent to

Eigenvalues & Eigenvectors (1) • Suppose is an n x n matrix • Definition: A vector is some constant is a right eigenvector of A if there such that – l is called a (right) eigenvalue of A • Definition: A row vector there is some constant is a left eigenvector of A if such that – n is called a (left) eigenvalue of A

Determinant (1) • Determinant of a square matrix – 2 x 2 matrix – 3 x 3 matrix • Define to be the submatrix of and the j-th column after deleting the i-th row

Determinant (2) • Determinant of a square matrix – n x n matrix • Example:

Determinant (3) • Theorem: An n x n matrix determinant is non-zero is invertible if and only if its • Special case of 2 x 2 matrix area of the parallelogram

Determinant and Eigenvalues • Eigenvalues of polynomial given as • Example: are the roots of its characteristic

Determinant (5) • Properties of determinants: – Sum of n eigenvalues of – Product of n eigenvalues of – – – If is invertible, – – For a triangular matrix, is equal to the trace of is equal to the det( ) , i. e. ,

Symmetric Matrices (1) • Definition: A real-valued matrix symmetric if i. e. , is said to be for all , where – Fun fact: The eigenvalues of a symmetric real matrix are real • Definition: A symmetric real matrix is said to be positive semidefinite (or non-negative definite) if • Definition: A symmetric real matrix positive definite if is said to be

Symmetric Matrices (2) • Theorem: The eigenvalues of a positive semidefinite matrix are real and non-negative. Similarly, the eigenvalues of a positive definite matrix are real and positive.

Matrix Decomposition (1) • Definition: A square matrix is said to be diagonalizable if it is similar to a diagonal matrix, i. e. , there is an invertible matrix such that , where is a diagonal matrix • Eigendecomposition (or spectral decomposition): Suppose that has n linearly independent eigenvectors Then, can be factorized as , called the spectral decomposition or eigenvalue decomposition, where consists of the n eigenvectors (as columns), and is a diagonal matrix with eigenvalues along the diagonal

Matrix Decomposition (2) • Theorem: Suppose that is an symmetric real matrix. Then, the n eigenvectors of are orthogonal. Thus, is an orthogonal matrix, i. e. , and the matrix If can be decomposed as is invertible, then

Range and Nullspace (1) • Definition: Let be an matrix. The range of , denoted by , is the set of all vectors in that can be written as linear combinations of the columns of , i. e. , The range is a subspace of. It dimension is called the rank of , denoted rank( ) (or rank ). The nullspace (or kernel) of , denoted by , is the set of all vectors mapped to zero by , i. e. , The nullspace is a subspace of

Range and Nullspace (2) • Comments – rank( ) = # of linearly independent columns in – rank( ) = # of linearly independent rows in • Definition: Matrix is said to be full rank if • Example: – rank( ) ? – Range and nullspace ?

Singular Values and Vectors • Suppose • Definition: The singular values of are the positive square roots of non-zero eigenvalues of either or • Definition: The right singular vectors of are the eigenvectors of associated with non-zero eigenvalues. Similarly, the left singular vectors of are the eigenvectors of associated with non-zero eigenvalues

Matrix Decomposition (3) • Singular value decomposition: Let be an with rank r. Then, can be decomposed as where satisfies , and the singular values of • Columns of matrix , , where consist of left singular vectors of consist of right singular vectors of matrix satisfies are

Pseudoinverse • Let with be the singular value decomposition of • Definition: The pseudoinverse or Moore-Penrose inverse of is defined to be – If , then – If is square and nonsingular, then