Finding Eigenvalues and Eigenvectors What is really important

Finding Eigenvalues and Eigenvectors What is really important?

Approaches l Find the characteristic polynomial – l Find the largest or smallest eigenvalue – – l Direct Power Method Inverse Power Method Find all the eigenvalues – – 2 Leverrier’s Method Jacobi’s Method Householder’s Method QR Method Danislevsky’s Method 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Finding the Characteristic Polynomial l Reduces to finding the coefficients of the polynomial for the matrix A Recall |l. I-A| = ln+anln-1+an-1 ln-2+…+a 2 l 1+a 1 Leverrier’s Method – – Set Bn = A and an = -trace(Bn) For k = (n-1) down to 1 compute l l 3 Bk = A (Bk+1 + ak+1 I) ak = - trace(Bk)/(n – k + 1) 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Vectors that Span a Space and Linear Combinations of Vectors l l 4 Given a set of vectors v 1, v 2, …, vn The vectors are said span a space V, if given any vector x ε V, there exist constants c 1, c 2, …, cn so that c 1 v 1 + c 2 v 2 +…+ cnvn = x and x is called a linear combination of the vi 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Linear Independence and a Basis l l l 5 Given a set of vectors v 1, v 2, …, vn and constants c 1, c 2, …, cn The vectors are linearly independent if the only solution to c 1 v 1 + c 2 v 2 +…+ cnvn = 0 (the zero vector) is c 1= c 2=…=cn = 0 A linearly independent, spanning set is called a basis 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Example 1: The Standard Basis l l 6 Consider the vectors v 1 = <1, 0, 0>, v 2 = <0, 1, 0>, and v 3 = <0, 0, 1> Clearly, c 1 v 1 + c 2 v 2 + c 3 v 3 = 0 c 1= c 2= c 3 = 0 Any vector <x, y, z> can be written as a linear combination of v 1, v 2, and v 3 as <x, y, z> = x v 1 + y v 2 + z v 3 The collection {v 1, v 2, v 3} is a basis for R 3; indeed, it is the standard basis and is usually denoted with vector names i, j, and k, respectively. 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Another Definition and Some Notation l l l 7 Assume that the eigenvalues for an n x n matrix A can be ordered such that |l 1| > |l 2| ≥ |l 3| ≥ … ≥ |ln-2| ≥ |ln-1| > |ln| Then l 1 is the dominant eigenvalue and |l 1| is the spectral radius of A, denoted r(A) The ith eigenvector will be denoted using superscripts as xi, subscripts being reserved for the components of x 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Power Methods: The Direct Method l l 8 Assume an n x n matrix A has n linearly independent eigenvectors e 1, e 2, …, en ordered by decreasing eigenvalues |l 1| > |l 2| ≥ |l 3| ≥ … ≥ |ln-2| ≥ |ln-1| > |ln| Given any vector y 0 ≠ 0, there exist constants ci, i = 1, …, n, such that y 0 = c 1 e 1 + c 2 e 2 +…+ cnen 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

The Direct Method (continued) l l l 9 If y 0 is not orthogonal to e 1, i. e. , (y 0)Te 1≠ 0, y 1 = Ay 0 = A(c 1 e 1 + c 2 e 2 +…+ cnen) = Ac 1 e 1 + Ac 2 e 2 +…+ Acnen = c 1 Ae 1 + c 2 Ae 2 +…+ cn. Aen Can you simplify the previous line? 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

The Direct Method (continued) l l l 10 If y 0 is not orthogonal to e 1, i. e. , (y 0)Te 1≠ 0, y 1 = Ay 0 = A(c 1 e 1 + c 2 e 2 +…+ cnen) = Ac 1 e 1 + Ac 2 e 2 +…+ Acnen = c 1 Ae 1 + c 2 Ae 2 +…+ cn. Aen y 1 = c 1 l 1 e 1 + c 2 l 2 e 2 +…+ cnlnen What is y 2 = Ay 1? 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

The Direct Method (continued) 11 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

The Direct Method (continued) 12 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

The Direct Method (continued) 13 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

The Direct Method (continued) l l 14 Note: any nonzero multiple of an eigenvector is also an eigenvector Why? Suppose e is an eigenvector of A, i. e. , Ae=le and c 0 is a scalar such that x = ce Ax = A(ce) = c (Ae) = c (le) = l (ce) = lx 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

The Direct Method (continued) 15 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Direct Method (continued) l l 16 Given an eigenvector e for the matrix A We have Ae = le and e 0, so e. Te 0 (a scalar) Thus, e. TAe = e. Tle = le. Te 0 So l = (e. TAe) / (e. Te) 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Direct Method (completed) 17 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Direct Method Algorithm 18 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Jacobi’s Method l l l 19 Requires a symmetric matrix May take numerous iterations to converge Also requires repeated evaluation of the arctan function Isn’t there a better way? Yes, but we need to build some tools. 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

What Householder’s Method Does l l l 20 Preprocesses a matrix A to produce an upper -Hessenberg form B The eigenvalues of B are related to the eigenvalues of A by a linear transformation Typically, the eigenvalues of B are easier to obtain because the transformation simplifies computation 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Definition: Upper-Hessenberg Form l 21 A matrix B is said to be in upper-Hessenberg form if it has the following structure: 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

A Useful Matrix Construction l l l Assume an n x 1 vector u 0 Consider the matrix P(u) defined by P(u) = I – 2(uu. T)/(u. Tu) Where – – – 22 I is the n x n identity matrix (uu. T) is an n x n matrix, the outer product of u with its transpose (u. Tu) here denotes the trace of a 1 x 1 matrix and is the inner or dot product 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Properties of P(u) l P 2(u) = I – – l P-1(u) = P(u) – l – 23 P(u) is its own inverse PT(u) = P(u) – l The notation here P 2(u) = P(u) * P(u) Can you show that P 2(u) = I? P(u) is its own transpose Why? P(u) is an orthogonal matrix 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Householder’s Algorithm l l Set Q = I, where I is an n x n identity matrix For k = 1 to n-2 – – – l 24 a = sgn(Ak+1, k)sqrt((Ak+1, k)2+ (Ak+2, k)2+…+ (An, k)2) u. T = [0, 0, …, Ak+1, k+ a, Ak+2, k, …, An, k] P = I – 2(uu. T)/(u. Tu) Q = QP A = PAP Set B = A 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Example 25 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Example 26 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Example 27 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Example 28 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Example 29 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Example 30 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Example 31 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

How Does It Work? l l Householder’s algorithm uses a sequence of similarity transformations B = P(uk) A P(uk) to create zeros below the first sub-diagonal uk=[0, 0, …, Ak+1, k+ a, Ak+2, k, …, An, k]T a = sgn(Ak+1, k)sqrt((Ak+1, k)2+ (Ak+2, k)2+…+ (An, k)2) By definition, – – 32 sgn(x) = 1, if x≥ 0 and sgn(x) = -1, if x<0 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

How Does It Work? (continued) l The matrix Q is orthogonal – – – l B = QT A Q hence Q B = Q QT A Q = A Q – 33 the matrices P are orthogonal Q is a product of the matrices P The product of orthogonal matrices is an orthogonal matrix Q QT = I (by the orthogonality of Q) 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

How Does It Work? (continued) l l l If ek is an eigenvector of B with eigenvalue lk, then B ek = lk ek Since Q B = A Q, A (Q ek) = Q (B ek) = Q (lk ek) = lk (Q ek) Note from this: – – 34 lk is an eigenvalue of A Q ek is the corresponding eigenvector of A 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

The QR Method: Start-up l Given a matrix A Apply Householder’s Algorithm to obtain a matrix B in upper-Hessenberg form l Select e>0 and m>0 l – – 35 e is a acceptable proximity to zero for subdiagonal elements m is an iteration limit 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

The QR Method: Main Loop 36 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

The QR Method: Finding The l’s 37 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Details Of The Eigenvalue Formulae 38 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Details Of The Eigenvalue Formulae 39 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Finding Roots of Polynomials l l l 40 Every n x n matrix has a monic characteristic polynomial of degree n Every monic polynomial of degree n has a corresponding n x n matrix for which it is the characteristic polynomial Thus, polynomial root finding is equivalent to finding eigenvalues 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Example Please!? !? !? l 41 Consider the monic polynomial of degree n f(x) = a 1 +a 2 x+a 3 x 2+…+anxn-1 +xn and the companion matrix 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Find The Eigenvalues of the Companion Matrix 42 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D

Find The Eigenvalues of the Companion Matrix 43 11/23/2020 DRAFT Copyright, Gene A Tagliarini, Ph. D