Chapter 7 Iterative Techniques in Matrix Algebra Solve

Chapter 7 Iterative Techniques in Matrix Algebra Solve Idea Similar to the fixed-point iterations for solving f(x) = 0 …… Equivalently transform into. Then start the iteration from an initial guess and obtain the (convergent) sequence. The accuracy can be controlled by number of iterations. Iterative techniques are practically used to solve sparse linear systems of equations. What to analyze? How to design an iterative scheme? Under what conditions that the sequence will converge? How fast can a method converge? Error estimation? 1/19

Chapter 7 Iterative Techniques in Matrix Algebra -- Norms of Vectors and Matrices 7. 1 Norms of Vectors and Matrices Vector Norms Definition: A vector norm on Rn is a function, || · ||, from Rn into R and C: with the following properties for all /* positive definite */ /* homogeneous */ /* triangle inequality */ Some popularly used norms: || xv||1 = n |x i =1 i | v || x || = max | x i | 1 i n 2/19 v || x || 2 = Euclidean norm n i=1 | xi | Note: 2 v || x || p = n i =1 1/ p | xi |p

Chapter 7 Iterative Techniques in Matrix Algebra -- Norms of Vectors and Matrices Definition: A sequence of vectors in Rn is said to converge to with respect to the norm || · || if, given any > 0, there exists an integer N( ) such that for all k N( ). Theorem: The sequence of vectors converges to respect to the norm || · || if and only if 2, …, n. in Rn with for each i = 1, Definition: If there exist positive constants C 1 and C 2 such that , then || · ||A and || · ||B are said to be equivalent. Theorem: All the vector norms on Rn are equivalent. HW: Read the proof of Theorem 7. 7 on p. 423 3/19

Chapter 7 Iterative Techniques in Matrix Algebra -- Norms of Vectors and Matrices Matrix Norms Definition: A matrix norm on the set of all n n matrices is a real- valued function, || · ||, defined on this set, satisfying for all n n matrices A and B and all C: /* positive definite */ /* homogeneous */ /* triangle inequality */ (4)* || AB || A || · || B || /* consistent */ When you have toenough analyze Oh In haven’t I had general, if we have the of error bound of AB – imagine you doing it new || ABconcepts? || A || What · || Bdo || I , need then withoutthe a consistent matrix norm… consistency the 3 norms are said to for? be consistent. 4/19

Chapter 7 Iterative Techniques in Matrix Algebra -- Norms of Vectors and Matrices Some popularly used norms: Frobenius Norm Natural Norm /* operator norm */ associated with the vector norm || · ||p /* spectral norm */ 5/19

Chapter 7 Iterative Techniques in Matrix Algebra -- Norms of Vectors and Matrices Proof (for Show that ): HW: p. 429 -430 #5(a), 7, 13 Show that Excuses for not Let row p be the maximum doing row, thathomework is I have the proof, but Take a special unit vector such that there isn't room to write it in this margin. 6/19 1

Chapter 7 Iterative Techniques in Matrix Algebra -- Eigenvalues and Eigenvectors 7. 2 Eigenvalues and Eigenvectors Spectral Radius Definition: The spectral radius Im (A) of a matrix A is defined by (A) = max | | where is an eigenvalue of A. Re Theorem: If A is an n n matrix, then (A) || A || for any natural norm || · ||. Proof: For any eigenvalue of A with eigenvector and Definition: We call an n n matrix A convergent if for all i, j = 1, 2, …, n we have HW: p. 436 #3 7/19

Chapter 7 Iterative Techniques in Matrix Algebra -- Iterative Techniques for Solving Linear Systems 7. 3 Iterative Techniques for Solving Linear Systems Jacobi Iterative Method In matrix form: A= D –U –L Tj Jacobi iterative matrix 8/19

Chapter 7 Iterative Techniques in Matrix Algebra -- Iterative Techniques for Solving Linear Systems Algorithm: Jacobi Iterative Method Solve given an initial approximation. Input: the number of equations and unknowns n; the matrix entries a[ ][ ]; the entries b[ ]; the initial approximation X 0[ ]; tolerance TOL; maximum number of iterations Nmax. Since Aofwill not be Output: approximate solution X[ ] or a message failure. bititerations, wasteful, changed during. Athe Step 1 Set k = 1; it? aisn’t = 0? Step 2 While ( k Nmax) do steps 3 -6 we can What reorderifthe iiequations Step 3 For i =wait 1, …, n so that aii 0. Otherwise X(k+1) must till all the entries of X(k) are A is singular. obtained. Hence. Set two vectors are ; /* compute xk */ needed to store the results. Step 4 If then Output (X[ ]); STOP; /* successful */ Step 5 For i = 1, …, n Set X 0[ ] = X [ ]; /* update X 0 */ Step 6 Set k ++; Step 7 Output (Maximum number of iterations exceeded); STOP. /* unsuccessful */ 9/19

Chapter 7 Iterative Techniques in Matrix Algebra -- Iterative Techniques for Solving Linear Systems Gauss - Seidel Iterative Method … … In matrix form: Gauss-Seidel iterative matrix 10/19 Tg Only one vector needs to be saved.

Chapter 7 Iterative Techniques in Matrix Algebra -- Iterative Techniques for Solving Linear Systems A mathematician about his colleague: " He made a lot of mistakes, but he made them in a good direction. I tried to copy this, but I found out that it is very difficult to make good mistakes. " Note: Neither of the methods are always convergent. And more, there are cases in which Jacobi method fails while Gauss-Seidel is convergent, and vice-versa. See Exercises 9 and 10. 11/19

Chapter 7 Iterative Techniques in Matrix Algebra -- Iterative Techniques for Solving Linear Systems Convergence of Iterative Methods Theorem: The following statements are equivalent: (1) (2) (3) (4) (5) A is a convergent matrix; limn || An || = 0 for some natural norm; limn || An || = 0 for all natural norms; (A) < 1; limn An for every Sufficient condition: ||T|| < 1 Necessary condition: 12/19 O ？

Chapter 7 Iterative Techniques in Matrix Algebra -- Iterative Techniques for Solving Linear Systems Rn, the sequence defined by for each k 1, converges to the unique solution of if and only if (T) < 1. Theorem: For any Proof: Given that (T) < 1, then p. 443 (T) < 1 for any (T) < 1 13/19

Chapter 7 Iterative Techniques in Matrix Algebra -- Iterative Techniques for Solving Linear Systems Theorem: If || T || < 1 for any natural matrix norm and is a given vector, then the sequence defined by converges for any Rn, to a vector Rn. And the following error bounds hold: Theorem: If A is strictly diagonally dominant, then for any choice of both the Jacobi and Gauss-Seidel methods give sequences that converge to the unique solution of Proof (Hint): Simply prove that for any | | 1 we have | I T | 0. That is, cannot be an eigenvalue of the corresponding iteration matrix T. 14/19

Chapter 7 Iterative Techniques in Matrix Algebra -- Iterative Techniques for Solving Linear Systems Relaxation Methods Examine Gauss - Seidel method from another angle: where ri(k) = ri (k) + Let x = x. For certain choice of positive , we can a ii /* residual */ reduce the norm of the residual vector and obtain faster (k) i (k – 1) i convergence. Such methods are called Relaxation Methods. 0< <1 15/19 /* Under- Relaxation methods */ =1 /* Gauss - Seidel */ >1 /* Successive Over- Relaxation methods */

Chapter 7 Iterative Techniques in Matrix Algebra -- Iterative Techniques for Solving Linear Systems In matrix form: Oooooh come on! It’s way too complicated to compute T , and you can’t expect me to get its spectral radius right! There’s gotta be a short cut … 16/19

Chapter 7 Iterative Techniques in Matrix Algebra -- Iterative Techniques for Solving Linear Systems Theorem: (Kahan) If aii 0 for each i = 1, 2, …, n. Then (T ) | – 1 |. This implies that the SOR method can converge only if 0 < < 2. Theorem: (Ostrowski-Reich) If A is positive definite and 0 < < 2, the SOR method converges for any choice of initial approximation. Theorem: If A is positive definite and tridiagonal, then (Tg) = [ (Tj)]2 < 1, and the optimal choice of for the SOR method is With this choice of , we have HW: p. 453 #13 17/19

Chapter 7 Iterative Techniques in Matrix Algebra -- Iterative Techniques for Solving Linear Systems Example: Given and an iterative method Then: For what values of that the method will converge? For what values of that the method will have the fastest convergence? Solution: Consider the eigenvalues of T = I + A 1 = 1+ , 2 = 1+ 3 Convergency requires ( T )<1 For what values of that (T) = max { | 1+ |, | 1+ 3 | } assumes its minimum? -2/3 < < 0 = - 1/2 -2/3 18/19 -1/3 0

Chapter 7 Iterative Techniques in Matrix Algebra -- Iterative Techniques for Solving Linear Systems Lab 04. Compare Methods of Jacobi with Gauss-Seidel Time Limit: 1 second; Points: 3 Use Jacobi and Gauss-Seidel methods to solve a given n×n linear system with an initial approximation. Note: When checking each aii , first scan downward for the entry with maximum absolute value (aii included). If that entry is non-zero, swap it to the diagonal. Otherwise if that entry is zero, scan upward for the entry with maximum absolute value. If that entry is non-zero, then add that row to the i-th row. 19/19