Linear Least Squares QR Factorization Systems of linear

Linear Least Squares QR Factorization

Systems of linear equations l Problem l Given to solve: M x = b Mx=b: Is there a solution? l Is the solution unique? l

Systems of linear equations Find a set of weights x so that the weighted sum of the columns of the matrix M is equal to the right hand

Systems of linear equations Existence A solution exists if: There exist weights, x 1, …. , x. N, such that: A solution exists when b is in the span of the columns of M

Systems of linear equations Uniqueness Suppose there exist weights, y 1, …. , y. N, not all zero, such that: Then: Mx = b Mx + My= b M(x+y) = b A solution is unique only if the columns of M are linearly independent.

QR factorization 1 l A matrix Q is said to be orthogonal if its columns are orthonormal, i. e. QT·Q=I. l Orthogonal transformations preserve the Euclidean norm since l Orthogonal matrices can transform vectors in various ways, such as rotation or reflections but they do not change the Euclidean length of the vector. Hence, they preserve the solution to a linear least squares problem.

QR factorization 2 Any matrix A(m·n) can be represented as A = Q·R , where Q(m·n) is orthonormal and R(n·n) is upper triangular:

QR factorization 2 Given A , let its QR decomposition be given as A=Q·R, where Q is an (m x n) orthonormal matrix and R is upper triangular. l QR factorization transform the linear least square problem into a triangular least squares. l Matlab Code: Q·R·x = b R·x = QT·b x=R-1·QT·b

Normal Equations Consider the system It can be a result of some physical measurements, which usually incorporate some errors. Since, we can not solve it exactly, we would like to minimize the error: r=b-Ax r 2=r. Tr=(b-Ax)T(b-Ax)=b. Tb-2 x. TATb+x. TATAx (r 2)x=0 - zero derivative is a (necessary) minimum condition -2 ATb+2 ATAx=0; ATAx = ATb; – Normal Equations

Normal Equations 2 ATAx = ATb – Normal Equations

Least squares via A=QR decomposition A(m, n)=Q(m, n)R(n, n), Q is orthogonal, therefore QTQ=I. QRx=b R(n, n)x=QT(n, m)b(m, 1) -well defined linear system x=R-1 QTb Q is found by Gram=Schmidt orthogonalization of A. How to find R? QR=A QTQR=QTA, but Q is orthogonal, therefore QTQ=I: R=QTA R is upper triangular, since in orthogonalization procedure only a 1, . . ak (without ak+1, …) are used to produce qk

Least squares via A=QR decomposition 2 Let us check the correctness: QRx=b T Rx=Q b -1 T x=R Q b

Last lecture reminder QR Factorization – By picture

QR Factorization – Minimization View Minimization Algorithm For i = 1 to N “For each Target Column” For j = 1 to i-1 “For each Source Column left of target” Orthogonalize Search Direction end Normalize end