Copyright of ShunFeng Su Least Squares Now we

®Copyright of Shun-Feng Su Least Squares Now, we go back to consider a simple problem: Ax=b To solve x ( n), where A m n and b m. Usually, we consider the case of m n. Then, if b is not in the range of A, it is not possible to find the solution. It is called inconsistent or overdetermined. The goal is to find a solution that can minimize ||Ax-b||2. (Least Square Approach) 1

®Copyright of Shun-Feng Su Least Squares To solve x in Ax=b is a very common problem in daily life. It is usually called linear regression. A linear expression of variable is commonly assumed for a relationship. To identify those linear coefficients for the relationship becomes to solve x in Ax=b. In order to have noise averaged, more data seta are collected. It is the case of m>n. In fact, there are various packages, like Matlab, have this function. 2

®Copyright of Shun-Feng Su Least Squares When rank(A)=n, (ATA)-1 exists. Then it is easy to verify that the minimizer is x*=(ATA)-1 ATb. It is also referred to as pseudo inverse. Proof: ||Ax-b||2=||A(x-x*)+(Ax*-b)||2 =||A(x-x*)||2 +||Ax*-b)||2+2[A(x-x*)]T(Ax*-b) is zero by using x*=(ATA)-1 ATb. Since ||A(x-x*)||2 is always non-negative, ||Ax-b||2 ||Ax*-b)||2 3 x*=(ATA)-1 ATb is the minimizer.

®Copyright of Shun-Feng Su Least Squares Geometric interpretation: h=Ax*=A(ATA)-1 ATb. h (A) (range of A) minimizes ||b-h||. 4 h is the orthogonal projection of b onto (A).

®Copyright of Shun-Feng Su Least Squares ATA play an important role in the least-squares solution. This matrix is called the Gram matrix. An alternative method of finding the minimizer is: f(x)=||Ax-b||2= 1/2 x. T(2 ATA)x-x. T(2 ATb)+b. Tb f(x)=2 ATAx-2 ATb By using the FONC (i. e. , f(x)=0), x*=(ATA)-1 ATb. Again, it faces the problem of matrix inverse. Alternative approaches are needed. 5

®Copyright of Shun-Feng Su Recursive Least Squares (RLS) algorithm is to find the minimizer in an iterative way. Given patterns (A 0, b 0), (A 1, b 1), …, (An-1, bn-1), find the minimizer recursively from those patterns. Let G 0=A 0 TA 0. From the above algorithm: x(0)=(G 0)-1 ATb 0 Now, we have a new pattern: (A 1, b 1). It is to minimizer: 6

®Copyright of Shun-Feng Su Recursive Least Squares It is easy to see x(1)=(G 1)-1 where G 1= . Then we can have G 1=G 0 +A 1 TA 1 and =A 0 Tb 0+ A 1 Tb 1. A 0 Tb 0=G 0 G 0 -1 A 0 Tb 0=G 0 x(0)=G 1 x(0)-A 1 TA 1 x(0). 7

®Copyright of Shun-Feng Su Recursive Least Squares Thus, x(1)=G 1 -1(G 1 x(0)-A 1 TA 1 x(0) +A 1 Tb 1) =x(0)+G 1 -1 A 1 T(b 1 -A 1 x(0)). Note G 1=G 0 +A 1 TA 1. Then we can have the following recursive update: Gk+1=Gk +Ak+1 TAk+1 x(k+1) =x(k)+Gk+1 -1 Ak+1 T(bk+1 -Ak+1 x(k)) bk+1 -Ak+1 x(k) is often called the innovation or error. 8 Gk+1 -1 Ak+1 T is the update gain.

®Copyright of Shun-Feng Su Recursive Least Squares But, in the above recursive relations, we need to calculate Gk+1 -1, which is not easy to obtain. To obtain the recursive form for Gk+1 -1 instead of Gk+1, a generalization of the Sherman-Morrison formula is employed. Lemma: (A+UV)-1=A-1 -(A-1 U)(I+VA-1 U)-1(VA-1). Of course, those related matrices must be nonsingular. 9

®Copyright of Shun-Feng Su Recursive Least Squares To replace Gk+1 in the recursive form by Gk+1 -1 (Pk+1) (with the use of the above Lemma): Pk+1=Pk-(Pk. Ak+1 T)(I+Ak+1 Pk. Ak+1 T)-1(Ak+1 Pk) x(k+1) =x(k)+Pk+1 Ak+1 T(bk+1 -Ak+1 x(k)) When Ak+1 is a single row as ak+1 T and bk+1 T is a scalar, the recursive algorithm is Pk+1=Pkx(k+1) =x(k)+Pk+1 ak+1 (bk+1 - ak+1 T x(k)) 10

®Copyright of Shun-Feng Su Recursive Least Squares 11

®Copyright of Shun-Feng Su Recursive Least Squares 12

®Copyright of Shun-Feng Su Least Squares Now, we go back to consider Ax=b for m n. Since now we have more unknowns than equations (x n, b m, and m n), there may exist infinite solutions. Then we can try to find x* to minimize some objective function. A common function considered is to minimize ||x||. It is easy to verify that the minimizer is x*=AT (AAT)-1 b. 13

®Copyright of Shun-Feng Su Least Squares Proof: ||x||2=||(x-x*)+x*||2 =||(x-x*)||2 +||x*||2+2 x*T(x-x*) Now to show x*T(x-x*) is zero. Using x*=AT (AAT)-1 b, x*T(x-x*)=[AT (AAT)-1 b]T(x- AT (AAT)-1 b) =b. T[(AAT)-1]T(Ax-AAT(AAT)-1 b) We know Ax=b. Then the above term is zero. With the similar idea, ||(x-x*)||2 is always nonnegative, ||x||2 ||x*||2. 14 x*=AT (AAT)-1 b is the minimizer.

®Copyright of Shun-Feng Su Least Squares 15

®Copyright of Shun-Feng Su Kaczmarz’s Algorithm This is an iterative version of finding x*=AT (AAT)-1 b. Let aj. T be the jth row of A and bj. T be the jth component of b. is a positive scalar, 0< <2. The recursive algorithm for x*=AT (AAT)-1 b is 1. Set i: = 0 and given x(0). 2. For j=1, …, m, x(im+j)=x(im+j-1)+ (bj-aj. Tx(im+j 1))a /a Ta j j j 3. Set i: = i+1; go to step 2. For every m steps (k=0, . . , m-1), it updates as 16 (k+1) (k) T

®Copyright of Shun-Feng Su Kaczmarz’s Algorithm 17

®Copyright of Shun-Feng Su Kaczmarz’s Algorithm 18

®Copyright of Shun-Feng Su Kaczmarz’s Algorithm For the case of x(0) 0, Kaczmarz’s algorithm converges to the unique point minimizing ||x-x(0)||. If we set =1, Kaczmarz’s algorithm has the property that at each iteration, b. R(k)+1 -a. R(k)+1 Tx(k+1)=0 where R(k) is the residue on m and is an integer in {0, …, m} satisfying k=lm+R(k) for some integer l. It is easy to verify that b. R(k)+1=a. R(k)+1 Tx* (from Ax=b). Then, a. R(k)+1(x(k+1)-x*)=0. It means x(k+1)-x* is 19 orthogonal to a. R(k)+1.

®Copyright of Shun-Feng Su Kaczmarz’s Algorithm 20

®Copyright of Shun-Feng Su Kaczmarz’s Algorithm 21 and 12. 16. Prob-7: 12. 2, 12. 3, 12. 13,