Chapter 10 Real Inner Products and LeastSquare cont

Chapter 10 Real Inner Products and Least-Square (cont. ) In this handout: Section 10. 5: Least-Squares

A common problem in business, science, engineering is to collect data and analyze them to predict future events. If such data are plotted, they constitute a scatter diagram which may provide useful insight into the underlying relationship between system variables. The data below appears to follow a straight line relationship. The problem is to determine the equation of the straight line that best fits the data. y x

Consider an arbitrary straight line, y = b 0 + b 1 x , to be fitted through these data points. For each data point, the error is the difference between the y-value of the point and the yvalue obtained from the straight line approximation.

The least-squares straight line Definition 1: The least-squares error E is the sum of the squares of the individual errors. That is, Definition 2: The least-squares straight line is the line that minimizes the least-squares error. We want to find the equation of the least-squares straight line: y = mx + c We seek values of m and c that minimizes the least-squares error.

The normal equations For each of N data points, the error is We want the values for m and c that minimize This occurs when Or, upon simplifying, when The last two equations are the normal equations for a leastsquares fit in two variables. Examples on the board.

Matrix representation of the normal equations Ideally, we would like to choose m and c so that yi = mxi + c for all data pairs (xi, yi), i=1, 2, …, N. That is, we want the values for m and c that solve the system or, equivalently, the matrix equation

Matrix representation of the normal equations This system has the standard form Ax=b where x = [m c], b = [y 1 y 2 … y. N], and A has two columns [x 1 x 2 … x. N] and [1 1 … 1]. Ax=b has a solution if and only if the data falls on a straight line. If not, then the system is inconsistent, and we seek a solution that minimizes the least-square error: The least-square solution which is given by the normal equations has the following matrix form

The least-squares solution for any linear system The concept of linear-squares can be generalized to any linear system Ax=b. • We are primarily interested in cases where the system is inconsistent. This generally occurs when A has more rows than columns. • Measurement errors are inevitable in observational and experimental sciences. • Errors can be smoothed out by averaging over many cases, i. e. , taking more measurements than are strictly necessary to determine parameters of system. • Resulting system is overdetermined (more rows than columns), so usually there is no exact solution. • The least-squares is an approximate solution to this kind of systems.

The least-squares solution for any linear system We seek the vector that minimizes the least-squares error defined by Theorem 1: If x has the property that Ax-b is orthogonal to the columns of A, then x minimizes the least-squares error. As a consequence to Theorem 1, x is the least-squares solution to Ax=b if and only if x is the solution to This set of normal equations is guaranteed to have a unique solution whenever the columns of A are linearly independent. The solution can be found using the techniques of previous chapters.

The least-squares solution of a linear system (example) • Find the least squares solution of the linear system Ax = b given by x 1 – x 2 = 4 3 x 1 + 2 x 2 = 1 -2 x 1 + 4 x 2 = 3 • Solution:

The least-squares solution of a linear system (example) Solution (cont. ): We have so the normal system ATAx = ATb in this case is Solving this system yields the least squares solution x 1 = 17/95, x 2 = 143/285