3.9: The Least-Squares Problem
View The Least-Squares Problem Using Matrices on YouTube
Suppose there is some experimental data that is suspected to satisfy a functional relationship. The simplest such relationship is linear, and suppose one wants to fit a straight line to the data. An example of such a linear regression problem is shown in Fig. \(\PageIndex{1}\).
In general, let the data consist of a set of \(n\) points given by \((x_1, y_1),\: (x_2, y_2),\cdots , (x_n, y_n)\). Here, the \(x\) values are exact, and the \(y\) values are noisy. We assume that a line of the form
\[y=\beta_0+\beta_1x\nonumber \]
is the best fit to the data. Although we know that the line will not go through all of the data points, we can still write down the resulting equations. We have
\[\begin{aligned}y_1&=\beta_0+\beta_zx_1, \\ y_2&=\beta_0+\beta_1x_2,\\ &\vdots \\ y_n&=\beta_0+\beta_1x_n.\end{aligned} \nonumber \]
These equations are a system of \(n\) equations in the two unknowns \(\beta_0\) and \(\beta_1\). The corresponding matrix equation is given by
\[\left(\begin{array}{cc}1&x_1 \\ 1&x_2 \\ \vdots &\vdots \\ 1&x_n\end{array}\right)\left(\begin{array}{c}\beta_0 \\ \beta_1\end{array}\right)=\left(\begin{array}{c}y_1\\y_2\\ \vdots \\ y_n\end{array}\right).\nonumber \]
This is an overdetermined system of equations that obviously has no solution. The problem of least-squares is to find the best solution of these equations for \(\beta_0\) and \(\beta_1\).
We can generalize this problem as follows. Suppose we are given the matrix equation
\[\text{Ax}=\text{b}\nonumber \]
that has no solution because \(\text{b}\) is not in the column space of \(\text{A}\). Instead of exactly solving this matrix equation, we want to solve another approximate equation that minimizes the error between \(\text{Ax}\) and \(\text{b}\). The error can be defined as the norm of \(\text{Ax} − \text{b}\), and the square of the error is just the sum of the squares of the components. Our search is for the least-squares solution.