Understand the orthogonal decomposition of a vector with respect to a subspace.
Understand the relationship between orthogonal decomposition and orthogonal projection.
Understand the relationship between orthogonal decomposition and the closest vector on / distance to a subspace.
Learn the basic properties of orthogonal projections as linear transformations and as matrix transformations.
Recipes: orthogonal projection onto a line, orthogonal decomposition by solving a system of equations, orthogonal projection via a complicated matrix product.
Let be a subspace of and let be a vector in . In this section, we will learn to compute the closest vector to in . The vector is called the orthogonal projection of onto . This is exactly what we will use to almost solve matrix equations, as discussed in the introduction to Chapter 6.
Orthogonal Decomposition
We begin by fixing some notation.
Definition : Notation
Let be a subspace of and let be a vector in . We denote the closest vector to on by .
To say that is the closest vector to on means that the difference is orthogonal to the vectors in
Figure
In other words, if then we have where is in and is in . The first order of business is to prove that the closest vector always exists.
Theorem: Orthogonal Decomposition
Let be a subspace of and let be a vector in . Then we can write uniquely as
where is the closest vector to on and is in .
Proof
Let so by Fact 6.2.1 in Section 6.2. Let be a basis for and let be a basis for . We showed in the proof of this Fact 6.2.1 in Section 6.2 that is linearly independent, so it forms a basis for . Therefore, we can write
where and . Since is orthogonal to the vector is the closest vector to on so this proves that such a decomposition exists.
As for uniqueness, suppose that
for in and in . Rearranging gives
Since and are subspaces, the left side of the equation is in and the right side is in . Therefore, is in and in so it is orthogonal to itself, which implies . Hence and which proves uniqueness.
Definition : Orthogonal Decomposition and Orthogonal Projection
Let be a subspace of and let be a vector in . The expression
for in and in is called the orthogonal decomposition of with respect to and the closest vector is the orthogonal projection of onto .
Since is the closest vector on to the distance from to the subspace is the length of the vector from to i.e., the length of . To restate:
Note: Closest Vector and Distance
Let be a subspace of and let be a vector in .
The orthogonal projection is the closest vector to in .
The distance from to is .
Example : Orthogonal decomposition with respect to the -plane
Let be the -plane in so is the -axis. It is easy to compute the orthogonal decomposition of a vector with respect to this
We see that the orthogonal decomposition in this case expresses a vector in terms of a “horizontal” component (in the -plane) and a “vertical” component (on the -axis).
Figure
Figure : Orthogonal decomposition of a vector with respect to the -plane in . Note that is in the -plane and is in the -axis. Click and drag the head of the vector to see how the orthogonal decomposition changes.
Example : Orthogonal decomposition of a vector in
If is in a subspace then the closest vector to in is itself, so and . Conversely, if then is contained in because is contained in .
Example : Orthogonal decomposition of a vector in
If is a subspace and is in then the orthogonal decomposition of is where is in and is in . It follows that . Conversely, if then the orthogonal decomposition of is so is in .
Example : Interactive: Orthogonal decomposition in
Figure: Orthogonal decomposition of a vector with respect to a line in . Note that is in and is in the line perpendicular to . Click and drag the head of the vector to see how the orthogonal decomposition changes.
Example : Interactive: Orthogonal decomposition in
Figure: Orthogonal decomposition of a vector with respect to a plane in . Note that is in and is in the line perpendicular to . Click and drag the head of the vector to see how the orthogonal decomposition changes.
Example : Interactive: Orthogonal decomposition in
Figure : Orthogonal decomposition of a vector with respect to a line in . Note that is in and is in the plane perpendicular to . Click and drag the head of the vector to see how the orthogonal decomposition changes.
Now we turn to the problem of computing and . Of course, since really all we need is to compute . The following theorem gives a method for computing the orthogonal projection onto a column space. To compute the orthogonal projection onto a general subspace, usually it is best to rewrite the subspace as the column space of a matrix, as in Note 2.6.3 in Section 2.6.
Theorem
Let be an matrix, let and let be a vector in . Then the matrix equation
in the unknown vector is consistent, and is equal to for any solution .
Proof
Let be the orthogonal decomposition with respect to . By definition lies in and so there is a vector in with . Choose any such vector . We know that lies in which is equal to by Recipe: Shortcuts for Computing Orthogonal Complements in Section 6.2. We thus have
and so
This exactly means that is consistent. If is any solution to then by reversing the above logic, we conclude that .
Example :Orthogonal Projection onto a Line
Let be a line in and let be a vector in . By Theorem , to find we must solve the matrix equation where we regard as an matrix (the column space of this matrix is exactly ). But and so is a solution of and hence
Figure
To reiterate:
Recipe: Orthogonal Projection onto a Line
If is a line, then
for any vector .
Remark: Simple proof for the formula for projection onto a line
In the special case where we are projecting a vector in onto a line our formula for the projection can be derived very directly and simply. The vector is a multiple of say . This multiple is chosen so that is perpendicular to as in the following picture.
Figure
In other words,
Using the distributive property for the dot product and isolating the variable gives us that
and so
Example : Projection onto a line in
Compute the orthogonal projection of onto the line spanned by and find the distance from to .
Solution
First we find
The distance from to is
Figure
Figure: Distance from the line .
Example : Projection onto a line in
Let
and let be the line spanned by . Compute and .
Solution
Figure: Orthogonal projection onto the line .
When is a matrix with more than one column, computing the orthogonal projection of onto means solving the matrix equation . In other words, we can compute the closest vector by solving a system of linear equations. To be explicit, we state Theorem as a recipe:
Recipe: Compute an Orthogonal Decomposition
Let be a subspace of . Here is a method to compute the orthogonal decomposition of a vector with respect to
Rewrite as the column space of a matrix . In other words, find a a spanning set for and let be the matrix with those columns.
Compute the matrix and the vector .
Form the augmented matrix for the matrix equation in the unknown vector and row reduce.
This equation is always consistent; choose one solution . Then
Example : Projection onto the -plane
Use Theorem to compute the orthogonal decomposition of a vector with respect to the -plane in .
Solution
A basis for the -plane is given by the two standard coordinate vectors
Let be the matrix with columns
Then
It follows that the unique solution of is given by the first two coordinates of so
We have recovered this Example .
Example : Projection onto a plane in
Let
Compute and the distance from to .
Solution
We have to solve the matrix equation where
We have
We form an augmented matrix and row reduce:
It follows that
The distance from to is
Figure: Orthogonal projection onto the plane .
Example : Projection onto another plane in
Let
Compute .
Solution
Method 1: First we need to find a spanning set for . We notice that is the solution set of the homogeneous equation so . We know how to compute a basis for a null space: we row reduce and find the parametric vector form. The matrix is already in reduced row echelon form. The parametric form is so the parametric vector form is
and hence a basis for is given by
We let be the matrix whose columns are our basis vectors:
Hence .
Now we can continue with step 1 of the recipe. We compute
We write the linear system as an augmented matrix and row reduce:
Hence we can take so
Figure: Orthogonal projection onto the plane .
Method 2: In this case, it is easier to compute . Indeed, since the orthogonal complement is the line
Using the formula for projection onto a line, Example , gives
Hence we have
as above.
Example : Projection onto a -space in
Let
Compute the orthogonal decomposition of with respect to .
Solution
We have to solve the matrix equation where
We compute
We form an augmented matrix and row reduce:
It follows that
In the context of the above recipe, if we start with a basis of then it turns out that the square matrix is automatically invertible! (It is always the case that is square and the equation is consistent, but need not be invertible in general.)
Corollary
Let be an matrix with linearly independent columns and let . Then the matrix is invertible, and for all vectors in we have
Proof
We will show that which implies invertibility by Theorem 5.1.1 in Section 5.1. Suppose that . Then so by Theorem . But (the orthogonal decomposition of the zero vector is just so and therefore is in . Since the columns of are linearly independent, we have so as desired.
Let be a vector in and let be a solution of . Then so .
The corollary applies in particular to the case where we have a subspace of and a basis for . To apply the corollary, we take to be the matrix with columns .
Example : Computing a projection
Continuing with the above Example , let
Compute using the formula .
Solution
Clearly the spanning vectors are noncollinear, so according to Corollary , we have where
We compute
so
So, for example, if this formula tells us that .
Orthogonal Projection
In this subsection, we change perspective and think of the orthogonal projection as a function of . This function turns out to be a linear transformation with many nice properties, and is a good example of a linear transformation which is not originally defined as a matrix transformation.
Proposition: Properties of Orthogonal Projections
Let be a subspace of and define by . Then:
is a linear transformation.
if and only if is in .
if and only if is in .
.
The range of is .
Proof
We have to verify the defining properties of linearity, Definition 3.3.1 in Section 3.3. Let be vectors in and let and be their orthogonal decompositions. Since and are subspaces, the sums and are in and respectively. Therefore, the orthogonal decomposition of is so Now let be a scalar. Then is in and is in so the orthogonal decomposition of is and therefore, Since satisfies the two defining properties, Definition 3.3.1 in Section 3.3, it is a linear transformation.
See Example .
See Example .
For any in the vector is in so by 2. Any vector in is in the range of because for such vectors. On the other hand, for any vector in the output is in so is the range of .
We compute the standard matrix of the orthogonal projection in the same way as for any other transformation, Theorem 3.3.1 in Section 3.3: by evaluating on the standard coordinate vectors. In this case, this means projecting the standard coordinate vectors onto the subspace.
Example : Matrix of a projection
Let be the line in spanned by the vector and define by . Compute the standard matrix for .
Solution
The columns of are and . We have
Example : Matrix of a projection
Let be the line in spanned by the vector
and define by . Compute the standard matrix for .
Solution
The columns of are and . We have
Example : Matrix of a projection
Continuing with Example , let
and define by . Compute the standard matrix for .
Solution
The columns of are and . Let
To compute each we solve the matrix equation for then use the equality . First we note that
For we form an augmented matrix and row reduce:
We do the same for
and for
It follows that
In the previous Example , we could have used the fact that
forms a basis for so that
by the Corollary . In this case, we have already expressed as a matrix transformation with matrix . See this Example .
Note
Let be a subspace of with basis and let be the matrix with columns . Then the standard matrix for is
We can translate the above properties of orthogonal projections, Proposition , into properties of the associated standard matrix.
Proposition: Properties of Projection Matrices
Let be a subspace of define by and let be the standard matrix for . Then:
If then 1 is an eigenvalue of and the 1-eigenspace for is .
If then 0 is an eigenvalue of and the 0-eigenspace for is .
is similar to the diagonal matrix with ones and zeros on the diagonal, where
For the final assertion, we showed in the proof of this Theorem that there is a basis of of the form where is a basis for and is a basis for . Each is an eigenvector of indeed, for we have
because is in and for we have
because is in . Therefore, we have found a basis of eigenvectors, with associated eigenvalues ( ones and zeros). Now we use Theorem 5.4.1 in Section 5.4.
We emphasize that the properties of projection matrices, Proposition , would be very hard to prove in terms of matrices. By translating all of the statements into statements about linear transformations, they become much more transparent. For example, consider the projection matrix we found in Example . Just by looking at the matrix it is not at all obvious that when you square the matrix you get the same matrix back.
Example
Continuing with above Example , we showed that
is the standard matrix of the orthogonal projection onto
One can verify by hand that (try it!). We compute as the null space of
The free variable is and the parametric form is so that
It follows that has eigenvectors
with eigenvalues respectively, so that
Remark
As we saw in Example , if you are willing to compute bases for and then this provides a third way of finding the standard matrix for projection onto indeed, if is a basis for and is a basis for then
where the middle matrix in the product is the diagonal matrix with ones and zeros on the diagonal. However, since you already have a basis for it is faster to multiply out the expression as in Corollary .
Remark: Reflections
Let be a subspace of and let be a vector in . The reflection of over is defined to be the vector
In other words, to find one starts at then moves to then continues in the same direction one more time, to end on the opposite side of .
Figure
Since we also have
We leave it to the reader to check using the definition that:
The -eigenspace of is and the -eigenspace of is .
is similar to the diagonal matrix with ones on the diagonal and negative ones.