4.10: Orthogonal Projections

Learning Objectives

• Find the orthogonal projection of a vector onto a subspace.
• Find the orthogonal complement of a subspace.

In this section, we compute orthogonal projections of vectors onto subspaces.

An important use of the Gram-Schmidt Process is in orthogonal projections, the focus of this section.

You may recall that a subspace of $\mathbb{R}^n$ is a set of vectors which contains the zero vector, and is closed under addition and scalar multiplication. Let’s call such a subspace $W$. In particular, a plane in $\mathbb{R}^n$ which contains the origin, $\left(0,0, \cdots, 0 \right)$, is a subspace of $\mathbb{R}^n$.

Suppose a point $Y$ in $\mathbb{R}^n$ is not contained in $W$, then what point $Z$ in $W$ is closest to $Y$? Using the Gram-Schmidt Process, we can find such a point. Let $\vec{y}, \vec{z}$ represent the position vectors of the points $Y$ and $Z$ respectively, with $\vec{y}-\vec{z}$ representing the vector connecting the two points $Y$ and $Z$. It will follow that if $Z$ is the point on $W$ closest to $Y$, then $\vec{y} - \vec{z}$ will be perpendicular to $W$ (can you see why?); in other words, $\vec{y} - \vec{z}$ is orthogonal to $W$ (and to every vector contained in $W$) as in the following diagram.

Figure $\PageIndex{1}$ This diagram illustrates the concept of orthogonal projection of a vector onto a subspace. It shows a vector $\vec y$, a subspace $W$, and the orthogonal projection of $\vec y$ onto $W$, denoted as $\vec z$. The diagram helps visualize the relationship between the vector, the subspace, and the projection. ( CC BY-NC-SA 4.0; Kuttler via A First Course in LINEAR ALGEBRA)

The vector $\vec{z}$ is called the orthogonal projection of $\vec{y}$ on $W$. The definition is given as follows.

Definition $\PageIndex{1}$: Orthogonal Projection

Let $W$ be a subspace of $\mathbb{R}^n$, and $Y$ be any point in $\mathbb{R}^n$. Then the orthogonal projection of $Y$ onto $W$ is given by $$\vec{z} = \mathrm{proj}_{W}\left( \vec{y}\right) = \left( \frac{\vec{y} \cdot \vec{w}_1}{ \| \vec{w}_1 \| ^2}\right) \vec{w}_1 + \left( \frac{\vec{y} \cdot \vec{w}_2}{ \| \vec{w}_2 \| ^2}\right) \vec{w}_2 + \cdots + \left( \frac{\vec{y} \cdot \vec{w}_m}{ \| \vec{w}_m \| ^2}\right) \vec{w}_m\nonumber$$ where $\{\vec{w}_1, \vec{w}_2, \cdots, \vec{w}_m \}$ is any orthogonal basis of $W$.

Therefore, in order to find the orthogonal projection, we must first find an orthogonal basis for the subspace. Note that one could use an orthonormal basis, but it is not necessary in this case since as you can see above the normalization of each vector is included in the formula for the projection.

Before we explore this further through an example, we show that the orthogonal projection does indeed yield a point $Z$ (the point whose position vector is the vector $\vec{z}$ above) which is the point of $W$ closest to $Y$.

Theorem $\PageIndex{1}$: Approximation Theorem

Let $W$ be a subspace of $\mathbb{R}^n$ and $Y$ any point in $\mathbb{R}^n$. Let $Z$ be the point whose position vector is the orthogonal projection of $Y$ onto $W$.

Then, $Z$ is the point in $W$ closest to $Y$.

Proof

First $Z$ is certainly a point in $W$ since it is in the span of a basis of $W$.

To show that $Z$ is the point in $W$ closest to $Y$, we wish to show that $|\vec{y}-\vec{z}_1| > |\vec{y}-\vec{z}|$ for all $\vec{z}_1 \neq \vec{z} \in W$. We begin by writing $\vec{y}-\vec{z}_1 = (\vec{y} - \vec{z}) + (\vec{z} - \vec{z}_1)$. Now, the vector $\vec{y} - \vec{z}$ is orthogonal to $W$, and $\vec{z} - \vec{z}_1$ is contained in $W$. Therefore these vectors are orthogonal to each other. By the Pythagorean Theorem, we have that $$\| \vec{y} - \vec{z}_1 \| ^2 = \| \vec{y} - \vec{z} \| ^2 + \| \vec{z} -\vec{z}_1 \| ^2 > \| \vec{y} - \vec{z} \| ^2\nonumber$$ This follows because $\vec{z} \neq \vec{z}_1$ so $\| \vec{z} -\vec{z}_1 \| ^2 > 0.$

Hence, $\| \vec{y} - \vec{z}_1 \| ^2 > \| \vec{y} - \vec{z} \| ^2$. Taking the square root of each side, we obtain the desired result.

Consider the following example.

Example $\PageIndex{1}$: Orthogonal Projection

Let $W$ be the plane through the origin given by the equation $x - 2y + z = 0$.
Find the point in $W$ closest to the point $Y = (1,0,3)$.

Solution

We must first find an orthogonal basis for $W$. Notice that $W$ is characterized by all points $(a,b,c)$ where $c = 2b-a$. In other words, $$W = \left[ \begin{array}{c} a \\ b \\ 2b - a \end{array} \right] = a \left[ \begin{array}{c} 1 \\ 0 \\ -1 \end{array} \right] + b \left[ \begin{array}{c} 0 \\ 1 \\ 2 \end{array} \right], \; a,b \in \mathbb{R}\nonumber$$

We can thus write $W$ as $$\begin{aligned} W &= \mbox{span} \left\{ \vec{u}_1, \vec{u}_2 \right\} \\ &= \mbox{span} \left\{ \left[ \begin{array}{r} 1 \\ 0 \\ -1 \end{array} \right], \left[ \begin{array}{c} 0 \\ 1 \\ 2 \end{array} \right] \right\}\end{aligned}$$

Notice that this span is a basis of $W$ as it is linearly independent. We will use the Gram-Schmidt Process to convert this to an orthogonal basis, $\left\{\vec{w}_1, \vec{w}_2 \right\}$. In this case, as we remarked it is only necessary to find an orthogonal basis, and it is not required that it be orthonormal.

$$\vec{w}_1 = \vec{u}_1 = \left[ \begin{array}{r} 1 \\ 0 \\ -1 \end{array} \right]\nonumber$$ $$\begin{aligned} \vec{w}_2 &= \vec{u}_2 - \left( \frac{ \vec{u}_2 \cdot \vec{w}_1}{ \| \vec{w}_1 \| ^2} \right) \vec{w}_1\\ &= \left[ \begin{array}{c} 0 \\ 1 \\ 2 \end{array} \right] - \left( \frac{-2}{2}\right) \left[ \begin{array}{r} 1 \\ 0 \\ -1 \end{array} \right] \\ &= \left[ \begin{array}{c} 0 \\ 1 \\ 2 \end{array} \right] + \left[ \begin{array}{r} 1 \\ 0 \\ -1 \end{array} \right] \\ &= \left[ \begin{array}{c} 1 \\ 1 \\ 1 \end{array} \right]\end{aligned}$$

Therefore an orthogonal basis of $W$ is $$\left\{ \vec{w}_1, \vec{w}_2 \right\} = \left\{ \left[ \begin{array}{r} 1 \\ 0 \\ -1 \end{array} \right], \left[ \begin{array}{c} 1 \\ 1 \\ 1 \end{array} \right] \right\}\nonumber$$

We can now use this basis to find the orthogonal projection of the point $Y=(1,0,3)$ on the subspace $W$. We will write the position vector $\vec{y}$ of $Y$ as $\vec{y} = \left[ \begin{array}{c} 1 \\ 0 \\ 3 \end{array} \right]$. We compute the projection as follows: $$\begin{aligned} \vec{z} &= \mathrm{proj}_{W}\left( \vec{y}\right)\\ &= \left( \frac{\vec{y} \cdot \vec{w}_1}{ \| \vec{w}_1 \| ^2}\right) \vec{w}_1 + \left( \frac{\vec{y} \cdot \vec{w}_2}{ \| \vec{w}_2 \| ^2}\right) \vec{w}_2 \\ &= \left( \frac{-2}{2} \right) \left[ \begin{array}{r} 1 \\ 0 \\ -1 \end{array} \right] + \left( \frac{4}{3} \right) \left[ \begin{array}{c} 1 \\ 1 \\ 1 \end{array} \right] \\ &= \left[ \begin{array}{c} \frac{1}{3} \\ \frac{4}{3} \\ \frac{7}{3} \end{array} \right]\end{aligned}$$

Therefore the point $Z$ on $W$ closest to the point $(1,0,3)$ is $\left( \frac{1}{3}, \frac{4}{3}, \frac{7}{3} \right)$.

Recall that the vector $\vec{y} - \vec{z}$ is perpendicular (orthogonal) to all the vectors contained in the plane $W$. Using a basis for $W$, we can in fact find all such vectors which are perpendicular to $W$. We call this set of vectors the orthogonal complement of $W$ and denote it $W^{\perp}$.

Definition $\PageIndex{2}$: Orthogonal Complement

Let $W$ be a subspace of $\mathbb{R}^n$. Then the orthogonal complement of $W$, written $W^{\perp}$, is the set of all vectors $\vec{x}$ such that $\vec{x} \cdot \vec{z} = 0$ for all vectors $\vec{z}$ in $W$. $$W^{\perp} = \{ \vec{x} \in \mathbb{R}^n \; \mbox{such that} \; \vec{x} \cdot \vec{z} = 0 \; \mbox{for all} \; \vec{z} \in W \}\nonumber$$

The orthogonal complement is defined as the set of all vectors which are orthogonal to all vectors in the original subspace. It turns out that it is sufficient that the vectors in the orthogonal complement be orthogonal to a spanning set of the original space.

Proposition $\PageIndex{1}$: Orthogonal to Spanning Set

Let $W$ be a subspace of $\mathbb{R}^n$ such that $W = \mathrm{span} \left\{ \vec{w}_1, \vec{w}_2, \cdots, \vec{w}_m \right\}$. Then $W^{\perp}$ is the set of all vectors which are orthogonal to each $\vec{w}_i$ in the spanning set.

The following proposition demonstrates that the orthogonal complement of a subspace is itself a subspace.

Proposition $\PageIndex{2}$: The Orthogonal Complement

Let $W$ be a subspace of $\mathbb{R}^n$. Then the orthogonal complement $W^{\perp}$ is also a subspace of $\mathbb{R}^n$.

Consider the following proposition.

Proposition $\PageIndex{3}$: Orthogonal Complement of $\mathbb{R}^n$

The complement of $\mathbb{R}^n$ is the set containing the zero vector: $$(\mathbb{R}^n)^{\perp} = \left\{ \vec{0} \right\}\nonumber$$ Similarly, $$\left\{ \vec{0} \right\}^{\perp} = (\mathbb{R}^n). \nonumber$$

Proof

Here, $\vec{0}$ is the zero vector of $\mathbb{R}^n$. Since $\vec{x}\cdot\vec{0}=0$ for all $\vec{x}\in\mathbb{R}^n$, $\mathbb{R}^n\subseteq\{ \vec{0}\}^{\perp}$. Since $\{ \vec{0}\}^{\perp}\subseteq\mathbb{R}^n$, the equality follows, i.e., $\{ \vec{0}\}^{\perp}=\mathbb{R}^n$.

Again, since $\vec{x}\cdot\vec{0}=0$ for all $\vec{x}\in\mathbb{R}^n$, $\vec{0}\in (\mathbb{R}^n)^{\perp}$, so $\{ \vec{0}\}\subseteq(\mathbb{R}^n)^{\perp}$. Suppose $\vec{x}\in\mathbb{R}^n$, $\vec{x}\neq\vec{0}$. Since $\vec{x}\cdot\vec{x}=||\vec{x}||^2$ and $\vec{x}\neq\vec{0}$, $\vec{x}\cdot\vec{x}\neq 0$, so $\vec{x}\not\in(\mathbb{R}^n)^{\perp}$. Therefore $(\mathbb{R}^n)^{\perp}\subseteq \{\vec{0}\}$, and thus $(\mathbb{R}^n)^{\perp}=\{\vec{0}\}$.

In the next example, we will look at how to find $W^{\perp}$.

Example $\PageIndex{2}$: Orthogonal Complement

Let $W$ be the plane through the origin given by the equation $x - 2y + z = 0$. Find a basis for the orthogonal complement of $W$.

Solution

From Example $\PageIndex{1}$ we know that we can write $W$ as $$W = \mbox{span} \left\{ \vec{u}_1, \vec{u}_2 \right\} = \mbox{span} \left\{ \left[ \begin{array}{r} 1 \\ 0 \\ -1 \end{array} \right], \left[ \begin{array}{c} 0 \\ 1 \\ 2 \end{array} \right] \right\}\nonumber$$

In order to find $W^{\perp}$, we need to find all $\vec{x}$ which are orthogonal to every vector in this span.

Let $\vec{x} = \left[ \begin{array}{c} x_1 \\ x_2 \\ x_3 \end{array} \right]$. In order to satisfy $\vec{x} \cdot \vec{u}_1 = 0$, the following equation must hold. $$x_1 - x_3 = 0\nonumber$$

In order to satisfy $\vec{x} \cdot \vec{u}_2 = 0$, the following equation must hold. $$x_2 + 2x_3 = 0\nonumber$$

Both of these equations must be satisfied, so we have the following system of equations. $$\begin{array}{c} x_1 - x_3 = 0 \\ x_2 + 2x_3 = 0 \end{array}\nonumber$$

To solve, set up the augmented matrix.

$$\left[ \begin{array}{rrr|r} 1 & 0 & -1 & 0 \\ 0 & 1 & 2 & 0 \end{array} \right]\nonumber$$

Using Gaussian Elimination, we find that $W^{\perp} = \mbox{span} \left\{ \left[ \begin{array}{r} 1 \\ -2 \\ 1 \end{array} \right] \right\}$, and hence $\left\{ \left[ \begin{array}{r} 1 \\ -2 \\ 1 \end{array} \right] \right\}$ is a basis for $W^{\perp}$.

The following results summarize the important properties of the orthogonal projection.

Theorem $\PageIndex{2}$: Orthogonal Projection

Let $W$ be a subspace of $\mathbb{R}^n$, $Y$ be any point in $\mathbb{R}^n$, and let $Z$ be the point in $W$ closest to $Y$. Then,

1. The position vector $\vec{z}$ of the point $Z$ is given by $\vec{z} = \mathrm{proj}_{W}\left( \vec{y}\right)$
2. $\vec{z} \in W$ and $\vec{y} - \vec{z} \in W^{\perp}$
3. $| Y - Z | < | Y - Z_1 |$ for all $Z_1 \neq Z \in W$

Consider the following example of this concept.

Example $\PageIndex{3}$: Find a Vector Closest to a Given Vector

Let $$\vec{x}_1=\left[\begin{array}{c} 1\\ 0\\ 1\\ 0 \end{array}\right], \vec{x}_2=\left[\begin{array}{c} 1\\ 0\\ 1\\ 1 \end{array}\right], \vec{x}_3=\left[\begin{array}{c} 1\\ 1\\ 0\\ 0 \end{array}\right], \mbox{ and } \vec{v}=\left[\begin{array}{c} 4\\ 3\\ -2\\ 5 \end{array}\right].\nonumber$$ We want to find the vector in $W =\mathrm{span}\{\vec{x}_1, \vec{x}_2,\vec{x}_3\}$ closest to $\vec{y}$.

Solution

We will first use the Gram-Schmidt Process to construct the orthogonal basis, $B$, of $W$: $$B=\left\{ \left[\begin{array}{c} 1\\ 0\\ 1\\ 0 \end{array}\right], \left[\begin{array}{c} 0\\ 0\\ 0\\ 1 \end{array}\right], \left[\begin{array}{r} 1\\ 2\\ -1\\ 0 \end{array}\right] \right\}.\nonumber$$

By Theorem $\PageIndex{2}$, $$\mathrm{proj}_U(\vec{v}) = \frac{2}{2} \left[\begin{array}{c} 1\\ 0\\ 1\\ 0 \end{array}\right] + \frac{5}{1}\left[\begin{array}{c} 0\\ 0\\ 0\\ 1 \end{array}\right] + \frac{12}{6}\left[\begin{array}{r} 1\\ 2\\ -1\\ 0 \end{array}\right] = \left[\begin{array}{r} 3\\ 4\\ -1\\ 5 \end{array}\right]\nonumber$$ is the vector in $U$ closest to $\vec{y}$.

Consider the next example.

Example $\PageIndex{4}$: Vector Written as a Sum of Two Vectors

Let $W$ be a subspace given by $W = \mbox{span} \left\{ \left[ \begin{array}{c} 1 \\ 0 \\ 1 \\ 0 \\ \end{array} \right], \left[ \begin{array}{c} 0 \\ 1 \\ 0 \\ 2 \\ \end{array} \right] \right\}$, and $Y = (1,2,3,4)$.
Find the point $Z$ in $W$ closest to $Y$, and moreover write $\vec{y}$ as the sum of a vector in $W$ and a vector in $W^{\perp}$.

Solution

From Theorem $\PageIndex{3}$ the point $Z$ in $W$ closest to $Y$ is given by $\vec{z} = \mathrm{proj}_{W}\left( \vec{y}\right)$.

Notice that since the above vectors already give an orthogonal basis for $W$, we have:

$$\begin{aligned} \vec{z} &= \mathrm{proj}_{W}\left( \vec{y}\right)\\ &= \left( \frac{\vec{y} \cdot \vec{w}_1}{ \| \vec{w}_1 \| ^2}\right) \vec{w}_1 + \left( \frac{\vec{y} \cdot \vec{w}_2}{ \| \vec{w}_2 \| ^2}\right) \vec{w}_2 \\ &= \left( \frac{4}{2} \right) \left[ \begin{array}{c} 1 \\ 0 \\ 1 \\ 0 \end{array} \right] + \left( \frac{10}{5} \right) \left[ \begin{array}{c} 0 \\ 1 \\ 0 \\ 2 \end{array} \right] \\ &= \left[ \begin{array}{c} 2 \\ 2 \\ 2 \\ 4 \end{array} \right]\end{aligned}$$

Therefore the point in $W$ closest to $Y$ is $Z = (2,2,2,4)$.
Now, we need to write $\vec{y}$ as the sum of a vector in $W$ and a vector in $W^{\perp}$. This can easily be done as follows: $$\vec{y} = \vec{z} + (\vec{y} - \vec{z})\nonumber$$ since $\vec{z}$ is in $W$ and as we have seen $\vec{y} - \vec{z}$ is in $W^{\perp}$.
The vector $\vec{y} - \vec{z}$ is given by $$\vec{y} - \vec{z} = \left[ \begin{array}{c} 1 \\ 2 \\ 3 \\ 4 \end{array} \right] - \left[ \begin{array}{c} 2 \\ 2 \\ 2 \\ 4 \end{array} \right] = \left[ \begin{array}{r} -1 \\ 0 \\ 1 \\ 0 \end{array} \right]\nonumber$$ Therefore, we can write $\vec{y}$ as $$\left[ \begin{array}{c} 1 \\ 2 \\ 3 \\ 4 \end{array} \right] = \left[ \begin{array}{c} 2 \\ 2 \\ 2 \\ 4 \end{array} \right] + \left[ \begin{array}{r} -1 \\ 0 \\ 1 \\ 0 \end{array} \right]\nonumber$$

Example $\PageIndex{5}$: Point in a Plane Closest to a Given Point

Find the point $Z$ in the plane $3x+y-2z=0$ that is closest to the point $Y=(1,1,1)$.

Solution

The solution will proceed as follows.

1. Find a basis $X$ of the subspace $W$ of $\mathbb{R}^3$ defined by the equation $3x+y-2z=0$.
2. Orthogonalize the basis $X$ to get an orthogonal basis $B$ of $W$.
3. Find the projection on $W$ of the position vector of the point $Y$.

We now begin the solution.

1. $3x+y-2z=0$ is a system of one equation in three variables. Putting the augmented matrix in reduced row-echelon form: $$\left[\begin{array}{rrr|r} 3 & 1 & -2 & 0 \end{array}\right] \rightarrow \left[\begin{array}{rrr|r} 1 & \frac{1}{3} & -\frac{2}{3} & 0 \end{array}\right]\nonumber$$ gives general solution $x=\frac{1}{3}s+\frac{2}{3}t$, $y=s$, $z=t$ for any $s,t\in\mathbb{R}$. Then $$W=\mathrm{span} \left\{ \left[\begin{array}{r} -\frac{1}{3} \\ 1 \\ 0 \end{array}\right], \left[\begin{array}{r} \frac{2}{3} \\ 0 \\ 1 \end{array}\right]\right\}\nonumber$$ Let $X=\left\{ \left[\begin{array}{r} -1 \\ 3 \\ 0 \end{array}\right], \left[\begin{array}{r} 2 \\ 0 \\ 3 \end{array}\right]\right\}$. Then $X$ is linearly independent and $\mathrm{span}(X)=W$, so $X$ is a basis of $W$.
2. Use the Gram-Schmidt Process to get an orthogonal basis of $W$: $$\vec{f}_1=\left[\begin{array}{r} -1\\3\\0\end{array}\right]\mbox{ and }\vec{f}_2=\left[\begin{array}{r}2\\0\\3\end{array}\right]-\frac{-2}{10}\left[\begin{array}{r}-1\\3\\0\end{array}\right]=\frac{1}{5}\left[\begin{array}{r}9\\3\\15\end{array}\right].\nonumber$$ Therefore $B=\left\{\left[\begin{array}{r}-1\\3\\0\end{array}\right], \left[\begin{array}{r}3\\1\\5 \end{array}\right]\right\}$ is an orthogonal basis of $W$.
3. To find the point $Z$ on $W$ closest to $Y=(1,1,1)$, compute $$\begin{aligned} \mathrm{proj}_{W}\left[\begin{array}{r} 1 \\ 1 \\ 1 \end{array}\right] & = \frac{2}{10} \left[\begin{array}{r} -1 \\ 3 \\ 0 \end{array}\right] + \frac{9}{35}\left[\begin{array}{r} 3 \\ 1 \\ 5 \end{array}\right]\\ & = \frac{1}{7}\left[\begin{array}{r} 4 \\ 6 \\ 9 \end{array}\right].\end{aligned}$$ Therefore, $Z=\left( \frac{4}{7}, \frac{6}{7}, \frac{9}{7}\right)$.