Skip to main content
\(\newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\) \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\) \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\) \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\) \( \newcommand{\Span}{\mathrm{span}}\)
Mathematics LibreTexts

9.6 Orthogonal projections and minimization problems

Definition 9.6.1 Let \(V \) be a finite-dimensional inner product space and \(U\subset V \) be a subset (but not necessarily a subspace) of \(V\). Then the orthogonal complement of \(U \) is defined to be the set

\[    U^\bot =  \{ v \in V \mid \inner{u}{v}=0  ~\rm{for~ all}~ u\in U \} .\]

Note that, in fact, \(U^\bot \) is always a subspace of \(V \) (as you should check!) and that

\[   \{0\}^\bot = V \quad \text{and} \quad V^\bot = \{0\}. \]

In addition, if \(U_{1} \) and \(U_{2} \) are subsets of \(V \) satisfying \(U_1\subset U_2\), then \(U_2^\bot \subset U_1^\bot\).     Remarkably, if \(U\subset V \) is a subspace of \(V\), then we can say quite a bit more about \(U^{\bot}\).

Theorem 9.6.2. If \(U\subset V \) is a subspace of \(V\), then \(V=U\oplus U^\bot\).

Proof. We need to show two things:

  1. \(V=U+U^\bot\).
  2. \(U\cap U^\bot = \{0\}\).

To show Condition~1 holds, let \((e_1,\ldots,e_m) \) be an orthonormal basis of \(U\). Then, for all \(v\in V\), we can write
\begin{equation} \label{eq:U Ubot}
v=\underbrace{\inner{v}{e_1} e_1 + \cdots + \inner{v}{e_m} e_m}_u +
\underbrace{v-\inner{v}{e_1} e_1 - \cdots - \inner{v}{e_m} e_m}_w.  \tag{9.6.1}
\end{equation}
The vector \(u\in U\), and
\begin{equation*}
    \inner{w}{e_j} = \inner{v}{e_j} - \inner{v}{e_j} =0, ~\rm{for~ all}~  {j=1,2,\ldots,m  , }
\end{equation*}
since \((e_1,\ldots,e_m) \) is an orthonormal list of vectors. Hence, \(w\in U^\bot\). This implies that \(V=U+U^\bot\).

        To prove that Condition~2 also holds, let \(v\in U\cap U^\bot\). Then \(v \) has to be orthogonal to every vector in \(U\), including to itself, and so \(\inner{v}{v}=0\). However, this implies \(v=0 \) so that \(U\cap U^\bot=\{0\}\).

Example 9.6.3.  \(\mathbb{R}^2 \) is the direct sum of any two orthogonal lines, and \(\mathbb{R}^3 \) is the direct sum of any plane and any line orthogonal to the plane as illustrated in Figure 9.6.1. For example,
\begin{equation*}
\begin{split}
    \mathbb{R}^2 &= \{(x,0) \mid x\in \mathbb{R}\} \oplus \{ (0,y) \mid y\in\mathbb{R}\},\\
    \mathbb{R}^3 &= \{(x,y,0) \mid x,y\in \mathbb{R}\} \oplus \{(0,0,z) \mid z\in \mathbb{R}\}.
\end{split}
\end{equation*}

 

 

 

Figure 9.6.1: \( \mathbb{R^3} \) as a direct sum of a plane and a line.

Another fundamental fact about the orthogonal complement of a subspace is as follows.

Theorem 9.6.4. If \(U\subset V \) is a subspace of \(V\), then \(U=(U^\bot)^\bot\).

Proof.  First we show that \(U\subset (U^\bot)^\bot\). Let \(u\in U\). Then, for all \(v\in U^\bot\), we have \(\inner{u}{v}=0\). Hence, \(u\in (U^\bot)^\bot \) by the definition of \((U^\bot)^\bot\).

Next we show that \((U^\bot)^\bot\subset U\). Suppose \(0\neq v\in (U^\bot)^\bot \) such that \(v\not\in U\), and decompose \(v \) according to Theorem 9.6.2, i.e., as
\begin{equation*}
    v = u_1 + u_2 \in U \oplus U^\bot
\end{equation*}
with \(u_1\in U \) and \(u_2\in U^\bot\). Then \(u_2\neq 0 \) since \(v\not\in U\).

Furthermore, \(\inner{u_2}{v} = \inner{u_2}{u_2} \neq 0\). But then \(v \) is not in \((U^\bot)^\bot\), which contradicts our initial assumption. Hence, we must have that \((U^\bot)^\bot\subset U\).

         By Theorem 9.6.2, we have the decomposition \(V=U\oplus U^\bot \) for every subspace \(U\subset V\). This allows us to define the orthogonal projection \(P_U \) of \(V \) onto \(U\).

Definition 9.6.5.  Let \(U\subset V \) be a subspace of a finite-dimensional inner product space. Every \(v\in V \) can be uniquely written as \(v=u+w \) where \(u\in U \) and \(w\in U^\bot\). Define
\begin{equation*}
\begin{split}
    P_U:V &\to V,\\
    v &\mapsto u.
\end{split}
\end{equation*}

 

         Note that \(P_U \) is called a projection operator since it satisfies \(P_U^2=P_U\). Further, since we also have
\begin{equation*}
\begin{split}
    &\range(P_U)= U,\\
    &\kernel(P_U) = U^\bot,
\end{split}
\end{equation*}
it follows that \(\range(P_U) \bot \kernel(P_U)\). Therefore, \(P_U \) is called an orthogonal projection.

        The decomposition of a vector \(v\in V\) as given in Equation (9.6.1) yields the formula
\begin{equation} \label{eq:ortho decomp}
    P_U v = \inner{v}{e_1} e_1 + \cdots + \inner{v}{e_m} e_m,  \tag{9.6.2}
\end{equation}
where \((e_1,\ldots,e_m) \) is any orthonormal basis of \(U\). Equation (9.6.2) is a particularly useful tool for computing such things as the matrix of \(P_{U} \) with respect to the basis \((e_1,\ldots,e_m)\).

Let us now apply the inner product to the following minimization problem: Given a subspace \(U\subset V \) and a vector \(v\in V\), find the vector \(u\in U \) that is closest to the vector \(v\). In other words, we want to make \(\norm{v-u} \) as small as possible. The next proposition shows that \(P_Uv \) is the closest point in \(U \) to the vector \(v \) and that this minimum is, in fact, unique.

Proposition 9.6.6.  Let \(U\subset V \) be a subspace of \(V \) and \(v\in V\). Then

\[   \norm{v-P_Uv} \le \norm{v-u} \qquad \text{for every \(u\in U\).}\]

Furthermore, equality holds if and only if \(u=P_Uv\).

Proof.  Let \(u\in U \) and set \(P:=P_U \) for short. Then
\begin{equation*}
\begin{split}
    \norm{v-P v}^2 &\le \norm{v-P v}^2 + \norm{Pv-u}^2\\
    &= \norm{(v-P v)+(P v-u)}^2 = \norm{v-u}^2,
\end{split}
\end{equation*}
where the second line follows from the Pythagorean Theorem 9.3.2~\ref{thm:pythagoras} since \(v-Pv\in U^\bot \) and \(Pv-u\in U\). Furthermore, equality holds only if \(\norm{Pv-u}^2=0\), which is equivalent to \(Pv=u\).

Example 9.6.7. Consider the plane \(U\subset \mathbb{R}^3 \) through 0 and perpendicular to the vector \(u=(1,1,1)\). Using the standard norm on \(\mathbb{R}^3\), we can calculate the distance of the point \(v=(1,2,3) \) to \(U \) using Proposition 9.6.6. In particular, the distance \(d \) between \(v \) and \(U \) is given by \(d=\norm{v-P_Uv}\). Let \((\frac{1}{\sqrt{3}}u,u_1,u_2) \) be a basis for \(\mathbb{R}^3 \) such that \((u_1,u_2) \) is an orthonormal basis of \(U\). Then, by Equation (9.6.2), we have
\begin{align*}
    v-P_Uv & = (\frac{1}{3}\inner{v}{u}u+\inner{v}{u_1}u_1+\inner{v}{u_2}u_2) - (\inner{v}{u_1}u_1 +\inner{v}{u_2}u_2)\\
           & = \frac{1}{3}\inner{v}{u}u\\
           & = \frac{1}{3}\inner{(1,2,3)}{(1,1,1)}(1,1,1)\\
           & = (2,2,2).
\end{align*}
Hence, \(d=\norm{(2,2,2)}=2\sqrt{3}\).

Contributors

Both hardbound and softbound versions of this textbook are available online at WorldScientific.com.