3.7: Orthogonal Projections
View Orthogonal Projections on YouTube
Suppose that \(V\) is an \(n\)-dimensional vector space and \(W\) is a \(p\)-dimensional subspace of \(V\). Let \(\{\text{s}_1,\: \text{s}_2,\:\cdots , \text{s}_p\}\) be an orthonormal basis for \(W\). Extending the basis for \(W\), let \(\{\text{s}_1,\: \text{s}_2,\cdots , \text{s}_p,\: \text{t}_1,\: \text{t}_2,\cdots , \text{t}_{n−p}\}\) be an orthonormal basis for \(V\).
Any vector \(\text{v}\) in \(V\) can be written in terms of the basis for \(V\) as
\[\text{v}=a_1\text{s}_1+a_2\text{s}_2+\cdots +a_p\text{s}_p+b_1\text{t}_1+b_2\text{t}_2+b_{n-p}\text{t}_{n-p}.\nonumber \]
The orthogonal projection of \(\text{v}\) onto \(W\) is then defined as
\[\text{v}_{\text{proj}_w}=a_1\text{s}_1+a_2\text{s}_2+\cdots +a_p\text{s}_p,\nonumber \]
that is, the part of \(\text{v}\) that lies in \(W\).
If you only know the vector \(v\) and the orthonormal basis for \(W\), then the orthogonal projection of \(\text{v}\) onto \(W\) can be computed from
\[\text{v}_{\text{proj}_w}=(\text{v}^{\text{T}}\text{s}_1)\text{s}_1+(\text{v}^{\text{T}}\text{s}_2)\text{s}_2+\cdots +(\text{v}^{\text{T}}\text{s}_p)\text{s}_p,\nonumber \]
that is, \(a_1=\text{v}^{\text{T}}\text{s}_1,\: a_2=\text{v}^{\text{T}}\text{s}_2\), etc.
We can prove that the vector \(\text{v}_{\text{proj}_W}\) is the vector in \(W\) that is closest to \(\text{v}\). Let \(\text{w}\) be any vector in \(W\) different than \(\text{v}_{\text{proj}_W}\), and expand \(\text{w}\) in terms of the basis vectors for \(W\):
\[\text{w}=c_1\text{s}_1+c_2\text{s}_2+\cdots +c_p\text{s}_p.\nonumber \]
The distance between \(\text{v}\) and \(\text{w}\) is given by the norm \(||\text{v} − \text{w}||\), and we have
\[\begin{aligned}||\text{v}-\text{w}||^2&=(a_1-c_1)^2+(a_2-c_2)^2+\cdots +(a_p-c_p)^2+b_1^2+b_2^2+\cdots +b_{n-p}^2 \\ &\geq b_1^2+b_2^2+\cdots +b_{n-p}^2=||\text{v}-\text{v}_{\text{proj}_w}||^2,\end{aligned} \nonumber \]
or \(||\text{v}-\text{v}_{\text{proj}_w}||\leq ||\text{v}-\text{w}||\), a result that will be used later in the problem of least squares.