9.2: Gram-Schmidt Orthogonalization
- Page ID
- 21855
\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)
\( \newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\)
( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\)
\( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)
\( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\)
\( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)
\( \newcommand{\Span}{\mathrm{span}}\)
\( \newcommand{\id}{\mathrm{id}}\)
\( \newcommand{\Span}{\mathrm{span}}\)
\( \newcommand{\kernel}{\mathrm{null}\,}\)
\( \newcommand{\range}{\mathrm{range}\,}\)
\( \newcommand{\RealPart}{\mathrm{Re}}\)
\( \newcommand{\ImaginaryPart}{\mathrm{Im}}\)
\( \newcommand{\Argument}{\mathrm{Arg}}\)
\( \newcommand{\norm}[1]{\| #1 \|}\)
\( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\)
\( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\AA}{\unicode[.8,0]{x212B}}\)
\( \newcommand{\vectorA}[1]{\vec{#1}} % arrow\)
\( \newcommand{\vectorAt}[1]{\vec{\text{#1}}} % arrow\)
\( \newcommand{\vectorB}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\( \newcommand{\vectorC}[1]{\textbf{#1}} \)
\( \newcommand{\vectorD}[1]{\overrightarrow{#1}} \)
\( \newcommand{\vectorDt}[1]{\overrightarrow{\text{#1}}} \)
\( \newcommand{\vectE}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash{\mathbf {#1}}}} \)
\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \)
\( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)
\(\newcommand{\avec}{\mathbf a}\) \(\newcommand{\bvec}{\mathbf b}\) \(\newcommand{\cvec}{\mathbf c}\) \(\newcommand{\dvec}{\mathbf d}\) \(\newcommand{\dtil}{\widetilde{\mathbf d}}\) \(\newcommand{\evec}{\mathbf e}\) \(\newcommand{\fvec}{\mathbf f}\) \(\newcommand{\nvec}{\mathbf n}\) \(\newcommand{\pvec}{\mathbf p}\) \(\newcommand{\qvec}{\mathbf q}\) \(\newcommand{\svec}{\mathbf s}\) \(\newcommand{\tvec}{\mathbf t}\) \(\newcommand{\uvec}{\mathbf u}\) \(\newcommand{\vvec}{\mathbf v}\) \(\newcommand{\wvec}{\mathbf w}\) \(\newcommand{\xvec}{\mathbf x}\) \(\newcommand{\yvec}{\mathbf y}\) \(\newcommand{\zvec}{\mathbf z}\) \(\newcommand{\rvec}{\mathbf r}\) \(\newcommand{\mvec}{\mathbf m}\) \(\newcommand{\zerovec}{\mathbf 0}\) \(\newcommand{\onevec}{\mathbf 1}\) \(\newcommand{\real}{\mathbb R}\) \(\newcommand{\twovec}[2]{\left[\begin{array}{r}#1 \\ #2 \end{array}\right]}\) \(\newcommand{\ctwovec}[2]{\left[\begin{array}{c}#1 \\ #2 \end{array}\right]}\) \(\newcommand{\threevec}[3]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \end{array}\right]}\) \(\newcommand{\cthreevec}[3]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \end{array}\right]}\) \(\newcommand{\fourvec}[4]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]}\) \(\newcommand{\cfourvec}[4]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]}\) \(\newcommand{\fivevec}[5]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]}\) \(\newcommand{\cfivevec}[5]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]}\) \(\newcommand{\mattwo}[4]{\left[\begin{array}{rr}#1 \amp #2 \\ #3 \amp #4 \\ \end{array}\right]}\) \(\newcommand{\laspan}[1]{\text{Span}\{#1\}}\) \(\newcommand{\bcal}{\cal B}\) \(\newcommand{\ccal}{\cal C}\) \(\newcommand{\scal}{\cal S}\) \(\newcommand{\wcal}{\cal W}\) \(\newcommand{\ecal}{\cal E}\) \(\newcommand{\coords}[2]{\left\{#1\right\}_{#2}}\) \(\newcommand{\gray}[1]{\color{gray}{#1}}\) \(\newcommand{\lgray}[1]{\color{lightgray}{#1}}\) \(\newcommand{\rank}{\operatorname{rank}}\) \(\newcommand{\row}{\text{Row}}\) \(\newcommand{\col}{\text{Col}}\) \(\renewcommand{\row}{\text{Row}}\) \(\newcommand{\nul}{\text{Nul}}\) \(\newcommand{\var}{\text{Var}}\) \(\newcommand{\corr}{\text{corr}}\) \(\newcommand{\len}[1]{\left|#1\right|}\) \(\newcommand{\bbar}{\overline{\bvec}}\) \(\newcommand{\bhat}{\widehat{\bvec}}\) \(\newcommand{\bperp}{\bvec^\perp}\) \(\newcommand{\xhat}{\widehat{\xvec}}\) \(\newcommand{\vhat}{\widehat{\vvec}}\) \(\newcommand{\uhat}{\widehat{\uvec}}\) \(\newcommand{\what}{\widehat{\wvec}}\) \(\newcommand{\Sighat}{\widehat{\Sigma}}\) \(\newcommand{\lt}{<}\) \(\newcommand{\gt}{>}\) \(\newcommand{\amp}{&}\) \(\definecolor{fillinmathshade}{gray}{0.9}\)Suppose that \(M\) is an \(m\)-dimensional subspace with basis
\[\{x_{1}, \cdots, x_{m}\} \nonumber\]
We transform this into an orthonormal basis
\[\{q_{1}, \cdots, q_{m}\} \nonumber\]
for \(M\) via
1. Set \(y_{1} = x_{1}\) and \(q_{1} = \frac{y_{1}}{||y_{1}||}\)
2. \(y_{2} = x_{2}\) minus the projection of \(x_{2}\) onto the line spanned by \(q_{1}\).
That is,\[y_{2} = x_{2}-q_{1}(q_{1}^{T}q_{1})^{-1}q_{1}^{T}x_{2} = x_{2}-q_{1}q_{1}^{T}x_{2} \nonumber\]
Set \(q_{2} = \frac{y_{2}}{||y_{2}||}\) and \(Q_{2} = \{q_{1}, q_{2}\}\)
3. \(y_{3} = x_{3}\) minus the projection of \(x_{3}\) onto the plane spanned by \(q_{1}\) and \(q_{2}\). That is,
\[y_{3} = x_{3}-Q_{2}(Q_{2}^{T}Q_{2})^{-1}Q_{2}^{T} x_{3} = x_{3}-q_{1}q_{1}^{T}x_{3} \nonumber\]
Set \(q_{3} = \frac{y_{3}}{||y_{3}||}\) and \(Q_{3} = \{q_{1}, q_{2}, q_{3}\}\). Continue in this fashion through step (m)
- (m) \(y_{m} = x_{m}\) minus its projection onto the subspace spanned by the columns of \(Q_{m-1}\)
\[y_{m} = x_{m}-Q_{m-1}(Q_{m-1}^{T}Q_{m-1})^{-1}Q_{m-1}^{T} x_{m}x_{m}- \sum_{j=1}^{m-1} q_{j}q_{j}^{T} x_{m} \nonumber\]
Set \(q_{m} = \frac{y_{m}}{||y_{m}||}\). To take a simple example, let us orthogonalize the following basis for \(\mathbb{R}^3\)
\[\begin{array}{ccc} {x_{1} = \begin{pmatrix} {1}\\{0}\\{0} \end{pmatrix}}&{x_{2} = \begin{pmatrix} {1}\\{1}\\{0} \end{pmatrix}}&{x_{3} = \begin{pmatrix} {1}\\{1}\\{1} \end{pmatrix}} \end{array} \nonumber\]
- \(q_{1} = y_{1} = x_{1}\)
- \(y_{2} = x_{2}-q_{1}q_{1}^{T}x_{2} = \begin{pmatrix} {0}&{1}&{0} \end{pmatrix}^T\) and so, \(q_{2} = y_{2}\)
- \(y_{3} = x_{3}-q_{2}q_{2}^{T}x_{3} = \begin{pmatrix} {0}&{0}&{1} \end{pmatrix}^T\) and so, \(q_{3} = y_{3}\)
We have arrived at
\[\begin{array}{ccc} {q_{1} = \begin{pmatrix} {1}\\{0}\\{0} \end{pmatrix}}&{q_{2} = \begin{pmatrix} {0}\\{1}\\{0} \end{pmatrix}}&{q_{3} = \begin{pmatrix} {0}\\{0}\\{1} \end{pmatrix}} \end{array} \nonumber\]
Once the idea is grasped the actual calculations are best left to a machine. Matlab accomplishes this via the orth command. Its implementation is a bit more sophisticated than a blind run through our steps (1) through (m). As a result, there is no guarantee that it will return the same basis. For example
>>X=[1 1 1;0 1 1 ;0 0 1]; >>Q=orth(X) Q= 0.7370 -0.5910 0.3280 0.5910 0.3280 -0.7370 0.3280 0.7370 0.5910
This ambiguity does not bother us, for one orthogonal basis is as good as another. Let us put this into practice, via (10.8).