Search

Text Color

Margin Size

Font Type

Enable Dyslexic Font

9.5: The Gram-Schmidt Orthogonalization procedure

Last updated

Mar 5, 2021
Save as PDF
- 9.4: Orthonormal bases
- 9.6: Orthogonal projections and minimization problems

$\newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} }$

$\newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}}$

$\newcommand{\id}{\mathrm{id}}$ $\newcommand{\Span}{\mathrm{span}}$

( \newcommand{\kernel}{\mathrm{null}\,}\) $\newcommand{\range}{\mathrm{range}\,}$

$\newcommand{\RealPart}{\mathrm{Re}}$ $\newcommand{\ImaginaryPart}{\mathrm{Im}}$

$\newcommand{\Argument}{\mathrm{Arg}}$ $\newcommand{\norm}[1]{\| #1 \|}$

$\newcommand{\inner}[2]{\langle #1, #2 \rangle}$

$\newcommand{\Span}{\mathrm{span}}$

$\newcommand{\id}{\mathrm{id}}$

$\newcommand{\Span}{\mathrm{span}}$

$\newcommand{\kernel}{\mathrm{null}\,}$

$\newcommand{\range}{\mathrm{range}\,}$

$\newcommand{\RealPart}{\mathrm{Re}}$

$\newcommand{\ImaginaryPart}{\mathrm{Im}}$

$\newcommand{\Argument}{\mathrm{Arg}}$

$\newcommand{\norm}[1]{\| #1 \|}$

$\newcommand{\inner}[2]{\langle #1, #2 \rangle}$

$\newcommand{\Span}{\mathrm{span}}$ $\newcommand{\AA}{\unicode[.8,0]{x212B}}$

$\newcommand{\vectorA}[1]{\vec{#1}} % arrow$

$\newcommand{\vectorAt}[1]{\vec{\text{#1}}} % arrow$

$\newcommand{\vectorB}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} }$

$\newcommand{\vectorC}[1]{\textbf{#1}}$

$\newcommand{\vectorD}[1]{\overrightarrow{#1}}$

$\newcommand{\vectorDt}[1]{\overrightarrow{\text{#1}}}$

$\newcommand{\vectE}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash{\mathbf {#1}}}}$

$\newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} }$

$\newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}}$

$\newcommand{\avec}{\mathbf a}$

$\newcommand{\bvec}{\mathbf b}$

$\newcommand{\cvec}{\mathbf c}$

$\newcommand{\dvec}{\mathbf d}$

$\newcommand{\dtil}{\widetilde{\mathbf d}}$

$\newcommand{\evec}{\mathbf e}$

$\newcommand{\fvec}{\mathbf f}$

$\newcommand{\nvec}{\mathbf n}$

$\newcommand{\pvec}{\mathbf p}$

$\newcommand{\qvec}{\mathbf q}$

$\newcommand{\svec}{\mathbf s}$

$\newcommand{\tvec}{\mathbf t}$

$\newcommand{\uvec}{\mathbf u}$

$\newcommand{\vvec}{\mathbf v}$

$\newcommand{\wvec}{\mathbf w}$

$\newcommand{\xvec}{\mathbf x}$

$\newcommand{\yvec}{\mathbf y}$

$\newcommand{\zvec}{\mathbf z}$

$\newcommand{\rvec}{\mathbf r}$

$\newcommand{\mvec}{\mathbf m}$

$\newcommand{\zerovec}{\mathbf 0}$

$\newcommand{\onevec}{\mathbf 1}$

$\newcommand{\real}{\mathbb R}$

$\newcommand{\twovec}[2]{\left[\begin{array}{r}#1 \\ #2 \end{array}\right]}$

$\newcommand{\ctwovec}[2]{\left[\begin{array}{c}#1 \\ #2 \end{array}\right]}$

$\newcommand{\threevec}[3]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \end{array}\right]}$

$\newcommand{\cthreevec}[3]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \end{array}\right]}$

$\newcommand{\fourvec}[4]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]}$

$\newcommand{\cfourvec}[4]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \end{array}\right]}$

$\newcommand{\fivevec}[5]{\left[\begin{array}{r}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]}$

$\newcommand{\cfivevec}[5]{\left[\begin{array}{c}#1 \\ #2 \\ #3 \\ #4 \\ #5 \\ \end{array}\right]}$

$\newcommand{\mattwo}[4]{\left[\begin{array}{rr}#1 \amp #2 \\ #3 \amp #4 \\ \end{array}\right]}$

$\newcommand{\laspan}[1]{\text{Span}\{#1\}}$

$\newcommand{\bcal}{\cal B}$

$\newcommand{\ccal}{\cal C}$

$\newcommand{\scal}{\cal S}$

$\newcommand{\wcal}{\cal W}$

$\newcommand{\ecal}{\cal E}$

$\newcommand{\coords}[2]{\left\{#1\right\}_{#2}}$

$\newcommand{\gray}[1]{\color{gray}{#1}}$

$\newcommand{\lgray}[1]{\color{lightgray}{#1}}$

$\newcommand{\rank}{\operatorname{rank}}$

$\newcommand{\row}{\text{Row}}$

$\newcommand{\col}{\text{Col}}$

$\renewcommand{\row}{\text{Row}}$

$\newcommand{\nul}{\text{Nul}}$

$\newcommand{\var}{\text{Var}}$

$\newcommand{\corr}{\text{corr}}$

$\newcommand{\len}[1]{\left|#1\right|}$

$\newcommand{\bbar}{\overline{\bvec}}$

$\newcommand{\bhat}{\widehat{\bvec}}$

$\newcommand{\bperp}{\bvec^\perp}$

$\newcommand{\xhat}{\widehat{\xvec}}$

$\newcommand{\vhat}{\widehat{\vvec}}$

$\newcommand{\uhat}{\widehat{\uvec}}$

$\newcommand{\what}{\widehat{\wvec}}$

$\newcommand{\Sighat}{\widehat{\Sigma}}$

$\newcommand{\lt}{<}$

$\newcommand{\gt}{>}$

$\newcommand{\amp}{&}$

$\definecolor{fillinmathshade}{gray}{0.9}$

We now come to a fundamentally important algorithm, which is called the Gram-Schmidt orthogonalization procedure. This algorithm makes it possible to construct, for each list of linearly independent vectors (resp. basis), a corresponding orthonormal list (resp. orthonormal basis).

Theorem 9.5.1

If $(v_1,\ldots,v_m)$ is a list of linearly independent vectors in $V$ , then there exists an orthonormal list $(e_1,\ldots,e_m)$ such that

$\Span(v_1,\ldots,v_k) = \Span(e_1,\ldots,e_k), \quad \text{for all $k=1,\ldots,m$.} \label{9.5.1}$

Proof

The proof is constructive, that is, we will actually construct vectors $e_1,\ldots,e_m$ having the desired properties. Since $(v_1,\ldots,v_m)$ is linearly independent, $v_k\neq 0$ for each $k=1,2,\ldots,m$ . Set $e_1=\frac{v_1}{\norm{v_1}}$ . Then $e_{1}$ is a vector of norm 1 and satisfies Equation (9.5.1) for $k=1$ . Next, set

$\begin{equation*} e_2 = \frac{v_2 - \inner{v_2}{e_1} e_1}{\norm{v_2 - \inner{v_2}{e_1} e_1}}. \end{equation*}$

This is, in fact, the normalized version of the orthogonal decomposition Equation(9.3.1)~ $\eqref{eq:orthogonal decomp}$ . I.e.,

$\begin{equation*} w = v_2 - \inner{v_2}{e_1}e_1, \end{equation*}$

where $w\bot e_1$ . Note that $\norm{e_2}=1$ and $\Span(e_1,e_2)=\Span(v_1,v_2)$ .

Now, suppose that $e_1,\ldots,e_{k-1}$ have been constructed such that $(e_1,\ldots,e_{k-1})$ is an orthonormal list and $\Span(v_1,\ldots,v_{k-1}) = \Span(e_1,\ldots,e_{k-1})$ . Then define
$\begin{equation*} e_k = \frac{v_k - \inner{v_k}{e_1}e_1 - \inner{v_k}{e_2} e_2 - \cdots -\inner{v_k}{e_{k-1}} e_{k-1}} {\norm{v_k - \inner{v_k}{e_1}e_1 - \inner{v_k}{e_2} e_2 - \cdots -\inner{v_k}{e_{k-1}} e_{k-1}}}. \end{equation*}$

Since $(v_1,\ldots,v_k)$ is linearly independent, we know that $v_k\not\in \Span(v_1,\ldots,v_{k-1})$ . Hence, we also know that $v_k\not\in \Span(e_1,\ldots,e_{k-1})$ . It follows that the norm in the definition of $e_k$ is not zero, and so $e_k$ is well-defined (i.e., we are not dividing by zero). Note that a vector divided by its norm has norm 1 so that $\norm{e_k}=1$ . Furthermore,

$\begin{equation*} \begin{split} \inner{e_k}{e_i} &= \left\langle \frac{v_k - \inner{v_k}{e_1}e_1 - \inner{v_k}{e_2} e_2 - \cdots -\inner{v_k}{e_{k-1}} e_{k-1}} {\norm{v_k - \inner{v_k}{e_1}e_1 - \inner{v_k}{e_2} e_2 - \cdots -\inner{v_k}{e_{k-1}} e_{k-1}}}, e_i \right\rangle \\ &= \frac{\inner{v_k}{e_i} - \inner{v_k}{e_i}} {\norm{v_k - \inner{v_k}{e_1}e_1 - \inner{v_k}{e_2} e_2 - \cdots -\inner{v_k}{e_{k-1}} e_{k-1}}}=0, \end{split} \end{equation*}$

for each $1\le i<k$ . Hence, $(e_1,\ldots,e_k)$ is orthonormal.

$\square$

From the definition of $e_k$ , we see that $v_k\in \Span(e_1,\ldots,e_k)$ so that $\Span(v_1,\ldots,v_k) \subset \Span(e_1,\ldots,e_k)$ . Since both lists $(e_1,\ldots,e_k)$ and $(v_1,\ldots,v_k)$ are linearly independent, they must span subspaces of the same dimension and therefore are the same subspace. Hence Equation (9.5.1) holds.

Example $\PageIndex{2}$

Take $v_1=(1,1,0)$ and $v_2=(2,1,1)$ in $\mathbb{R}^3$ . The list $(v_1,v_2)$ is linearly independent (as you should verify!). To illustrate the Gram-Schmidt procedure, we begin by setting
$\begin{equation*} e_1 = \frac{v_1}{\norm{v_1}} = \frac{1}{\sqrt{2}} (1,1,0). \end{equation*}$
Next, set
$\begin{equation*} e_2 = \frac{v_2 - \inner{v_2}{e_1}e_1}{\norm{v_2 - \inner{v_2}{e_1}e_1}}. \end{equation*}$
The inner product $\inner{v_2}{e_1}=\frac{1}{\sqrt{2}}\inner{(1,1,0)}{(2,1,1)}=\frac{3}{\sqrt{2}}$ ,
so
$\begin{equation*} u_2 = v_2 - \inner{v_2}{e_1}e_1 = (2,1,1) - \frac{3}{2}(1,1,0) = \frac{1}{2}(1,-1,2). \end{equation*}$
Calculating the norm of $u_2$ , we obtain $\norm{u_2}=\sqrt{\frac{1}{4}(1+1+4)} = \frac{\sqrt{6}}{2}$ .
Hence, normalizing this vector, we obtain
$\begin{equation*} e_2 = \frac{u_2}{\norm{u_2}} = \frac{1}{\sqrt{6}}(1,-1,2). \end{equation*}$
The list $(e_1,e_2)$ is therefore orthonormal and has the same span as $(v_1,v_2)$ .

Corollary 9.5.3.

Every finite-dimensional inner product space has an orthonormal basis.

Proof

Let $(v_1,\ldots,v_n)$ be any basis for $V$ . This list is linearly independent and spans $V$ . Apply the Gram-Schmidt procedure to this list to obtain an orthonormal list $(e_1,\ldots,e_n)$ , which still spans $V$ by construction. By Proposition9.4.2~ $\ref{prop:orth li}$ , this list is linearly independent and hence a basis of $V$ .

Corollary 9.5.4.

Every orthonormal list of vectors in $V$ can be extended to an orthonormal basis of $V$ .

Proof

Let $(e_1,\ldots,e_m)$ be an orthonormal list of vectors in $V$ . By Proposition9.4.2~ $\ref{prop:orth li}$ , this list is linearly independent and hence can be extended to a basis $(e_1,\ldots,e_m,v_1,\ldots,v_k)$ of $V$ by the Basis Extension Theorem. Now apply the Gram-Schmidt procedure to obtain a new orthonormal basis $(e_1,\ldots,e_m,f_1,\ldots,f_k)$ . The first $m$ vectors do not change since they already are orthonormal. The list still spans $V$ and is linearly independent by Proposition9.4.2~ $\ref{prop:orth li}$ and therefore forms a basis.

Recall Theorem7.5.3~ $\ref{thm:ComplexLinearMapsUpperTriangularWrtSomeBasis}$ : given an operator $T \in \mathcal{L}(V, V)$ on a complex vector space $V$ , there exists a basis $B$ for $V$ such that the matrix $M(T)$ of $T$ with respect to $B$ is upper triangular. We would like to extend this result to require the additional property of orthonormality.

Corollary 9.5.5

Let $V$ be an inner product space over $\mathbb{F}$ and $T\in\mathcal{L}(V,V)$ . If $T$ is upper-triangular with respect to some basis, then $T$ is upper-triangular with respect to some orthonormal basis.

Proof

Let $(v_1,\ldots,v_n)$ be a basis of $V$ with respect to which $T$ is upper-triangular. Apply the Gram-Schmidt procedure to obtain an orthonormal basis $(e_1,\ldots,e_n)$ , and note that

$\Span(e_1,\ldots,e_k) = \Span(v_1,\ldots,v_k), \quad \text{for all $1\le k\le n$.}$

We proved before that $T$ is upper-triangular with respect to a basis $(v_1,\ldots,v_n)$ if and only if $\Span(v_1,\ldots,v_k)$ is invariant under $T$ for each $1\le k\le n$ . Since these spans are unchanged by the Gram-Schmidt procedure, $T$ is still upper triangular for the corresponding orthonormal basis.

Contributors

Both hardbound and softbound versions of this textbook are available online at WorldScientific.com.

Contributors

Support Center

How can we help?