Skip to main content
Mathematics LibreTexts

9.5: The Gram-Schmidt Orthogonalization procedure

  • Page ID
    260
  • \( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \) \( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)\(\newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\) \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\) \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\) \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\) \( \newcommand{\Span}{\mathrm{span}}\) \(\newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\) \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\) \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\) \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\) \( \newcommand{\Span}{\mathrm{span}}\)\(\newcommand{\AA}{\unicode[.8,0]{x212B}}\)

    We now come to a fundamentally important algorithm, which is called the Gram-Schmidt orthogonalization procedure. This algorithm makes it possible to construct, for each list of linearly independent vectors (resp. basis), a corresponding orthonormal list (resp. orthonormal basis).

    Theorem 9.5.1

    If \((v_1,\ldots,v_m) \) is a list of linearly independent vectors in \(V\), then there exists an orthonormal list \((e_1,\ldots,e_m) \) such that

    \[ \Span(v_1,\ldots,v_k) = \Span(e_1,\ldots,e_k), \quad \text{for all \(k=1,\ldots,m\).} \label{9.5.1} \]

    Proof

    The proof is constructive, that is, we will actually construct vectors \(e_1,\ldots,e_m \) having the desired properties. Since \((v_1,\ldots,v_m) \) is linearly independent, \(v_k\neq 0 \) for each \(k=1,2,\ldots,m\). Set \(e_1=\frac{v_1}{\norm{v_1}}\). Then \(e_{1} \) is a vector of norm 1 and satisfies Equation (9.5.1) for \(k=1\). Next, set

    \[\begin{equation*}
    e_2 = \frac{v_2 - \inner{v_2}{e_1} e_1}{\norm{v_2 - \inner{v_2}{e_1} e_1}}.
    \end{equation*} \]

    This is, in fact, the normalized version of the orthogonal decomposition Equation(9.3.1)~\eqref{eq:orthogonal decomp}. I.e.,

    \[\begin{equation*}
    w = v_2 - \inner{v_2}{e_1}e_1,
    \end{equation*} \]

    where \(w\bot e_1\). Note that \(\norm{e_2}=1 \) and \(\Span(e_1,e_2)=\Span(v_1,v_2)\).

    Now, suppose that \(e_1,\ldots,e_{k-1} \) have been constructed such that \((e_1,\ldots,e_{k-1})\) is an orthonormal list and \(\Span(v_1,\ldots,v_{k-1}) = \Span(e_1,\ldots,e_{k-1})\). Then define
    \[\begin{equation*}
    e_k = \frac{v_k - \inner{v_k}{e_1}e_1 - \inner{v_k}{e_2} e_2 - \cdots -\inner{v_k}{e_{k-1}} e_{k-1}}
    {\norm{v_k - \inner{v_k}{e_1}e_1 - \inner{v_k}{e_2} e_2 - \cdots -\inner{v_k}{e_{k-1}} e_{k-1}}}.
    \end{equation*} \]

    Since \((v_1,\ldots,v_k) \) is linearly independent, we know that \(v_k\not\in \Span(v_1,\ldots,v_{k-1})\). Hence, we also know that \(v_k\not\in \Span(e_1,\ldots,e_{k-1})\). It follows that the norm in the definition of \(e_k \) is not zero, and so \(e_k \) is well-defined (i.e., we are not dividing by zero). Note that a vector divided by its norm has norm 1 so that \(\norm{e_k}=1\). Furthermore,

    \[\begin{equation*}
    \begin{split}
    \inner{e_k}{e_i} &= \left\langle \frac{v_k - \inner{v_k}{e_1}e_1 - \inner{v_k}{e_2} e_2 - \cdots
    -\inner{v_k}{e_{k-1}} e_{k-1}}
    {\norm{v_k - \inner{v_k}{e_1}e_1 - \inner{v_k}{e_2} e_2 - \cdots -\inner{v_k}{e_{k-1}} e_{k-1}}},
    e_i \right\rangle \\
    &= \frac{\inner{v_k}{e_i} - \inner{v_k}{e_i}}
    {\norm{v_k - \inner{v_k}{e_1}e_1 - \inner{v_k}{e_2} e_2 - \cdots -\inner{v_k}{e_{k-1}} e_{k-1}}}=0,
    \end{split}
    \end{equation*} \]

    for each \(1\le i<k\). Hence, \((e_1,\ldots,e_k) \) is orthonormal.

    \(\square\)

    From the definition of \(e_k\), we see that \(v_k\in \Span(e_1,\ldots,e_k) \) so that \(\Span(v_1,\ldots,v_k) \subset \Span(e_1,\ldots,e_k)\). Since both lists \((e_1,\ldots,e_k) \) and \((v_1,\ldots,v_k) \) are linearly independent, they must span subspaces of the same dimension and therefore are the same subspace. Hence Equation (9.5.1) holds.

    Example \(\PageIndex{2}\)

    Take \(v_1=(1,1,0) \) and \(v_2=(2,1,1) \) in \(\mathbb{R}^3\). The list \((v_1,v_2) \) is linearly independent (as you should verify!). To illustrate the Gram-Schmidt procedure, we begin by setting
    \[\begin{equation*}
    e_1 = \frac{v_1}{\norm{v_1}} = \frac{1}{\sqrt{2}} (1,1,0).
    \end{equation*} \]
    Next, set
    \[\begin{equation*}
    e_2 = \frac{v_2 - \inner{v_2}{e_1}e_1}{\norm{v_2 - \inner{v_2}{e_1}e_1}}.
    \end{equation*} \]
    The inner product \(\inner{v_2}{e_1}=\frac{1}{\sqrt{2}}\inner{(1,1,0)}{(2,1,1)}=\frac{3}{\sqrt{2}}\),
    so
    \[\begin{equation*}
    u_2 = v_2 - \inner{v_2}{e_1}e_1 = (2,1,1) - \frac{3}{2}(1,1,0) = \frac{1}{2}(1,-1,2).
    \end{equation*} \]
    Calculating the norm of \(u_2\), we obtain \(\norm{u_2}=\sqrt{\frac{1}{4}(1+1+4)} = \frac{\sqrt{6}}{2}\).
    Hence, normalizing this vector, we obtain
    \[\begin{equation*}
    e_2 = \frac{u_2}{\norm{u_2}} = \frac{1}{\sqrt{6}}(1,-1,2).
    \end{equation*} \]
    The list \((e_1,e_2) \) is therefore orthonormal and has the same span as \((v_1,v_2)\).

    Corollary 9.5.3.

    Every finite-dimensional inner product space has an orthonormal basis.

    Proof

    Let \((v_1,\ldots,v_n) \) be any basis for \(V\). This list is linearly independent and spans \(V\). Apply the Gram-Schmidt procedure to this list to obtain an orthonormal list \((e_1,\ldots,e_n)\), which still spans \(V \) by construction. By Proposition9.4.2~\ref{prop:orth li}, this list is linearly independent and hence a basis of \(V\).

    Corollary 9.5.4.

    Every orthonormal list of vectors in \(V \) can be extended to an orthonormal basis of \(V\).

    Proof

    Let \((e_1,\ldots,e_m) \) be an orthonormal list of vectors in \(V\). By Proposition9.4.2~\ref{prop:orth li}, this list is linearly independent and hence can be extended to a basis \((e_1,\ldots,e_m,v_1,\ldots,v_k) \) of \(V \) by the Basis Extension Theorem. Now apply the Gram-Schmidt procedure to obtain a new orthonormal basis \((e_1,\ldots,e_m,f_1,\ldots,f_k)\). The first \(m \) vectors do not change since they already are orthonormal. The list still spans \(V \) and is linearly independent by Proposition9.4.2~\ref{prop:orth li} and therefore forms a basis.

    Recall Theorem7.5.3~\ref{thm:ComplexLinearMapsUpperTriangularWrtSomeBasis}: given an operator \(T \in \mathcal{L}(V, V) \) on a complex vector space \(V\), there exists a basis \(B \) for \(V\) such that the matrix \(M(T)\) of \(T\) with respect to \(B \) is upper triangular. We would like to extend this result to require the additional property of orthonormality.

    Corollary 9.5.5

    Let \(V \) be an inner product space over \(\mathbb{F} \) and \(T\in\mathcal{L}(V,V)\). If \(T \) is upper-triangular with respect to some basis, then \(T \) is upper-triangular with respect to some orthonormal basis.

    Proof

    Let \((v_1,\ldots,v_n) \) be a basis of \(V \) with respect to which \(T \) is upper-triangular. Apply the Gram-Schmidt procedure to obtain an orthonormal basis \((e_1,\ldots,e_n)\), and note that

    \[ \Span(e_1,\ldots,e_k) = \Span(v_1,\ldots,v_k), \quad \text{for all \(1\le k\le n\).} \]

    We proved before that \(T \) is upper-triangular with respect to a basis \((v_1,\ldots,v_n) \) if and only if \(\Span(v_1,\ldots,v_k) \) is invariant under \(T \) for each \(1\le k\le n\). Since these spans are unchanged by the Gram-Schmidt procedure, \(T \) is still upper triangular for the corresponding orthonormal basis.


    This page titled 9.5: The Gram-Schmidt Orthogonalization procedure is shared under a not declared license and was authored, remixed, and/or curated by Isaiah Lankham, Bruno Nachtergaele, & Anne Schilling.