$$\newcommand{\id}{\mathrm{id}}$$ $$\newcommand{\Span}{\mathrm{span}}$$ $$\newcommand{\kernel}{\mathrm{null}\,}$$ $$\newcommand{\range}{\mathrm{range}\,}$$ $$\newcommand{\RealPart}{\mathrm{Re}}$$ $$\newcommand{\ImaginaryPart}{\mathrm{Im}}$$ $$\newcommand{\Argument}{\mathrm{Arg}}$$ $$\newcommand{\norm}[1]{\| #1 \|}$$ $$\newcommand{\inner}[2]{\langle #1, #2 \rangle}$$ $$\newcommand{\Span}{\mathrm{span}}$$

# 14.3: Relating Orthonormal Bases

[ "article:topic", "authortag:waldron", "orthogonal", "authorname:waldron", "Change of Basis" ]

$$\newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} }$$

$$\newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}}$$

Suppose $$T=\{u_{1}, \ldots, u_{n} \}$$ and $$R=\{w_{1}, \ldots, w_{n} \}$$ are two orthonormal bases for $$\Re^{n}$$. Then:

\begin{eqnarray*}
w_{1} &=& (w_{1}\cdot u_{1}) u_{1} + \cdots + (w_{1}\cdot u_{n})u_{n}\\
& \vdots & \\
w_{n} &=& (w_{n}\cdot u_{1}) u_{1} + \cdots + (w_{n}\cdot u_{n})u_{n}\\
\Rightarrow w_{i} &=& \sum_{j} u_{j}(u_{j}\cdot w_{i}) \\
\end{eqnarray*}

Thus the matrix for the change of basis from $$T$$ to $$R$$ is given by

$P = (P^{j}_{i}) = (u_{j}\cdot w_{i}).$

We would like to calculate the product $$PP^{T}$$. For that, we first develop a dirty trick for products of dot products:

$$(u.v)(w.z)=(u^{T} v) (w^{T} z) = u^{T} (v w^{T}) z\, .$$

The object $$v w^{T}$$ is the square matrix made from the outer product of $$v$$ and $$w$$! Now we are ready to compute the components of the matrix product $$PP^{T}$$:

\begin{eqnarray*}
\sum_{i}(u_{j}\cdot w_{i})(w_{i}\cdot u_{k})&=&
\sum_{i}(u_{j}^{T} w_{i}) (w_{i}^{T} u_{k})\\
&=& u_{j}^{T} \left[\sum_{i} (w_{i} w_{i}^{T}) \right] u_{k} \\
&\stackrel{(*)}=& u_{j}^{T} I_{n} u_{k} \\\
&=& u_{j}^{T} u_{k} = \delta_{jk}.
\end{eqnarray*}

The equality $$(*)$$ is explained below. Assuming $$(*)$$ holds, we have shown that $$PP^{T}=I_{n}$$, which implies that

$P^{T}=P^{-1}.$

The equality in the line $$(*)$$ says that $$\sum_{i} w_{i} w_{i}^{T}=I_{n}$$. To see this, we examine $$\left(\sum_{i} w_{i} w_{i}^{T}\right)v$$ for an arbitrary vector $$v$$. We can find constants $$c^{j}$$ such that $$v=\sum_{j} c^{j}w_{j}$$, so that:

\begin{eqnarray*}
\left(\sum_{i} w_{i} w_{i}^{T}\right)v &=& \left(\sum_{i} w_{i} w_{i}^{T}\right)\left(\sum_{j} c^{j}w_{j}\right) \\
&=& \sum_{j} c^{j} \sum_{i} w_{i} w_{i}^{T} w_{j} \\
&=& \sum_{j} c^{j} \sum_{i} w_{i} \delta_{ij} \\
&=& \sum_{j} c^{j} w_{j} \textit{ since all terms with $$i\neq j$$ vanish}\\
&=&v.
\end{eqnarray*}

Thus, as a linear transformation, $$\sum_{i} w_{i} w_{i}^{T}=I_{n}$$ fixes every vector, and thus must be the identity $$I_{n}$$.

Definition: Orthonality

A matrix $$P$$ is orthogonal if $$P^{-1}=P^{T}$$.

Then to summarize,

Theorem: Orthonormality

A change of basis matrix $$P$$ relating two orthonormal bases is an orthogonal matrix. $$\textit{i.e.}$$, $$P^{-1}=P^T.$$

Example 123

Consider $$\Re^{3}$$ with the orthonormal basis

$S=\left\{ u_{1}=\begin{pmatrix}\frac{2}{\sqrt{6}}\\ \frac{1}{\sqrt{6}}\\ \frac{-1}{\sqrt{6}}\end{pmatrix}, u_{2}=\begin{pmatrix}0\\ \frac{1}{\sqrt{2}}\\ \frac{1}{\sqrt{2}}\end{pmatrix}, u_{3}=\begin{pmatrix}\frac{1}{\sqrt{3}}\\ \frac{-1}{\sqrt{3}}\\ \frac{1}{\sqrt{3}}\end{pmatrix} \right\}.$

Let $$E$$ be the standard basis $$\{e_{1},e_{2},e_{3} \}$$. Since we are changing from the standard basis to a new basis, then the columns of the change of basis matrix are exactly the standard basis vectors. Then the change of basis matrix from $$E$$ to $$S$$ is given by:

\begin{eqnarray*}
P=(P^{j}_{i})=(e_{j}u_{i})&=&
\begin{pmatrix}
e_{1}\cdot u_{1} & e_{1}\cdot u_{2} & e_{1}\cdot u_{3} \\
e_{2}\cdot u_{1} & e_{2}\cdot u_{2} & e_{2}\cdot u_{3} \\
e_{3}\cdot u_{1} & e_{3}\cdot u_{2} & e_{3}\cdot u_{3} \\
\end{pmatrix} \\
= \begin{pmatrix}
u_{1} & u_{2} & u_{3}
\end{pmatrix}
&=&
\begin{pmatrix}
\frac{2}{\sqrt{6}} & 0 & \frac{1}{\sqrt{3}} \\
\frac{1}{\sqrt{6}}& \frac{1}{\sqrt{2}}&\frac{-1}{\sqrt{3}}\\
\frac{-1}{\sqrt{6}}& \frac{1}{\sqrt{2}}&\frac{1}{\sqrt{3}}\\
\end{pmatrix}. \\
\end{eqnarray*}

From our theorem, we observe that:

\begin{eqnarray*}
P^{-1}=P^{T} &=& \begin{pmatrix}u_{1}^{T}\\u_{2}^{T}\\u_{3}^{T}\end{pmatrix} \\
&=& \begin{pmatrix}
\frac{2}{\sqrt{6}}& \frac{1}{\sqrt{6}}& \frac{-1}{\sqrt{6}}\\
0 & \frac{1}{\sqrt{2}}&\frac{1}{\sqrt{2}} \\
\frac{1}{\sqrt{3}}& \frac{-1}{\sqrt{3}}&\frac{1}{\sqrt{3}}\\
\end{pmatrix}. \\
\end{eqnarray*}

We can check that $$P^{T}P=I$$ by a lengthy computation, or more simply, notice that

\begin{eqnarray*}
(P^{T}P)_{ij}
&=& \begin{pmatrix}u_{1}^{T}\\u_{2}^{T}\\u_{3}^{T}\end{pmatrix} \begin{pmatrix}u_{1} & u_{2}& u_{3}\end{pmatrix} \\
&=& \begin{pmatrix}1&0&0\\0&1&0\\0&0&1\end{pmatrix}.
\end{eqnarray*}

Above we are using orthonormality of the $$u_{i}$$ and the fact that matrix multiplication amounts to taking dot products between rows and columns. It is also very important to realize that the columns of an $$\textit{orthogonal}$$ matrix are made from an $$\textit{orthonormal}$$ set of vectors.

Remark: (Orthonormal Change of Basis and Diagonal Matrices)

Suppose $$D$$ is a diagonal matrix and we are able to use an orthogonal matrix $$P$$ to change to a new basis. Then the matrix $$M$$ of $$D$$ in the new basis is:

$M = PDP^{-1} = PDP^{T}.$

Now we calculate the transpose of $$M$$.

\begin{eqnarray*}
M^{T} &=& (PDP^{T})^{T}\\
&=& (P^{T})^{T}D^{T}P^{T} \\
&=& PDP^{T}\\
&=& M
\end{eqnarray*}

The matrix $$M=PDP^{T}$$ is symmetric!