5.6: Appendix- The Least Squares Approximation

In the first section of this chapter we showed that we can expand functions over an infinite set of basis functions as

$$f(x)=\sum_{n=1}^{\infty} c_{n} \phi_{n}(x)\nonumber$$

and that the generalized Fourier coefficients are given by

$$c_{n}=\frac{\left\langle\phi_{n}, f\right\rangle}{\left\langle\phi_{n}, \phi_{n}\right\rangle} .\nonumber$$

In this section we turn to a discussion of approximating $f(x)$ by the partial sums $\sum_{n=1}^{N} c_{n} \phi_{n}(x)$ and showing that the Fourier coefficients are the best coefficients minimizing the deviation of the partial sum from $f(x)$. This will lead us to a discussion of the convergence of Fourier series.

More specifically, we set the following goal:

Goal

To find the best approximation of $f(x)$ on $[a, b]$ by $S_{N}(x)=\sum_{n=1}^{N} c_{n} \phi_{n}(x)$ for a set of fixed functions $\phi_{n}(x)$; i.e., to find the expansion coefficients, $c_{n}$, such that $S_{N}(x)$ approximates $f(x)$ in the least squares sense.

We want to measure the deviation of the finite sum from the given function. Essentially, we want to look at the error made in the approximation. This is done by introducing the mean square deviation:

$$E_{N}=\int_{a}^{b}\left[f(x)-S_{N}(x)\right]^{2} \rho(x) d x,\nonumber$$

where we have introduced the weight function $\rho(x)>0$. It gives us a sense as to how close the $N$ th partial sum is to $f(x)$.

We want to minimize this deviation by choosing the right $c_{n}$ ’s. We begin by inserting the partial sums and expand the square in the integrand:

$$\begin{align} E_{N}=& \int_{a}^{b}\left[f(x)-S_{N}(x)\right]^{2} \rho(x) d x\nonumber \\ =& \int_{a}^{b}\left[f(x)-\sum_{n=1}^{N} c_{n} \phi_{n}(x)\right]^{2} \rho(x) d x\nonumber \\ =& \int_{a}^{b} f^{2}(x) \rho(x) d x-2 \int_{a}^{b} f(x) \sum_{n=1}^{N} c_{n} \phi_{n}(x) \rho(x) d x\nonumber \\ &+\int_{a}^{b} \sum_{n=1}^{N} c_{n} \phi_{n}(x) \sum_{m=1}^{N} c_{m} \phi_{m}(x) \rho(x) d x\label{eq:1} \end{align}$$

Looking at the three resulting integrals, we see that the first term is just the inner product of $f$ with itself. The other integrations can be rewritten after interchanging the order of integration and summation. The double sum can be reduced to a single sum using the orthogonality of the $\phi_{n}$ ’s. Thus, we have

$$\begin{align} E_{N} &=\langle f, f\rangle-2 \sum_{n=1}^{N} c_{n}\left\langle f, \phi_{n}\right\rangle+\sum_{n=1}^{N} \sum_{m=1}^{N} c_{n} c_{m}\left\langle\phi_{n}, \phi_{m}\right\rangle\nonumber \\ &=\langle f, f\rangle-2 \sum_{n=1}^{N} c_{n}\left\langle f, \phi_{n}\right\rangle+\sum_{n=1}^{N} c_{n}^{2}\left\langle\phi_{n}, \phi_{n}\right\rangle .\label{eq:2} \end{align}$$

We are interested in finding the coefficients, so we will complete the square in $c_{n}$. Focusing on the last two terms, we have

$$\begin{align} &-2 \sum_{n=1}^{N} c_{n}<f, \phi_{n}>+\sum_{n=1}^{N} c_{n}^{2}<\phi_{n}, \phi_{n}>\nonumber \\ =& \sum_{n=1}^{N}<\phi_{n}, \phi_{n}>c_{n}^{2}-2<f, \phi_{n}>c_{n}\nonumber \\ =& \sum_{n=1}^{N}<\phi_{n}, \phi_{n}>\left[c_{n}^{2}-\frac{2<f, \phi_{n}>c_{n}}{\left\langle\phi_{n}, \phi_{n}\right\rangle}\right]\nonumber \\ =& \sum_{n=1}^{N}<\phi_{n}, \phi_{n}>\left[\left(c_{n}-\frac{\left\langle f, \phi_{n}\right\rangle}{\left\langle\phi_{n}, \phi_{n}\right\rangle}\right)^{2}-\left(\frac{\left\langle f, \phi_{n}\right\rangle}{<\phi_{n}, \phi_{n}>}\right)^{2}\right] .\label{eq:3} \end{align}$$

To this point we have shown that the mean square deviation is given as

$$E_{N}=\langle f, f\rangle+\sum_{n=1}^{N}\left\langle\phi_{n}, \phi_{n}\right\rangle\left[\left(c_{n}-\frac{\left\langle f, \phi_{n}\right\rangle}{\left\langle\phi_{n}, \phi_{n}\right\rangle}\right)^{2}-\left(\frac{\left\langle f, \phi_{n}\right\rangle}{\left\langle\phi_{n}, \phi_{n}\right\rangle}\right)^{2}\right] .\nonumber$$

So, $E_{N}$ is minimized by choosing

$$c_{n}=\frac{\left\langle f, \phi_{n}\right\rangle}{\left\langle\phi_{n}, \phi_{n}\right\rangle} .\nonumber$$

However, these are the Fourier Coefficients. This minimization is often referred to as Minimization in Least Squares Sense.

Inserting the Fourier coefficients into the mean square deviation yields

$$0 \leq E_{N}=\langle f, f\rangle-\sum_{n=1}^{N} c_{n}^{2}\left\langle\phi_{n}, \phi_{n}\right\rangle .\nonumber$$

Thus, we obtain Bessel’s Inequality:

$$<f, f>\geq \sum_{n=1}^{N} c_{n}^{2}<\phi_{n}, \phi_{n}>.\nonumber$$

For convergence, we next let $N$ get large and see if the partial sums converge to the function. In particular, we say that the infinite series converges in the mean if

$$\int_{a}^{b}\left[f(x)-S_{N}(x)\right]^{2} \rho(x) d x \rightarrow 0 \text { as } N \rightarrow \infty .\nonumber$$

Letting $N$ get large in Bessel's inequality shows that the sum $\sum_{n=1}^N c_n^2 < \phi_n,\phi_n>$ converges if

$$<f, f>=\int_{a}^{b} f^{2}(x) \rho(x) d x<\infty .\nonumber$$

The space of all such $f$ is denoted $L_{\rho}^{2}(a, b)$, the space of square integrable functions on $(a, b)$ with weight $\rho(x)$.

From the $n$th term divergence test from calculus we know that $\sum a_{n}$ converges implies that $a_{n} \rightarrow 0$ as $n \rightarrow \infty$. Therefore, in this problem the terms $c_{n}^{2}<\phi_{n}, \phi_{n}>$ approach zero as $n$ gets large. This is only possible if the $c_{n}$ ’s go to zero as $n$ gets large. Thus, if $\sum_{n=1}^{N} c_{n} \phi_{n}$ converges in the mean to $f$, then $\int_{a}^{b}\left[f(x)-\sum_{n=1}^{N} c_{n} \phi_{n}\right]^{2} \rho(x) d x$ approaches zero as $N \rightarrow \infty$. This implies from the above derivation of Bessel’s inequality that

$$<f, f>-\sum_{n=1}^{N} c_{n}^{2}\left(\phi_{n}, \phi_{n}\right) \rightarrow 0 .\nonumber$$

This leads to Parseval’s equality:

$$\langle f, f\rangle=\sum_{n=1}^{\infty} c_{n}^{2}\left\langle\phi_{n}, \phi_{n}\right\rangle .\nonumber$$

Parseval’s equality holds if and only if

$$\lim _{N \rightarrow \infty} \int_{a}^{b}\left(f(x)-\sum_{n=1}^{N} c_{n} \phi_{n}(x)\right)^{2} \rho(x) d x=0 .\nonumber$$

If this is true for every square integrable function in $L_{\rho}^{2}(a, b)$, then the set of functions $\left\{\phi_{n}(x)\right\}_{n=1}^{\infty}$ is said to be complete. One can view these functions as an infinite dimensional basis for the space of square integrable functions on $(a, b)$ with weight $\rho(x)>0$.

One can extend the above limit $c_{n} \rightarrow 0$ as $n \rightarrow \infty$, by assuming that $\frac{\phi_{n}(x)}{\left\|\phi_{n}\right\|}$ is uniformly bounded and that $\int_{a}^{b}|f(x)| \rho(x) d x<\infty$. This is the RiemannLebesgue Lemma, but will not be proven here.